Beta
320660

Web Crawler Architecture over Cloud Computing compared with Grid Computing

Article

Last updated: 28 Dec 2024

Subjects

-

Tags

-

Abstract

Web Crawler is considered as the core module of web search engines. It should be designed to cover high percent of Internet and adapt on scaling and in a distributed architecture. The crawler architecture has an effect on the quantity of fetched web pages in a determined time. Cloud computing is a type of computing paradigm that is characterized by a set of powerful points such as excitability, scalability, dynamism, and resource provisioning on demand, where these features are adding value in the crawler architecture. In this article, we propose an architecture for the web crawler that is designed over the cloud computing. The web crawler needs highly intensive computation, storage, and bandwidth. These resources can be provisioned by the cloud computing on demand with superior flexibility in changing as in the proposed architecture. We implemented and experimented the proposed architecture over cloud computing and evaluated the results of running. We also proposed another architecture based on grid computing to compare the results of the experiments over cloud computing with results over grid computing to evaluate the cloud-based architecture. Cloud computing has a higher performance than the grid computing. The proposed crawler over cloud computing exploited the features of cloud computing such as scalability, reliability, and flexibility through a well-defined service based architecture. Moreover, the results highlighted the enhancement in performance of the cloud-based architecture against the grid-based and monolithic.

DOI

10.21608/mjcis.2019.320660

Keywords

Web Crawler, grid computing, Cloud Computing, Architecture, Grid-based Crawler, Cloud-based Crawler

Authors

First Name

M. E.

Last Name

ElAraby

MiddleName

-

Affiliation

CS Dept. Faculty of Computers and information, Mansoura, Beni-Suef University

Email

mohamed.elaraby@fcis.bsu.edu.eg

City

-

Orcid

-

First Name

Sherihan

Last Name

Mohamed

MiddleName

-

Affiliation

CS Dept. Faculty of Computers and information, Mansoura University

Email

-

City

-

Orcid

-

First Name

Hossam

Last Name

M. Moftah

MiddleName

-

Affiliation

CS Dept. Faculty of Computers and information, Beni-Suef University

Email

-

City

-

Orcid

-

First Name

M. Z.

Last Name

Rashad

MiddleName

-

Affiliation

CS Dept. Faculty of Computers and information, Mansoura University

Email

-

City

-

Orcid

-

Volume

15

Article Issue

1

Related Issue

43865

Issue Date

2019-06-01

Receive Date

2023-10-09

Publish Date

2019-06-01

Page Start

1

Page End

11

Print ISSN

2090-1666

Online ISSN

2090-1674

Link

https://mjcis.journals.ekb.eg/article_320660.html

Detail API

https://mjcis.journals.ekb.eg/service?article_code=320660

Order

320,660

Type

Original Research Articles.

Type Code

1,784

Publication Type

Journal

Publication Title

Mansoura Journal for Computer and Information Sciences

Publication Link

https://mjcis.journals.ekb.eg/

MainTitle

Web Crawler Architecture over Cloud Computing compared with Grid Computing

Details

Type

Article

Created At

28 Dec 2024