Prospective Term Based Incremental Migrating Crawler

Abstract

With majority of people using Internet for daily information, there is a need to develop efficient search engines that can provide accurate , precise and timely information to the users. In this paper a novel architecture for Design of an Incremental Migrating Crawler Using Prospective Terms based Ranking has been proposed, developed and implemented. The proposed crawler design focuses on user interest and choice for prioritizing URLs for downloading documents. To effectively utilize network bandwidth migrating mobile agents called Migrants are dispatched to download data in the proximity of the data itself. Since web sites store data on multiple sites therefore factors like network-load, trust, Server load and Network distance etc are being used to select the best server to dispatch the migrant so that fast crawling of documents may be achieved. Furthermore the work has also put forward the amalgamation of web content, structure and web usage mining techniques along with user preferences for ranking web pages so that the user gets desired results in less number of clicks. The proposed architecture has been compared with Scalable Migrating crawler in terms of efficiency, crawling time, network load etc and it has been observed that proposed crawler provides better results in terms of load distribution, reducing network load and bandwidth utilization, elimination of duplicate documents and retrieving user relevant pages.