Creating A Web Crawler In Java Ee September 15, 2014 Fran I am creating a web crawler using Java EE Technologies. I have created a crawler service which contains the result of the WebCrawler in term CrawlerElement Currently I am using JSOUP Library in order to do this. But it is not reliable I am attempting the connection three times and also timeout is 10seconds still It is unreliable. By unreliable I mean even if it can be accessed publicly, It can not be accessed by the crawler program. I know it could be due to robots.
Read full article from Creating A Web Crawler In Java Ee | Sugar World
No comments:
Post a Comment