Crawler4j is an open source Java crawler which provides a simple interface for crawling the Web. You can setup a multi-threaded web crawler in 5 minutes! Sample Usage You need to create a crawler class that extends WebCrawler . This class decides which URLs should be crawled and handles the downloaded page. The following is a sample implementation: public class MyCrawler extends WebCrawler { private final static Pattern FILTERS = Pattern.compile(".*(\\.(css|js|bmp|gif|jpe?g" + "|png|tiff?|mid|mp2|mp3|mp4" + "|wav|avi|mov|mpeg|ram|m4v|pdf" + "|rm|smil|wmv|swf|wma|zip|rar|gz))$");
Read full article from crawler4j - Open Source Web Crawler for Java - Google Project Hosting
No comments:
Post a Comment