powerful web spidering, scraping, creeping crawling framework

 Scrappy is an easy (and hopefully fun) way of scraping, spidering, and/or
 harvesting information from web pages, web services, and more. Scrappy is a
 feature rich, flexible, intelligent web automation tool.