User

Recommend this pageIf this page was useful to you, please recommend it to fellow websurfers:

Home > Scripts > External Content / Webservices

Rating
0 vote You may rate this script by simply clicking on the appropriate star (5 stars is the best rating).
Language
	Python
License

Website Live Preview Download

Scrapy

Requirements

Python 2.7
pip and setuptools Python packages. Nowadays pip requires and installs setuptools if not installed. Python 2.7.9 and later include pip by default, so you may have it already.
lxml. Most Linux distributions ships prepackaged versions of lxml. Otherwise refer to http://lxml.de/installation.html
OpenSSL. This comes preinstalled in all operating systems, except Windows where the Python installer ships it bundled.

Features

Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.

Even though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler.

What else?

Built-in support for selecting and extracting data from HTML/XML sources using extended CSS selectors and XPath expressions, with helper methods to extract using regular expressions.
An interactive shell console (IPython aware) for trying out the CSS and XPath expressions to scrape data, very useful when writing or debugging your spiders.
Built-in support for generating feed exports in multiple formats (JSON, CSV, XML) and storing them in multiple backends (FTP, S3, local filesystem)
Robust encoding support and auto-detection, for dealing with foreign, non-standard and broken encoding declarations.
Strong extensibility support, allowing you to plug in your own functionality using signals and a well-defined API (middlewares, extensions, and pipelines).
Wide range of built-in extensions and middlewares for handling:
- cookies and session handling
- HTTP features like compression, authentication, caching
- user-agent spoofing
- robots.txt
- crawl depth restriction
- and more
A Telnet console for hooking into a Python console running inside your Scrapy process, to introspect and debug your crawler
Plus other goodies like reusable spiders to crawl sites from Sitemaps and XML/CSV feeds, a media pipeline for automatically downloading images (or any other media) associated with the scraped items, a caching DNS resolver, and much more!

Comments

Webmaster’s Rating

Tested & Working

This label indicates that I could install the script without major problems and I could run the script without the need for major modifications to the original code (except for the usual configuration, of course). The label does, on the other hand, not indicate that scripts without this label would not work properly. The missing of the „tested & working“-label just indicates that I did not yet test these scripts.

Webmaster’s Choice

This label indicates that the owner of this website uses this script / application for his own projects with success and satisfaction. This does, however, not indicate that these projects could not have been realized using other scripts / applications as well or that other scripts would not fit the demands of other projects as well or even better.