This rather long (counting the Python code) tutorial assumes that you are interested in scraping data off the web using Scrapy. It might also be limited to Linux.
Installing scrapy
Here is how to install a Python package using pip:
pip install scrapyCode language: Bash (bash)Note, if you need to you can also use pip to install a specific version of Scrapy as well as upgrade pip, if you have an old version. Now, to create a new Scrapy project, open up a terminal and create a directory where you will store your Scrapy projects change do that directory and run this (change ‘psysci’ to whatever your project will be named):
scrapy startproject psysciCode language: Bash (bash)Item
We start with the Item class and create an object to store our scraped data in. In my example I scrape the article title, type of article, the article’s abstract, when the article was received and when it got accepted for publication, which year it was publicized (and month) and the articles keywords.