Creating a Scrapy Project

Scrapy comes with an efficient command-line tool, also called the ‘Scrapy tool’. Commands are used for different purposes and, accept a different set of arguments, and options. To write the Spider code, we begin by creating, a Scrapy project, by executing the following command, at the terminal:

scrapy startproject gfg_spiderfollowlink

This should create a ‘gfg_spiderfollowlink’ folder in your current directory. It contains a ‘scrapy.cfg’, which is a configuration file, of the project. The folder structure is as shown below –

The folder contains items.py,middlerwares.py and other settings files, along with the ‘spiders’ folder. 

Keep the contents of the configuration files as they are currently.

In this article, we will use Scrapy, for scraping data, presenting on linked webpages, and, collecting the same. We will scrape data from the website ‘https://quotes.toscrape.com/’.