Extracting Data from one Webpage

The code for web scraping is written in the spider code file. To create the spider file, we will make use of the ‘genspider’ command. Please note, that this command is executed at the same level where scrapy.cfg file is present. 

We are scraping all quotes present, on ‘http://quotes.toscrape.com/’. Hence, we will run the command as:

scrapy genspider gfg_spilink "quotes.toscrape.com"

The above command will create a spider file, “gfg_spilink.py” in the ‘spiders’ folder. The default code, for the same, is as follows:

We will scrape all Quotes Title, Authors, and Tags from the website “quotes.toscrape.com”. The website landing page looks as shown below:

Scrapy provides us, with Selectors, to “select” parts of the webpage, desired. Selectors are CSS or XPath expressions, written to extract data from HTML documents. In this tutorial, we will make use of XPath expressions, to select the details we need. 

Let us understand the steps for writing the selector syntax in the  spider code:

• Firstly, we will write the code in the parse() method. This is the default callback method, present in the spider class, responsible for processing the response received. The data extraction code, using Selectors, will be written here.
• For writing the XPath expressions, we will select the element on the webpage, say Right-Click, and choose the Inspect option. This will allow us to view its CSS attributes.
• When we right-click on the first Quote and choose Inspect, we can see it has the CSS ‘class’ attribute “quote”. Similarly, all the other quotes on the webpage have the same CSS ‘class’ attribute. It can be seen below:

Hence, the XPath expression, for the same, can be written as – quotes = response.xpath(‘//*[@class=”quote”]’). This syntax will fetch all elements, having “quote”, as the CSS ‘class’ attribute. The quotes present on further pages have the same CSS attribute. For example, the quotes present on Page 3, of the website, belong to the  ‘class’ attribute, as shown below –

We need to fetch the Quote Title, Author, and Tags of all the Quotes. Hence, we will write XPath expressions for extracting them, in a loop. 

• The CSS ‘class’ attribute, for Quote Title, is “text”. Hence, the XPath expression, for the same, would be – quote.xpath(‘.//*[@class=”text”]/text()’).extract_first(). The text() method, will extract the text, of the Quote title. The extract_first() method, will give the first matching value, with the CSS attribute “text”. The dot operator ‘.’ in the start, indicates extracting data, from a single quote.
• The CSS  attributes, “class” and “itemprop”, for author element, is “author”. We can use, any of these, in the XPath expression. The syntax would be – quote.xpath(‘.//*[@itemprop=”author”]/text()’).extract(). This will extract, the Author name, where the CSS ‘itemprop’ attribute is ‘author’.
• The CSS  attributes, “class” and “itemprop”, for tags element, is “keywords”. We can use, any of these, in the XPath expression. Since there are many tags, for any quote, looping through them, will be tedious. Hence, we will extract the CSS attribute “content”, from every quote. The XPath expression for the same is – quote.xpath(‘.//*[@itemprop=”keywords”]/@content’).extract(). This will extract, all tags values, from “content” attribute, for quotes.
• We use ‘yield’ syntax to get the data. We can collect, and, transfer data to CSV, JSON, and other file formats, by using ‘yield’.

If we observe the code till here, it will crawl and extract data for one webpage. The code is as follows –

In this article, we will use Scrapy, for scraping data, presenting on linked webpages, and, collecting the same. We will scrape data from the website ‘https://quotes.toscrape.com/’.