How to use LXML In Python
Step-by-step Approach
- We will use requests.get to retrieve the web page with our data.
- We use html.fromstring to parse the content using the lxml parser.
- We create the correct XPath query and use the lxml xpath function to get the required element.
Example 1:
Below is a program based on the above approach which uses a particular URL.
Python
# Import required modulesfrom lxml import htmlimport requests# Request the pagepage = requests.get('http://econpy.pythonanywhere.com/ex/001.html')# Parsing the page# (We need to use page.content rather than # page.text because html.fromstring implicitly# expects bytes as input.)tree = html.fromstring(page.content) # Get element using XPathbuyers = tree.xpath('//div[@title="buyer-name"]/text()')print(buyers) |
Output:
Example 2:
Another example for an E-commerce website, URL.
Python
# Import required modulesfrom lxml import htmlimport requests# Request the pagepage = requests.get('https://webscraper.io/test-sites/e-commerce/allinone')# Parsing the pagetree = html.fromstring(page.content)# Get element using XPathprices = tree.xpath( '//div[@class="col-sm-4 col-lg-4 col-md-4"]/div/div[1]/h4[1]/text()')print(prices) |
Output:
Web Scraping using lxml and XPath in Python
Prerequisites: Introduction to Web Scraping
In this article, we will discuss the lxml python library to scrape data from a webpage, which is built on top of the libxml2 XML parsing library written in C. When compared to other python web scraping libraries like BeautifulSoup and Selenium, the lxml package gives an advantage in terms of performance. Reading and writing large XML files takes an indiscernible amount of time, making data processing easier & much faster.
We will be using the lxml library for Web Scraping and the requests library for making HTTP requests in Python. These can be installed in the command line using the pip package installer for Python.
Getting data from an element on the webpage using lxml requires the usage of Xpaths.
Contact Us