Web scraping with Python

Web scraping is basically the process of automatically mining data or collecting information from the World Wide Web. Web scraping software may access the World Wide Web directly using the Hypertext Transfer Protocol, or through the use of a web browser. While web scraping can be done manually by a software user, the term” Web scraping” typically refers to automated processes implemented using a bot or web crawler. A small script in Python is normally adequate.

Web scraping with JavaScript is done with the libraries cheerio and Request. however, we will be using this Python code here

Python Example

Library used: Requests, lxml, BeautifulSoup
install beautifulsoup4, requests, lxml

1. import requests
2. from bs4 import BeautifulSoup
3. url = 'http://BBC.com'
4. url_get = requests.get(url)
5. soup = BeautifulSoup(url_get.content, 'html.parser')
6. print(soup.find('title').text)

Output

BBC - Breaking News, U.K., World, Weather, Entertainment & Video News