How to Scrape Website Data with Python’s Beautiful Soup Library

Kaggle is an excellent resource for datasets to be used for data analysis and machine learning. However, if the data you need has not already been collated, scraping website data is easy to do! Please note: make sure to read the website’s rules about the legal use of their data.

My example will be extracting the names of dogs from the dog breeds wikipedia article.

Step 1: Install Beautiful Soup

pip install beautifulsoup4

Step 2: The script is as follows (I am using Python 3.7.0)


#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import os

#import the Beautiful soup module
from bs4 import BeautifulSoup

#Import the request attribute of the urllib module
import urllib.request

#Add URL of website to be scraped
quote_page = 'https://en.wikipedia.org/wiki/List_of_dog_breeds'

#Query the website and store the code in the variable page
page = urllib.request.urlopen(quote_page)

#Parse the HTML code using Beautiful Soup and store it as the variable soup
soup = BeautifulSoup(page, 'html.parser')

#Assign all rows to the rows variable, alternatively could use: 
#rows = soup.find('table').findAll('tr')
rows = soup.table.findAll('tr')

#Declare the variable f to open a CSV file called dogs
#w represents write, the kind of permission/operation to be done on the file
#set encoding to UTF to avoid errors for special characters
f= open('dogs.csv','w+', encoding='utf-8')

#Create a for loop to iterate through the 2nd <td> within each row <tr>
#Except for the 1st and last row (since those two do not contain a <td> tag)
#.text removes all tags
#Write results to dogs.csv with new line after each result
for row in rows[1:-1]:
    dog = row.findAll('td')[1].text
    f.write(str(dog) + '\n')

#Close the instance of the file dogs.csv stored
f.close()

Step 3: Run the script. Each dog name will be written to each row of the dogs.csv file.

Leave a Reply

Your email address will not be published. Required fields are marked *