Python Web Scraping for Beginners: Extract Data from Any Website

Web scraping is one of the most powerful skills in a Python developer's toolkit. It lets you extract data from any website automatically — product prices, job listings, news articles, research data, or any publicly available information. Instead of copying data manually, a Python script can do it in seconds.

This beginner-friendly tutorial will teach you how to build a web scraper from scratch using Python, BeautifulSoup, and the requests library. By the end, you will have working scripts that extract real data from websites and save it to CSV files. This is a key skill in your Python automation journey.

What Is Web Scraping?

Web scraping is the automated process of extracting data from websites. When you visit a website, your browser downloads HTML code and renders it visually. A web scraper does the same thing programmatically — it downloads the HTML and then parses it to extract specific pieces of information.

Common use cases include price monitoring and comparison across e-commerce sites, collecting job listings from multiple job boards, gathering research data for academic or business analysis, monitoring competitors' product offerings, and building datasets for machine learning projects.

Is Web Scraping Legal?

Web scraping is generally legal when you scrape publicly available data, respect the website's robots.txt file, do not overload the server with too many requests, follow the website's terms of service, and do not scrape personal or copyrighted data for commercial redistribution. Always check a website's robots.txt file (available at website.com/robots.txt) before scraping.

Setting Up Your Web Scraping Environment

You need two main libraries for basic web scraping:

pip install requests beautifulsoup4

The requests library handles downloading web pages (making HTTP requests), and BeautifulSoup4 handles parsing the HTML to extract the data you need.

For advanced scraping that requires JavaScript rendering, you will also need Selenium:

pip install selenium

Your First Web Scraper: Step by Step

Let us build a scraper that extracts article titles and links from a blog or news site. This demonstrates the fundamental pattern you will use in every scraping project.

Step 1: Fetch the Web Page

import requests
from bs4 import BeautifulSoup

url = "https://news.ycombinator.com/"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}

response = requests.get(url, headers=headers)

if response.status_code == 200:
    print("Page fetched successfully!")
    html_content = response.text
else:
    print(f"Failed to fetch page. Status: {response.status_code}")

The User-Agent header makes your request look like it comes from a regular browser rather than a script. Many websites block requests without a proper User-Agent.

Step 2: Parse the HTML

soup = BeautifulSoup(html_content, "html.parser")

# Find all article title links
titles = soup.find_all("span", class_="titleline")

for i, title in enumerate(titles[:10], 1):
    link = title.find("a")
    if link:
        print(f"{i}. {link.text}")
        print(f"   URL: {link.get('href')}")
        print()

Step 3: Save Data to CSV

import csv

def scrape_and_save(url, output_file):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
    }
    
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, "html.parser")
    
    titles = soup.find_all("span", class_="titleline")
    
    with open(output_file, "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["Title", "URL"])
        
        for title in titles:
            link = title.find("a")
            if link:
                writer.writerow([link.text, link.get("href", "")])
    
    print(f"Saved {len(titles)} articles to {output_file}")

scrape_and_save("https://news.ycombinator.com/", "articles.csv")

Understanding BeautifulSoup Selectors

BeautifulSoup provides several methods to find elements in HTML. Mastering these selectors is the key to effective web scraping.

# Find the first element matching a tag
first_h1 = soup.find("h1")

# Find all elements matching a tag
all_paragraphs = soup.find_all("p")

# Find by CSS class
items = soup.find_all("div", class_="product-card")

# Find by ID
header = soup.find("div", id="main-header")

# Find by attribute
links = soup.find_all("a", attrs={"data-type": "external"})

# CSS selector syntax
prices = soup.select("div.product span.price")

# Get text content
text = element.get_text(strip=True)

# Get attribute value
href = link.get("href")
src = image.get("src")

Building a Product Price Scraper

Here is a more practical example — a script that scrapes product information and tracks prices:

import requests
from bs4 import BeautifulSoup
import csv
from datetime import datetime

class ProductScraper:
    def __init__(self):
        self.session = requests.Session()
        self.session.headers.update({
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
        })
        self.products = []
    
    def scrape_page(self, url):
        try:
            response = self.session.get(url, timeout=10)
            response.raise_for_status()
            return BeautifulSoup(response.text, "html.parser")
        except requests.RequestException as e:
            print(f"Error fetching {url}: {e}")
            return None
    
    def extract_products(self, soup):
        if not soup:
            return []
        
        items = soup.find_all("div", class_="product-item")
        
        for item in items:
            name_el = item.find("h2", class_="product-name")
            price_el = item.find("span", class_="price")
            
            if name_el and price_el:
                self.products.append({
                    "name": name_el.get_text(strip=True),
                    "price": price_el.get_text(strip=True),
                    "scraped_at": datetime.now().isoformat()
                })
    
    def save_to_csv(self, filename):
        if not self.products:
            print("No products to save.")
            return
        
        with open(filename, "w", newline="", encoding="utf-8") as f:
            writer = csv.DictWriter(f, fieldnames=self.products[0].keys())
            writer.writeheader()
            writer.writerows(self.products)
        
        print(f"Saved {len(self.products)} products to {filename}")

scraper = ProductScraper()
soup = scraper.scrape_page("https://example-store.com/products")
scraper.extract_products(soup)
scraper.save_to_csv("products.csv")

Handling Pagination

Most websites split their content across multiple pages. Here is how to scrape across all pages:

def scrape_all_pages(base_url, max_pages=10):
    all_data = []
    
    for page in range(1, max_pages + 1):
        url = f"{base_url}?page={page}"
        print(f"Scraping page {page}...")
        
        response = requests.get(url, headers=headers)
        soup = BeautifulSoup(response.text, "html.parser")
        
        items = soup.find_all("div", class_="item")
        
        if not items:
            print(f"No items on page {page}. Stopping.")
            break
        
        for item in items:
            all_data.append(extract_item_data(item))
        
        import time
        time.sleep(2)  # Be polite, wait between requests
    
    return all_data

When You Need Selenium for JavaScript-Heavy Sites

Some modern websites load their content dynamically using JavaScript. The requests library only downloads the initial HTML, so JavaScript-rendered content will be missing. Selenium solves this by controlling a real browser:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()

try:
    driver.get("https://example.com/dynamic-page")
    
    # Wait for content to load
    wait = WebDriverWait(driver, 10)
    elements = wait.until(
        EC.presence_of_all_elements_located(
            (By.CLASS_NAME, "product-card")
        )
    )
    
    for element in elements:
        name = element.find_element(By.CLASS_NAME, "name").text
        price = element.find_element(By.CLASS_NAME, "price").text
        print(f"{name}: {price}")

finally:
    driver.quit()

Best Practices for Web Scraping

Add delays between requests. Use time.sleep() between requests to avoid overloading the server. A 1-3 second delay is usually appropriate.

Use sessions for efficiency. The requests.Session() object reuses TCP connections and handles cookies automatically, making your scraper faster and more reliable.

Handle errors gracefully. Websites change their structure frequently. Always wrap your parsing code in try-except blocks and validate that elements exist before accessing them.

Respect robots.txt. Check the website's robots.txt file to understand which pages you are allowed to scrape.

Cache responses during development. While building and testing your scraper, save the HTML locally so you do not make repeated requests to the server.

Combining Web Scraping with Other Automation

Web scraping becomes even more powerful when combined with other Python automation skills. You can set up a scraper that monitors prices and sends you an email notification when a price drops. Or you can schedule your scraper to run daily and save the results to a spreadsheet. You can also organize the downloaded data files automatically.

Conclusion

Web scraping with Python opens up a world of data that was previously only accessible through manual effort. Start with the basic requests and BeautifulSoup approach for static websites, and move to Selenium when you encounter JavaScript-heavy sites. Always scrape responsibly and respect website terms of service.

To continue building your Python automation skills, return to our complete Python Automation Guide for more projects and learning paths.