---
title: "Web scraping with Python in 2026: a step-by-step guide"
category: "guides"
url: https://keirolabs.cloud/blog/web-scraping-python
---

## The 60-second version

**Web scraping with Python is three steps: fetch the page, parse the HTML, save what you need.** For most sites, `requests` plus `BeautifulSoup` does all three in about 15 lines. Add Playwright when the page builds itself with JavaScript, and a crawler loop when you need more than one page.

- **Static pages** (the HTML already contains the data): `requests` + `BeautifulSoup`. Start here.
- **JavaScript pages** (the HTML is an empty shell): Playwright renders the page in a real browser first.
- **Many pages:** read the site's sitemap for the URL list, then fetch each one politely.
- **Blocked or messy pages:** a scraping API fetches, renders and cleans the page for you and returns markdown or JSON.

Every code sample below runs on Python 3.10 or newer.

## Set up

```bash
python -m venv .venv && source .venv/bin/activate
pip install requests beautifulsoup4 lxml
```

`lxml` is optional, but BeautifulSoup parses several times faster with it than with the built-in `html.parser`.

## Step 1: fetch the page with requests

```python
import requests

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

r = requests.get("https://books.toscrape.com/", headers=HEADERS, timeout=30)
r.raise_for_status()
html = r.content  # bytes, so the parser reads the page's own charset
```

Four habits that save hours later:

1. **Always set a timeout.** Without one, a hung server hangs your script forever.
2. **Send a real browser User-Agent.** The default `python-requests/2.x` is blocked by many sites on sight. Current strings are in [random user agent strings](/blog/random-user-agent).
3. **Call `raise_for_status()`.** A 403 or 429 page parses fine and returns nothing, which looks like a bug in your selectors.
4. **Parse `r.content`, not `r.text`.** `requests` guesses the encoding from headers and often guesses wrong, which turns `£51.77` into `£51.77`. Bytes let the parser read the charset the page declares.

## Step 2: parse the HTML with BeautifulSoup

**Find the element in your browser's DevTools (right click, Inspect), then select it with the same CSS selector.**

```python
from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")

books = []
for card in soup.select("article.product_pod"):
    books.append({
        "title": card.h3.a["title"],
        "price": card.select_one("p.price_color").get_text(strip=True),
        "in_stock": "In stock" in card.select_one("p.availability").get_text(),
        "url": requests.compat.urljoin(r.url, card.h3.a["href"]),
    })

print(len(books), books[0])
```

| Task | BeautifulSoup |
|---|---|
| First match | `soup.select_one("css")` |
| All matches | `soup.select("css")` |
| Text | `el.get_text(strip=True)` |
| Attribute | `el["href"]` or `el.get("href")` |
| Relative to absolute URL | `urljoin(page_url, href)` |

If BeautifulSoup is too slow for your volume, [lxml is 13x faster and selectolax 40x faster in our benchmark](/blog/parse-html-python).

## Step 3: save the results

```python
import csv

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=books[0].keys())
    w.writeheader()
    w.writerows(books)
```

For anything you will query later, write to SQLite instead (`sqlite3` is in the standard library), so a rerun can skip pages you already have.

## Step 4: scrape many pages

**Follow the "next" link until there isn't one, and pause between requests.**

```python
import time

url, rows = "https://books.toscrape.com/", []
with requests.Session() as s:
    s.headers.update(HEADERS)
    while url:
        r = s.get(url, timeout=30)
        r.raise_for_status()
        soup = BeautifulSoup(r.content, "lxml")
        rows += [a["title"] for a in soup.select("article.product_pod h3 a")]
        nxt = soup.select_one("li.next a")
        url = requests.compat.urljoin(r.url, nxt["href"]) if nxt else None
        time.sleep(1)  # be polite: about one request a second per site

print(len(rows))
```

A `Session` reuses the TCP connection, which makes multi-page scrapes noticeably faster. For whole sites, don't crawl link by link: most sites publish a sitemap listing every URL, and reading it is one request ([how to find and read a sitemap](/blog/sitemap-extractor)).

## Step 5: pages that need JavaScript

**If the data is visible in the browser but missing from `r.text`, the page renders with JavaScript.** Check first whether the page loads its data from a JSON endpoint (DevTools, Network tab, filter by Fetch/XHR). Calling that endpoint directly with `requests` is faster than any browser.

If there is no clean endpoint, render the page with Playwright:

```python
# pip install playwright && playwright install chromium
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/")
    page.wait_for_selector("div.quote")
    html = page.content()
    browser.close()

soup = BeautifulSoup(html, "lxml")
print([q.get_text(strip=True) for q in soup.select("span.text")][:3])
```

Waits, infinite scroll, logins and blocking are covered in [web scraping with Playwright in Python](/blog/playwright-scraping).

## Why scrapers break, and what to do about it

| Symptom | Likely cause | Fix |
|---|---|---|
| 403 Forbidden | Bot detection on headers or TLS fingerprint | Real browser headers; `curl_cffi` to impersonate Chrome's TLS; or a browser |
| 429 Too Many Requests | Rate limit | Slow down, back off exponentially, respect `Retry-After` |
| Selectors return nothing | JavaScript rendering, or the markup changed | Check `r.text`; use Playwright; prefer stable attributes over class names |
| Works locally, fails on a server | Datacenter IP reputation | Residential proxies, or a scraping API |
| Cloudflare "Just a moment..." page | JavaScript challenge | A real browser, or a scraping API that handles it |

**Read the site's terms and `robots.txt` before you scrape.** Public data is generally fair to collect, but logins, personal data and copyrighted content carry legal risk, and hammering a small site can take it down.

## When to stop writing parsers

**Selectors are the part of a scraper that breaks.** Every redesign of the target site costs you an afternoon. If what you actually want is the page's text for an LLM, or a few fields from many different sites, a scraping API skips the parsing step.

[browserfoundry by Keirolabs](https://www.browserfoundry.cloud/?utm_source=keirolabs&utm_medium=blog&utm_campaign=web-scraping-python) `/extract` takes a URL and returns clean markdown, rendering JavaScript pages when needed. Pass a JSON Schema and it returns the fields instead:

```python
import os, requests

r = requests.post(
    "https://api.keirolabs.cloud/api/v2/extract",
    headers={"Authorization": f"Bearer {os.environ['KEIRO_API_KEY']}"},
    json={
        "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
        "json_schema": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "price": {"type": "string"},
                "in_stock": {"type": "boolean"},
                "description": {"type": "string"},
            },
        },
    },
    timeout=60,
)
print(r.json())
```

One extract costs 2 credits, about $0.005 on the Essential plan, and new accounts get 750 credits with no card. Write your own scraper when you control the volume and the target; use the API when you scrape many different sites or you're tired of fixing selectors. Comparisons of the main scraping APIs are in [the best web scraping APIs](/blog/best-web-scraping-apis-2026).