---
title: "Parse HTML in Python: BeautifulSoup vs lxml vs selectolax (benchmarked)"
category: "guides"
url: https://keirolabs.cloud/blog/parse-html-python
---

## The 60-second version

**To parse HTML in Python, use BeautifulSoup when you want the friendliest API, lxml when you want XPath and speed, and selectolax when you parse a lot of pages and speed is all that matters.** All three take an HTML string and let you select elements with CSS selectors.

We timed all of them on the same 996 KB page (Wikipedia's Python article), selecting all 2,240 links, averaged over 20 runs (Python 3.12, October 27, 2026):

| Parser | Time per page | vs BeautifulSoup |
|---|---|---|
| BeautifulSoup + `html.parser` | 357.5 ms | 1x |
| BeautifulSoup + `lxml` | 231.8 ms | 1.5x faster |
| lxml (XPath or CSS) | 26.7 ms | 13x faster |
| selectolax (lexbor) | 8.8 ms | 40x faster |

At 100,000 pages, that is the difference between ten hours of parsing and fifteen minutes.

## BeautifulSoup: the default choice

```bash
pip install beautifulsoup4 lxml
```

```python
from bs4 import BeautifulSoup

html = """
<div class="product" data-sku="A1">
  <h2>Mechanical keyboard</h2>
  <span class="price">$89.00</span>
  <a href="/p/a1">Details</a>
</div>
"""

soup = BeautifulSoup(html, "lxml")          # "html.parser" works with no extra install
card = soup.select_one("div.product")

print(card["data-sku"])                       # A1
print(card.h2.get_text(strip=True))           # Mechanical keyboard
print(card.select_one(".price").get_text())   # $89.00
print([a["href"] for a in soup.select("a[href]")])
```

**Why people pick it:** it forgives broken HTML, the API reads like English (`find`, `find_all`, `select`, `get_text`), and every tutorial on the internet uses it. **Why people leave it:** it is the slowest option by an order of magnitude.

| Task | BeautifulSoup |
|---|---|
| First match by CSS | `soup.select_one("div.price")` |
| All matches | `soup.select("a[href]")` |
| By tag and attribute | `soup.find("a", attrs={"rel": "next"})` |
| Text, trimmed | `el.get_text(strip=True)` |
| Attribute | `el.get("href")` |
| Parent / next sibling | `el.parent`, `el.find_next_sibling()` |

## lxml: XPath and 13x the speed

```bash
pip install lxml cssselect
```

```python
import lxml.html

tree = lxml.html.fromstring(html)

print(tree.xpath("//div[@class='product']/@data-sku")[0])         # A1
print(tree.xpath("string(//h2)").strip())                          # Mechanical keyboard
print(tree.cssselect("span.price")[0].text_content())              # $89.00
print(tree.xpath("//a[contains(text(), 'Details')]/@href"))         # ['/p/a1']
```

**XPath is the reason to choose lxml.** It can select by text content (`contains(text(), 'Details')`), walk up to parents (`..`), and pull attributes directly (`/@href`), which CSS selectors can't. Use `tree.make_links_absolute(base_url)` to turn every relative link into a full URL in one call.

## selectolax: the fastest

```bash
pip install selectolax
```

```python
from selectolax.lexbor import LexborHTMLParser

tree = LexborHTMLParser(html)
card = tree.css_first("div.product")

print(card.attributes["data-sku"])                 # A1
print(card.css_first("h2").text(strip=True))       # Mechanical keyboard
print([a.attributes["href"] for a in tree.css("a[href]")])
```

**Heads up if you're upgrading:** selectolax 1.0 removed the old Modest backend. `from selectolax.parser import HTMLParser` now raises an `ImportError`; swap it for `from selectolax.lexbor import LexborHTMLParser`. The rest of the API is nearly identical.

selectolax supports CSS selectors only, no XPath, and its API is smaller. For a crawler that parses millions of pages, that trade is worth 40x.

## Which parser should you use?

| You are... | Use |
|---|---|
| Learning, or scraping a few hundred pages | BeautifulSoup with the `lxml` backend |
| Writing selectors that depend on text or parents | lxml with XPath |
| Parsing tens of thousands of pages or more | selectolax |
| Parsing XML or RSS feeds | lxml (`lxml.etree`) |
| Getting page text for an LLM | None of them; see below |

## Five parsing mistakes

1. **Using regex on HTML.** It works until an attribute order or a line break changes. Use a parser.
2. **Selecting by generated class names.** Classes like `css-1x9k2d` change on every deploy. Prefer `id`, `data-*` attributes, `itemprop` or the page structure.
3. **Forgetting the page needs JavaScript.** If the element is in DevTools but not in your HTML string, the page renders client-side. Parse the HTML after a browser renders it ([Playwright guide](/blog/playwright-scraping)).
4. **Ignoring encoding.** With `requests`, pass `r.content` (bytes) to the parser instead of `r.text` when non-English text comes out garbled, so the parser reads the page's own charset declaration.
5. **Missing JSON in the page.** Many sites ship their data in `<script type="application/ld+json">` or `<script id="__NEXT_DATA__">`. One `json.loads` of that script beats twenty selectors.

```python
import json
data = json.loads(soup.select_one("script#__NEXT_DATA__").string)
```

## When you don't need a parser at all

**If you want the page's text for an LLM or a search index, writing selectors is the wrong job.** You want the main content with navigation, ads and footers removed, as markdown. [browserfoundry by Keirolabs](https://www.browserfoundry.cloud/?utm_source=keirolabs&utm_medium=blog&utm_campaign=parse-html-python) `/extract` does that in one call and can also return specific fields from a JSON Schema:

```python
import os, requests

r = requests.post(
    "https://api.keirolabs.cloud/api/v2/extract",
    headers={"Authorization": f"Bearer {os.environ['KEIRO_API_KEY']}"},
    json={"url": "https://en.wikipedia.org/wiki/Python_(programming_language)"},
    timeout=60,
)
print(r.json()["results"][0]["content"][:500])
```

2 credits a page, about $0.005 on the Essential plan; new accounts get 750 credits with no card. For the full fetch, parse and save workflow, read [web scraping with Python](/blog/web-scraping-python).