---
title: "Sitemap extractor: list every URL on a website (and build an llms.txt)"
category: "guides"
url: https://keirolabs.cloud/blog/sitemap-extractor
---

## The 60-second version

**A sitemap extractor finds a website's XML sitemaps and returns every URL in them, often with the date each page last changed.** It is the fastest way to list a whole site: one sitemap request replaces a crawl of thousands of links.

- **Where sitemaps live:** the `Sitemap:` lines in `/robots.txt` first, then common paths like `/sitemap.xml`, `/sitemap_index.xml` and `/wp-sitemap.xml`.
- **Watch for indexes:** big sites publish a sitemap index that points to more sitemaps, sometimes gzipped. One file holds at most 50,000 URLs ([sitemaps.org](https://www.sitemaps.org/protocol.html)).
- **Do it yourself:** 40 lines of standard-library Python, below.
- **Or one API call:** [browserfoundry by Keirolabs](https://www.browserfoundry.cloud/?utm_source=keirolabs&utm_medium=blog&utm_campaign=sitemap-extractor) `/sitemap` reads robots.txt, follows indexes and `.gz` files, filters by path and date, and costs 1 credit.
- **Bonus:** turn the URL list into an `llms.txt` file for AI agents.

## How to find a website's sitemap

**Check these in order; stop at the first that works.**

| Where | Example | Notes |
|---|---|---|
| `robots.txt` | `Sitemap: https://site.com/sitemap.xml` | The official place to declare sitemaps; may list several |
| `/sitemap.xml` | most sites | The common default |
| `/sitemap_index.xml` | WordPress with Yoast SEO | An index of per-type sitemaps |
| `/wp-sitemap.xml` | WordPress core (5.5+) | Built-in WordPress sitemap |
| `/sitemap-index.xml`, `/sitemap/sitemap.xml` | various | Less common |
| Google search | `site:example.com filetype:xml` | Last resort |

Two file types use the same `.xml` extension: a **urlset** (the page URLs, each with an optional `<lastmod>`) and a **sitemapindex** (links to other sitemaps). A good extractor follows the second into the first.

## A sitemap extractor in Python

**Standard library only: robots.txt discovery, indexes, gzip and lastmod.**

```python
import gzip, re, urllib.request
import xml.etree.ElementTree as ET

UA = {"User-Agent": "Mozilla/5.0 (compatible; sitemap-extractor)"}
NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"

def fetch(url: str) -> bytes | None:
    try:
        with urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=30) as r:
            body = r.read()
    except Exception:
        return None
    return gzip.decompress(body) if body[:2] == b"\x1f\x8b" else body

def find_sitemaps(site: str) -> list[str]:
    site = site.rstrip("/") if site.startswith("http") else f"https://{site}"
    robots = (fetch(f"{site}/robots.txt") or b"").decode("utf-8", "ignore")
    found = re.findall(r"(?im)^\s*sitemap:\s*(\S+)", robots)
    if found:
        return found
    return [u for u in (f"{site}/sitemap.xml", f"{site}/sitemap_index.xml", f"{site}/wp-sitemap.xml") if fetch(u)]

def extract(site: str, limit: int = 50_000) -> list[tuple[str, str | None]]:
    todo, seen, urls = find_sitemaps(site), set(), []
    while todo and len(urls) < limit:
        sm = todo.pop(0)
        if sm in seen:
            continue
        seen.add(sm)
        body = fetch(sm)
        if not body:
            continue
        root = ET.fromstring(body)
        if root.tag == f"{NS}sitemapindex":
            todo += [e.text.strip() for e in root.iter(f"{NS}loc")]
        else:
            for u in root.iter(f"{NS}url"):
                urls.append((u.findtext(f"{NS}loc", "").strip(), u.findtext(f"{NS}lastmod")))
    return urls[:limit]

pages = extract("keirolabs.cloud")
print(len(pages), pages[:3])
```

What this skips, and you may hit: sitemaps served as plain text (one URL per line), RSS and Atom feeds used as sitemaps, malformed XML, servers that block scripts, and sites that publish no sitemap at all (crawl links instead).

## Filter by section and date

**The two filters people actually use: one section of a site, and pages changed since a date.**

```python
from datetime import date

blog = [(u, m) for u, m in pages if "/blog/" in u]
recent = [(u, m) for u, m in pages if m and m[:10] >= date(2026, 10, 1).isoformat()]
```

Comparing `lastmod` between two runs is the cheapest way to monitor a competitor's site: new URLs are new pages, newer dates are edits.

## One API call instead

browserfoundry `/sitemap` does discovery, indexes, gzip and filtering server-side:

```bash
curl -X POST https://api.keirolabs.cloud/api/v2/sitemap \
  -H "Authorization: Bearer $KEIRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "linear.app", "path_prefix": "/changelog", "since": "2026-10-01"}'
```

A real response, October 27, 2026 (trimmed):

```json
{
  "site": "https://linear.app",
  "sitemaps": ["https://linear.app/sitemap.xml"],
  "source": "robots.txt",
  "urls": [
    { "url": "https://linear.app/changelog/2026-10-22-slack-and-github-loops", "lastmod": "2026-10-22T15:54:48Z" },
    { "url": "https://linear.app/changelog/2026-10-08-stacked-pull-requests", "lastmod": "2026-10-08T16:35:24Z" }
  ],
  "latency_ms": 659,
  "creditsCharged": 1
}
```

| Parameter | What it does |
|---|---|
| `url` | Domain or site URL |
| `path_prefix` | Only URLs whose path starts with this, e.g. `/blog` |
| `since` | Only URLs with a `lastmod` on or after this ISO date |
| `limit` | Max URLs, up to 50,000 |

1 credit per call, whatever the site size. Full reference in [the sitemap docs](https://docs.keirolabs.cloud/content/sitemap).

## Generate an llms.txt from a sitemap

**An `llms.txt` file is a markdown index of a site's important pages for AI agents** ([what llms.txt is](/blog/what-is-llms-txt)). The format: an H1 with the site name, a one-line summary in a blockquote, then sections of links with short descriptions. A sitemap gives you the links; the page titles and summaries come from the pages themselves.

```python
import os, requests

KEY = {"Authorization": f"Bearer {os.environ['KEIRO_API_KEY']}"}
API = "https://api.keirolabs.cloud/api/v2"

def llms_txt(site: str, name: str, summary: str, prefix: str = "/docs", n: int = 30) -> str:
    urls = requests.post(f"{API}/sitemap", headers=KEY, timeout=60,
                         json={"url": site, "path_prefix": prefix, "limit": n}).json()["urls"]
    lines = [f"# {name}", "", f"> {summary}", "", f"## {prefix.strip('/').title() or 'Pages'}", ""]
    for u in urls:
        page = requests.post(f"{API}/extract", headers=KEY, timeout=90, json={
            "url": u["url"],
            "json_schema": {"type": "object", "properties": {
                "title": {"type": "string"},
                "summary": {"type": "string", "description": "What this page covers, in one sentence"},
            }},
        }).json()
        data = (page.get("results") or [{}])[0].get("structuredData") or {}
        lines.append(f"- [{data.get('title') or u['url']}]({u['url']}): {data.get('summary', '')}".rstrip(": "))
    return "\n".join(lines) + "\n"

print(llms_txt("keirolabs.cloud", "Keirolabs", "Web search and extraction API for AI agents.", prefix="/blog"))
```

Cost: 1 credit for the sitemap plus 2 per page, so a 30-page `llms.txt` is 61 credits. Review the output before you publish it: an `llms.txt` is an editorial choice of your best pages, not a dump of every URL.

## Sitemap extractor FAQ, briefly

- **The site has no sitemap.** Crawl from the homepage instead, following same-domain links, or search `site:domain.com` to seed URLs.
- **The sitemap is huge.** Use `path_prefix` and `since` so you only fetch what changed.
- **URLs in the sitemap 404.** Sitemaps go stale. Check status codes before you rely on them.
- **Next step: the content.** Feed the URLs to `/extract` for clean markdown, as in [web scraping with Python](/blog/web-scraping-python).

New accounts get 750 credits with no card, enough for 750 sitemap calls.