---
title: "Web scraping with Playwright in Python: JavaScript pages, waits and blocking"
category: "guides"
url: https://keirolabs.cloud/blog/playwright-scraping
---

## The 60-second version

**Playwright scrapes pages that build their content with JavaScript, by loading them in a real headless browser and reading the rendered page.** Use it when `requests` returns an empty shell; skip it when the data is already in the HTML, because a browser is far slower and heavier than an HTTP request.

- **Install:** `pip install playwright` then `playwright install chromium`.
- **The core loop:** `goto` → wait for the element you need → read it with `locator` or `page.content()`.
- **Most bugs are waits.** Wait for a selector, not for a fixed number of seconds.
- **Speed it up** by blocking images, fonts and media, and reusing one browser for many pages.
- **Blocked anyway?** Headless Chromium is detectable. See the last two sections.

## Your first Playwright scraper

```bash
pip install playwright
playwright install chromium
```

```python
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/", wait_until="domcontentloaded")
    page.wait_for_selector("div.quote")

    quotes = [
        {
            "text": q.locator("span.text").inner_text(),
            "author": q.locator("small.author").inner_text(),
        }
        for q in page.locator("div.quote").all()
    ]
    browser.close()

print(len(quotes), quotes[0])
```

That page is empty without JavaScript: `requests.get` on the same URL returns no quotes. Playwright runs the page's scripts, then reads the result.

## Waiting correctly

**Wait for the thing you need to exist, never for a number of seconds.** `time.sleep(5)` is too slow on fast pages and too short on slow ones.

| Wait | When |
|---|---|
| `page.wait_for_selector("div.results")` | The data appears in a known element |
| `page.locator("text=Load more").click()` | Locator actions auto-wait for the element to be visible and enabled |
| `page.wait_for_load_state("networkidle")` | No requests for 500 ms; useful, but slow on pages with analytics or polling |
| `page.expect_response("**/api/search*")` | The data arrives from a known API call |

The last one is often the best scraper of all. **If the page fetches JSON from its own API, capture that response instead of parsing the DOM:**

```python
with page.expect_response(lambda r: "/api/" in r.url and r.status == 200) as resp:
    page.goto("https://example.com/search?q=keyboards")
data = resp.value.json()
```

## Infinite scroll

```python
page.goto("https://quotes.toscrape.com/scroll")
seen = 0
while True:
    page.mouse.wheel(0, 20000)
    page.wait_for_timeout(800)  # give the next batch time to request
    count = page.locator("div.quote").count()
    if count == seen:
        break
    seen = count
print(seen)
```

Cap the loop on large feeds (stop at a count or a date), or you will scroll forever.

## Clicking, typing and logging in

```python
page.goto("https://quotes.toscrape.com/login")
page.fill("#username", "demo")
page.fill("#password", "demo")
page.click("input[type=submit]")
page.wait_for_selector("a[href='/logout']")

page.context.storage_state(path="state.json")   # save cookies for the next run
# next run: browser.new_context(storage_state="state.json")
```

Only automate logins to accounts you own, and check the site's terms first. Scraping behind a login is where most legal trouble starts.

## Make it faster: block what you don't need

**Images, fonts and media are most of a page's bytes and none of its data.**

```python
BLOCK = {"image", "font", "media"}

page.route("**/*", lambda route: route.abort()
           if route.request.resource_type in BLOCK else route.continue_())
```

Then **reuse one browser and open a new context per site** instead of launching Chromium for every URL. Launching takes around a second; a new context takes milliseconds.

## Many pages concurrently with async

```python
import asyncio
from playwright.async_api import async_playwright

async def scrape(context, url):
    page = await context.new_page()
    try:
        await page.goto(url, wait_until="domcontentloaded", timeout=30000)
        return {"url": url, "title": await page.title()}
    finally:
        await page.close()

async def main(urls, concurrency=5):
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        context = await browser.new_context()
        sem = asyncio.Semaphore(concurrency)
        async def run(u):
            async with sem:
                return await scrape(context, u)
        results = await asyncio.gather(*(run(u) for u in urls), return_exceptions=True)
        await browser.close()
        return results

urls = [f"https://quotes.toscrape.com/js/page/{i}/" for i in range(1, 11)]
print(asyncio.run(main(urls)))
```

Five to ten pages at a time per machine is a sensible start; each tab uses tens to hundreds of megabytes.

## Why Playwright still gets blocked

**Headless Chromium announces itself in small ways, and bot protection checks them.** `navigator.webdriver` is true, some browser APIs behave differently, and requests from a datacenter IP carry a poor reputation. In order of effort:

1. **Look like a normal browser:** set a realistic viewport, locale and timezone in `new_context(...)`, and a current user agent ([list](/blog/random-user-agent)).
2. **Use headed mode or the new headless mode** (`channel="chromium"` in recent versions), which is closer to real Chrome.
3. **Slow down.** Most blocks are rate limits. One request a second per site avoids most of them.
4. **Residential proxies** for sites that block datacenter IPs: `launch(proxy={"server": "http://host:port", "username": "...", "password": "..."})`.

## When a browser is the wrong tool

**Running browsers is the expensive part of scraping:** memory, crashes, version upgrades, proxies. If you need the text of JavaScript pages rather than to click through them, an API that renders for you is less code to own. [browserfoundry by Keirolabs](https://www.browserfoundry.cloud/?utm_source=keirolabs&utm_medium=blog&utm_campaign=playwright-scraping) `/extract` with `mode: "deep"` renders the page and returns clean markdown, or the fields from a JSON Schema:

```python
import os, requests

r = requests.post(
    "https://api.keirolabs.cloud/api/v2/extract",
    headers={"Authorization": f"Bearer {os.environ['KEIRO_API_KEY']}"},
    json={"url": "https://quotes.toscrape.com/js/", "mode": "deep"},
    timeout=90,
)
print(r.json()["results"][0]["content"][:400])
```

Keep Playwright for flows: logins, forms, multi-step navigation. Use the API for "give me what this page says." New accounts get 750 credits with no card. For the basics of fetching and parsing, start with [web scraping with Python](/blog/web-scraping-python).