---
title: "Web scraping with JavaScript and Node.js in 2026"
category: "guides"
url: https://keirolabs.cloud/blog/web-scraping-javascript
---

## The 60-second version

**Web scraping with JavaScript in Node.js takes two packages: the built-in `fetch` to download the page and Cheerio to parse it.** Add Playwright when the page renders with JavaScript. Every example below runs on Node 20 or newer as an ES module (`"type": "module"` in `package.json`, or a `.mjs` file).

- **Static pages:** `fetch` + `cheerio`. Fast, no browser, about 15 lines.
- **JavaScript-rendered pages:** `playwright` loads the page in headless Chromium, then you read the DOM.
- **Many pages:** a small concurrency-limited loop, or the site's sitemap for the URL list.
- **Can you scrape in the browser itself?** Only pages on your own origin. CORS blocks cross-site `fetch` from a web page, so scrapers run in Node.

## Set up

```bash
mkdir scraper && cd scraper
npm init -y && npm pkg set type=module
npm install cheerio
```

## Step 1: fetch the HTML

```js
const HEADERS = {
  "User-Agent":
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
  "Accept-Language": "en-US,en;q=0.9",
};

const res = await fetch("https://books.toscrape.com/", {
  headers: HEADERS,
  signal: AbortSignal.timeout(30_000),
});
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const html = await res.text();
```

`fetch` is global in Node 18+, so there is nothing to install. `AbortSignal.timeout` stops a hung request; without it a stuck server stalls your script. Current browser strings are in [random user agent strings](/blog/random-user-agent).

## Step 2: parse it with Cheerio

**Cheerio gives you jQuery-style selectors over an HTML string, without a browser.**

```js
import * as cheerio from "cheerio";

const $ = cheerio.load(html);

const books = $("article.product_pod")
  .map((_, el) => ({
    title: $(el).find("h3 a").attr("title"),
    price: $(el).find("p.price_color").text().trim(),
    inStock: $(el).find("p.availability").text().includes("In stock"),
    url: new URL($(el).find("h3 a").attr("href"), res.url).href,
  }))
  .get();

console.log(books.length, books[0]);
```

| Task | Cheerio |
|---|---|
| All matches | `$("css")` |
| Inside an element | `$(el).find("css")` |
| Text | `.text().trim()` |
| Attribute | `.attr("href")` |
| To a plain array | `.map((i, el) => ...).get()` |
| Relative to absolute URL | `new URL(href, pageUrl).href` |

## Step 3: save the results

```js
import { writeFile } from "node:fs/promises";

await writeFile("books.json", JSON.stringify(books, null, 2));
```

## Step 4: many pages, politely

**Follow the "next" link, and wait between requests.**

```js
import * as cheerio from "cheerio";
import { setTimeout as sleep } from "node:timers/promises";

let url = "https://books.toscrape.com/";
const titles = [];

while (url) {
  const res = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(30_000) });
  if (!res.ok) throw new Error(`HTTP ${res.status} on ${url}`);
  const $ = cheerio.load(await res.text());
  titles.push(...$("article.product_pod h3 a").map((_, a) => $(a).attr("title")).get());
  const next = $("li.next a").attr("href");
  url = next ? new URL(next, res.url).href : null;
  await sleep(1000); // about one request a second per site
}

console.log(titles.length);
```

When you already have a list of URLs, run a few at a time instead of all at once. A tiny worker pool keeps you at a fixed concurrency without extra packages:

```js
async function pool(urls, size, fn) {
  const out = [];
  let i = 0;
  await Promise.all(Array.from({ length: size }, async () => {
    while (i < urls.length) {
      const u = urls[i++];
      out.push(await fn(u).catch((e) => ({ url: u, error: e.message })));
    }
  }));
  return out;
}
```

For a whole site, start from its sitemap rather than crawling links ([sitemap extractor](/blog/sitemap-extractor)).

## Step 5: JavaScript-rendered pages with Playwright

**If the data shows in your browser but not in `html`, the page builds itself client-side.** First look in DevTools (Network tab, Fetch/XHR) for the JSON request the page makes; calling it directly is faster than any browser. Otherwise render it:

```bash
npm install playwright && npx playwright install chromium
```

```js
import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto("https://quotes.toscrape.com/js/");
await page.waitForSelector("div.quote");

const quotes = await page.$$eval("div.quote", (els) =>
  els.map((el) => ({
    text: el.querySelector("span.text")?.textContent,
    author: el.querySelector("small.author")?.textContent,
  })),
);

console.log(quotes.slice(0, 3));
await browser.close();
```

**Puppeteer or Playwright?** Both drive headless Chromium with almost the same API. Playwright adds Firefox and WebKit, auto-waiting on actions, and a Python port, which is why most new scrapers start with it. More on waits, scrolling and blocking in [the Playwright scraping guide](/blog/playwright-scraping).

## Common errors

| Error | Cause | Fix |
|---|---|---|
| `403` | Bot detection: headers, TLS fingerprint or IP | Browser headers; Playwright; a scraping API |
| `429` | Rate limited | Lower concurrency, back off, honor `Retry-After` |
| Empty Cheerio result | JavaScript rendering, or changed markup | Check the raw HTML; use Playwright |
| `TypeError: fetch failed` | DNS, TLS or a reset connection | Retry with backoff; log `err.cause` |
| `ERR_REQUIRE_ESM` | `require()` of an ES module | Use `import` with `"type": "module"` |

Check the site's terms and `robots.txt` first, and stay away from logins and personal data.

## Skip the parser

**If you scrape many different sites, or you want page text for an LLM, selectors are maintenance you don't need.** [browserfoundry by Keirolabs](https://www.browserfoundry.cloud/?utm_source=keirolabs&utm_medium=blog&utm_campaign=web-scraping-javascript) `/extract` returns clean markdown for a URL, renders JavaScript pages when needed, and returns specific fields when you pass a JSON Schema:

```js
const r = await fetch("https://api.keirolabs.cloud/api/v2/extract", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.KEIRO_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://quotes.toscrape.com/js/",
    json_schema: {
      type: "object",
      properties: {
        quotes: {
          type: "array",
          items: { type: "object", properties: { text: { type: "string" }, author: { type: "string" } } },
        },
      },
    },
  }),
});
console.log(await r.json());
```

2 credits a page, about $0.005 on the Essential plan, with 750 free credits at signup and no card. The Python version of this guide is [web scraping with Python](/blog/web-scraping-python).