Web Scraping Techniques: From HTML Parsing to Headless Browsers

Plenty of useful data sits on web pages that offer no download or API: published rate tables, regulatory notices, prices, directories. Web scraping is the practice of collecting that data with code. The right technique depends on how the page is built, and choosing the simplest one that works saves a lot of maintenance. This post was written with Bohdan Boiprav.

Check for an easier source first

Before writing a scraper, look for an official API, a CSV or Excel download, an RSS feed or a data portal. These are stable, documented and clearly permitted. Open your browser’s network tab as well: many pages load their data from a JSON endpoint, and calling that endpoint is simpler and more reliable than parsing the page. Whether you are allowed to use it is a separate question, covered below.

Technique 1: Pattern matching on raw HTML

The oldest approach is to download the HTML as text and extract data with regular expressions or string slicing. It works for a tiny, fixed snippet, and it breaks as soon as the markup shifts: an extra attribute, a changed class name or reordered tags. HTML is not a regular language, and patterns that look right on one page quietly return wrong values on the next. I would use this only as a last resort.

Technique 2: Parsing the DOM

A better approach is to parse the HTML into a tree and query it. The Document Object Model (DOM) represents a page as nested elements, and you select what you need with CSS selectors or XPath, such as “the second cell in each row of the table with id rates”. Libraries such as Beautiful Soup and lxml do this.

The earlier draft of this post said that DOM-based scraping depends on a browser’s rendering engine and is resource-hungry. That conflated two things. A library that parses the HTML you downloaded needs no browser and is fast and light. The browser enters the picture only when the content is generated by JavaScript, which is the next technique. See the MDN introduction to the DOM for background.

The snippet below checks robots.txt rules and then parses a small table. It runs offline, using a string in place of a live page.

from urllib import robotparser
from bs4 import BeautifulSoup

# 1. Check robots.txt before fetching (offline demo: parse rules from a string)
rp = robotparser.RobotFileParser()
rp.parse(["User-agent: *", "Disallow: /private/"])
print(rp.can_fetch("my-research-bot", "https://example.com/rates"))
print(rp.can_fetch("my-research-bot", "https://example.com/private/x"))

# 2. Parse with a real HTML parser, not regular expressions
html = """
<table id="rates">
  <tr><th>Tenor</th><th>Yield</th></tr>
  <tr><td>2Y</td><td>3.10</td></tr>
  <tr><td>10Y</td><td>3.45</td></tr>
</table>
"""
soup = BeautifulSoup(html, "html.parser")
rows = soup.select("table#rates tr")[1:]
data = {r.find_all("td")[0].text: float(r.find_all("td")[1].text) for r in rows}
print(data)

This prints True, False and {'2Y': 3.1, '10Y': 3.45}. The yields are invented for the example. Selectors tied to meaningful attributes, such as an id or a data attribute, survive redesigns better than long positional paths.

Technique 3: Headless browsers

Many modern sites send an almost empty page and build the content in the browser with JavaScript. A plain HTTP request then returns no data. Headless browser tools such as Playwright, Selenium and Puppeteer drive a real browser engine without a visible window. They run the page’s scripts, wait for content to appear and then let you read the final DOM.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com/rates")
    page.wait_for_selector("table#rates")      # wait for JavaScript to render it
    html = page.content()                      # then parse as above
    browser.close()

The cost is real: a browser uses much more CPU and memory than an HTTP client and is slower, and the automation adds its own failure modes, such as timeouts and pop-ups. Use it only for pages that need it. If the data comes from a JSON endpoint behind the page, calling that directly is cheaper.

Technique 4: Crawling at scale

Scraping a few pages is a script. Scraping many thousands needs a crawler that manages a queue of URLs, retries failures, throttles requests and stores results. Scrapy is a mature Python framework for this. For jobs spread across several machines, projects such as Scrapy Cluster use Redis and Kafka to share the work. The earlier draft also described “probabilistic IP rotation” to avoid anti-scraping measures, and it named a tool I could not verify, so I have dropped both. Evading a site’s defences is a different activity from collecting data it intends to share, and it is where legal and ethical problems tend to start.

Being a good citizen

• Read robots.txt. The Robots Exclusion Protocol is a standard way for sites to say which paths crawlers should avoid. The RFC itself notes that these rules are not a form of access authorisation, so they do not replace the site’s terms of use.
• Read the terms of service, and check whether the data is personal, copyrighted or licensed.
• Identify your scraper with a descriptive User-Agent and a contact address.
• Rate-limit your requests and cache what you download, so you do not hit the same page twice.
• Keep the raw HTML you extracted from, so you can re-parse when your logic changes or a value looks wrong.

A note on machine learning

There is research on learning to extract structured records from semi-structured pages without hand-written selectors, including approaches based on graph neural networks. I would treat it as a research direction rather than a replacement for selectors. For a handful of known sources, a selector plus a validation check is easier to debug.

Choosing a technique

Start with an official source. If there is none, parse the DOM with a proper library. Use a headless browser only when the data is rendered by JavaScript and no underlying endpoint is available. Move to a crawling framework when volume demands it. Then validate what you collect: row counts, value ranges and missing fields, because a scraper that fails silently is worse than one that crashes.