Chapter 26, Advanced
Web crawling with BeautifulSoup
When there is no API, sometimes the only source is the HTML page. Extracting data from it is possible, but it is fragile, has legal and ethical limits, and should be the last resort.
Before writing a line
Can you collect data from this site? Three questions, in this order: does the site offer a API or an RSS feed (much more stable)? Do the terms of use allow automated collection? Does the robots.txt site allow access to that path? And, if the data includes people, does the LGPD apply? This chapter is purely technical and uses HTML written right here, without accessing any site.
The robots.txt can be read with the standard library:
from urllib.robotparser import RobotFileParser
regras = """User-agent: *
Disallow: /privado/
Crawl-delay: 5
"""
leitor = RobotFileParser()
leitor.parse(regras.splitlines())
print(leitor.can_fetch("meu-bot", "https://site.exemplo/noticias"))
print(leitor.can_fetch("meu-bot", "https://site.exemplo/privado/dados"))
print(leitor.crawl_delay("meu-bot"))
True
False
5
The Crawl-delay asks for 5 seconds between requests. Respecting this (and identifying yourself with a User-Agent real user agent) is the minimum courtesy, and it prevents your IP from being blocked.
Extracting data from HTML
The BeautifulSoup transforms the HTML into a tree that you query with CSS selectors, the same ones from the browser. Use the browser's developer tools (right-click, "Inspect") to find the tag and class of what you want:
uv add beautifulsoup4
from urllib.parse import urljoin
from bs4 import BeautifulSoup
HTML = """
<html><head><title>Notícias do dia</title></head><body>
<article><a class="manchete" href="/a">Primeira manchete</a><span class="hora">10:00</span></article>
<article><a class="manchete" href="/b">Segunda manchete</a><span class="hora">11:30</span></article>
<article><a class="manchete" href="https://outro.site/c">Terceira manchete</a></article>
<a href="/sobre">Sobre nós</a>
</body></html>
"""
sopa = BeautifulSoup(HTML, "html.parser")
print(sopa.title.text)
print([a.get_text(strip=True) for a in sopa.select("a.manchete")])
print([a["href"] for a in sopa.select("a.manchete")])
print([urljoin("https://site.exemplo", a["href"]) for a in sopa.select("a.manchete")])
Notícias do dia
['Primeira manchete', 'Segunda manchete', 'Terceira manchete']
['/a', '/b', 'https://outro.site/c']
['https://site.exemplo/a', 'https://site.exemplo/b', 'https://outro.site/c']
The select("a.manchete") only grabbed the links with that class, leaving "About Us" out. The urljoin transforms relative links (/a) into absolute ones and keeps those that were already absolute. The html.parser is the parser that comes with Python, without installing anything.
A pure function, easy to test
Separate fetching the HTML from extracting the data. The extraction, a pure function that takes text and returns data, can be tested without a network and without a site:
def extrair_manchetes(html: str, base: str) -> list[dict]:
sopa = BeautifulSoup(html, "html.parser")
manchetes = [
{"titulo": a.get_text(strip=True), "link": urljoin(base, a["href"])}
for a in sopa.select("a.manchete")
if a.get("href")
]
return manchetes
print(extrair_manchetes(HTML, "https://site.exemplo"))
[{'titulo': 'Primeira manchete', 'link': 'https://site.exemplo/a'}, {'titulo': 'Segunda manchete', 'link': 'https://site.exemplo/b'}, {'titulo': 'Terceira manchete', 'link': 'https://outro.site/c'}]
Inside an API
The route uses the extraction function and handles the most common failure case of a crawler: the page changed. When the selector fails to find anything, the collection returns an empty list without any error, and this can go unnoticed for weeks. It is better to fail noisily:
from typing import Annotated, Callable
import httpx2
from fastapi import Depends, FastAPI, HTTPException
from fastapi.testclient import TestClient
app = FastAPI()
def buscar_html() -> str:
resposta = httpx2.get(
"https://site.exemplo/noticias",
headers={"User-Agent": "meu-bot/1.0 (contato@exemplo.com)"},
timeout=5.0,
)
resposta.raise_for_status()
return resposta.text
@app.get("/manchetes")
def manchetes(html: Annotated[str, Depends(buscar_html)], limite: int = 5):
itens = extrair_manchetes(html, "https://site.exemplo")
if not itens:
raise HTTPException(502, "A estrutura da página mudou: nenhuma manchete encontrada")
return {"total": len(itens), "dados": itens[:limite]}
app.dependency_overrides[buscar_html] = lambda: HTML
cliente = TestClient(app)
print(cliente.get("/manchetes?limite=2").json())
app.dependency_overrides[buscar_html] = lambda: "<html><body>redesenhado</body></html>"
resposta = cliente.get("/manchetes")
print(resposta.status_code, resposta.json())
app.dependency_overrides.clear()
{'total': 3, 'dados': [{'titulo': 'Primeira manchete', 'link': 'https://site.exemplo/a'}, {'titulo': 'Segunda manchete', 'link': 'https://site.exemplo/b'}]}
502 {'detail': 'A estrutura da página mudou: nenhuma manchete encontrada'}
As buscar_html is a dependency, the test replaces it with fixed HTML (the dependency_overrides from chapter 24), and the route runs without any network access. The second test simulates the redesigned page and confirms that the failure appears as 502.
Crawler is fragile by nature
The structure of a page changes without warning, and your code breaks. Therefore: prefer an API or a feed when available; store in cache the result (chapter 28), so you don't request the page on every request; limit the frequency; and monitor the
502of the example above. A crawler in production is a maintenance debt, and I only accept that debt when there is no other way.
Exercise 1
Unique and absolute links
Write extrair_links(html, base), which returns all the links (<a href>) from the page as absolute addresses, without repetition and in the order they appear.