Ingestion · 2026
Durable web ingestion pipeline
Laboratory
- Role
- Independent engineer
- Stack
- Python, Camoufox, SQLite, Docker
- Status
- Laboratory
Problem
Scrapers that refetch the web on every crash, share one rate limit across sources, and dump broken records into the same table as good ones are not a pipeline. They are a script with optimism.
Constraint
This is a laboratory pattern, not a client metric page. Four demo sources ship with it, including a JavaScript-rendered page. Tests run offline against frozen HTML so a live site cannot flake the suite.
Architecture
- YAML sources
- Fetch + cache
- Parse
- Dedup
- Clean table
- Dead letter
Decisions
YAML per source
Rate limits, selectors, and retries live next to the source, not in a shared bag of globals.
Cache the raw HTML
A crash does not refetch the corpus. Replay is a local problem.
Two-layer deduplication and a dead letter
Duplicates and poison records do not enter the clean table. Retries happen when the server asked, not on a timer of hope.
Use this page to talk about durability. Use the e-commerce pipeline page if you need a shipped ingest with a morning table.
Outcome
Laboratory. Client metrics belong on the client pages, not here.
What was handed over
- How to add a source in YAML
- How to run the tests offline
- What the dead-letter queue looks like