Scrocle

E-commerce data pipeline · 2025

Daily competitor price ingest

Shipped

Role
Independent engineer
Stack
Python, BeautifulSoup, Selenium
Status
Shipped

Problem

Competitor prices lived in a morning ritual: open several storefronts, copy figures, paste a sheet. A layout change on one site broke the day. There was no retry, no per-source isolation, and no record of what failed.

Constraint

Multiple storefronts, different pagination, rate limits that are not documented. One broken parser must not take down the rest of the run.

Architecture

  1. Scheduler
  2. Storefronts
  3. Parsers
  1. Normalize
  2. Morning table
  3. Source log
A scheduler starts the job. Each storefront is fetched and parsed on its own. Records are normalized into one table. A failed source is retried or logged, not allowed to abort the others.

Decisions

  • One job, many sources

    A scheduled process pulls each storefront independently so a selector change is a source failure, not a pipeline failure.

  • BeautifulSoup where the HTML is stable, Selenium where it is not

    Not every page needs a browser. The ones that do are isolated so the rest stay cheap.

  • Normalize, then write

    The morning artefact is a table the client can open. Raw markup stays in the job's output when a source needs a replay.

If the brief is “we check prices by hand,” this is the pattern: scheduled ingest, isolated sources, a table in the morning, and a log when a page moves.

Outcome

Shipped. The sheet is no longer the pipeline. Client name stays off this page.

What was handed over

  1. How the job is scheduled
  2. Per-source notes and what a selector change looks like
  3. The output table and who opens it

hire@scrocle.cloud

Next project Sales data analysis Quarterly sales model and dashboard