Scraper Pipeline

Playwright crawlers, queues, and clean data models — scraping as a system, not a script.

Everything around the scraping that scripts skip: polite crawling with rate limits and backoff, rotating sessions, HTML-to-schema extractors, incremental sync with dedupe, and scheduled runs — persisted into a Postgres model you can build products on.

What's inside

• Polite crawling: rate limits, retries, backoff
• Rotating browser sessions and fingerprints
• HTML → schema extractors with validation
• Incremental sync and dedupe
• Scheduled jobs with run history
• Admin CLI for backfills and inspection

Quickstart

$ git clone git@github.com:devai-io/scraper-pipeline.git my-scraper$ cd my-scraper && uv sync$ cp .env.example .env$ uv run python -m scraper.cli crawl --site example