Autonomous scraping agent.

An LLM agent that explores web pages it has never seen and returns structured funding records, with no per-site selectors to maintain.

Stack: Python, Gemini, Postgres
Scale: 3,000+ opportunities indexed
Role: Designed and built

Flow

Seed URL
Fetch & render
LLM reads page
Decide: follow or extract
Structured record
Validate
Store

How it works

Traditional scrapers break whenever a site changes its markup. This agent is given a page and a target schema and decides for itself what the page is, whether to follow links, and which fields it can fill.

Extraction is schema-driven: the model returns a typed record, and a validator rejects anything that does not fit before it is written. Bad extractions become feedback rather than bad data.

Crawl state lives in Postgres so runs are resumable, and results feed straight into the matching pipeline's chunking and embedding steps.

Hard parts

Cost control: deciding when a page is worth a model call and when a cheap heuristic is enough.
Sites that render everything client-side, which meant driving a real browser for a subset of sources.

Outcome

Replaced a growing pile of hand-written scrapers with one agent, indexing 3,000+ opportunities and adding new sources without engineering work.