Projects / AI products
Autonomous scraping agent.
An LLM agent that explores web pages it has never seen and returns structured funding records, with no per-site selectors to maintain.
- Stack
- Python, Gemini, Postgres
- Scale
- 3,000+ opportunities indexed
- Role
- Designed and built
Flow.
- Seed URL
- Fetch & render
- LLM reads page
- Decide: follow or extract
- Structured record
- Validate
- Store
How it works.
Traditional scrapers break whenever a site changes its markup. This agent is given a page and a target schema and decides for itself what the page is, whether to follow links, and which fields it can fill.
The model returns a typed record against a fixed schema, and a validator rejects anything that does not fit before it is written. A bad extraction becomes a correction signal, never bad data.
Crawl state lives in Postgres so runs are resumable, and results feed straight into the matching pipeline's chunking and embedding steps.
Hard parts.
- Cost control: deciding when a page is worth a model call and when a cheap heuristic is enough.
- Sites that render everything client-side, which meant driving a real browser for a subset of sources.
Outcome.
Replaced a growing pile of hand-written scrapers with one agent, indexing 3,000+ opportunities and adding new sources without engineering work.