Codex Agents Scrape 107 Million Rows of UK Council Data

Ben's Bites has built an interactive, Apple Maps-style visualization of UK public spending by deploying Codex and a fleet of 656 autonomous subagents to scrape and process 107 million rows of data. The project targeted English local council payments exceeding £500, which are legally public but notoriously difficult to access and format. Codex initiated the process by spinning up three subagents to catalog 31 official sources, initially gathering 1.8 million rows of data worth £9 billion from eight councils into a single CSV file.

To scale the project to 339 councils across England and Wales, the developer transitioned the workload from a MacBook Air to a Mac Mini. Based on Codex's recommendation, the system utilized Parquet for columnar storage and DuckDB to manage the massive dataset. A single goal-oriented prompt ran for nearly two days across 552 threads, successfully collecting and formatting data from 319 councils, while six councils blocked the scrapers and Welsh councils were eventually excluded.

The AI agents also handled the front-end development and data cleaning. Four subagents classified 2,000 unknown vendors, resolved duplicate entries, and refined categories. To visualize the results, Codex generated multiple prototypes, publishing them directly to the here.now hosting platform. The final static site features clickable council outlines, category filters, and search functionality, bypassing a traditional backend by loading pre-processed data files directly.

For AI practitioners, this build highlights the viability of using hierarchical agent systems to automate complex, multi-step data engineering pipelines. By delegating specialized tasks—such as scraping, auditing, and categorizing—to autonomous subagents, a single developer can execute massive data collection and visualization projects that would traditionally require an entire engineering team.