About

Foreman vs. Foundations

Foreman is the pipeline: the code that scrapes, cleans, scores, and serves London planning-application data. Foundations is what Foreman publishes — the filtered, housing-relevant slice of that data, free to download or query. Think of Foreman as the machine and Foundations as what comes off the end of it.

How it works

flowchart LR
  A["planit.org.uk
borough planning registers"] -->|scrape| B[("Application
(Postgres)")]
  B -->|train + score| Chousing_relevance_score
  C -->|"> threshold"| D["Foundations"]
  C -.->|"≤ threshold, kept but excluded"| B
  D --> E["CSV / Parquet export"]
  D --> F["read-only API"]
  B --> G["Analysis: planning patterns"]
      

Three modules, run independently: scrape → score/publish → analyse.

The three modules

• Scraper. Fetches every London borough's applications from planit.org.uk, upserts them into a single Application table, and tracks each run so a failed or partial fetch can resume rather than re-scrape from scratch.
• Relevance. Trains a classifier that scores each application for housing relevance, then exports everything above the threshold as Foundations — the CSV/Parquet files and the API on this site all read from the same scored table.
• Analysis. Turns the same underlying data into published findings — see Planning patterns for the current one, on deadline-driven decision bunching.

Foundations vs. the Planning London Datahub

The GLA's own Planning London Datahub covers more of the development lifecycle (construction dates, tenure mix, developer identity) but has no equivalent of Foundations' consultation intensity fields — comment/document counts and statutory consultation days, the figures the planning patterns analysis is built on.

Get the data

Head back to the Foundations landing page for the CSV/Parquet downloads, or the API docs to query it directly.