Public Data Scrape

Background

Early-stage desk studies are a standing cost on every project. A junior engineer spends several hours per site pulling public datasets together and writing them up, and the output quality depends on which sources they happened to know about.

What I built

Given only a site boundary and a buffer distance, the workflow queries UK public datasets covering heritage, highways, Natural England, Met Office, the North Sea Transition Authority and more, then passes the extracted features to an LLM for report generation.

How it works

The report structure is fixed and professional-grade:

• Disclaimer and search basis
• One-page non-technical summary
• Engineering summary of key constraints and risks
• Full constraints overview table by theme
• Ranked priority findings register with confidence levels
• Priority risk and action register with suggested owners
• Data quality and limitations section

Outcome

On a Wolverhampton test case the tool correctly flagged heritage and conservation constraints, Strategic Road Network proximity, gas infrastructure records and environmental designations, while explicitly noting where key datasets (flood zones, sewer networks, ground investigation, mining) were unavailable and needed separate validation.

It functions as a first-pass constraints screen supporting early design coordination and consultation planning, not a definitive statutory search, and the report says so.

The write-up itself is generated by a language model, which can misread a returned feature or leave one out, so every report is read and checked by an engineer before it is used. That check is part of the workflow rather than a caveat attached to it.

Where it goes next

Combining automated geospatial harvesting with LLM report generation is an emerging approach in civil engineering. This is a working implementation rather than a prototype, and the next step is broadening the source register.

Technical reference

Taken from a generated report rather than from a tool manual, since the report is the product. Site-specific findings are left out here; what follows is the method, the structure of the output and the limits the report declares about itself.

The search

The workflow takes an input geometry or site boundary and searches public datasets within a buffer, defaulting to 50 metres. Returned features may sit inside the boundary, intersect it, or fall only within the surrounding buffer, and the report is explicit that the spatial relationship needs validating against source geometry and current survey information rather than being assumed from the fact a record came back.

That distinction does more work than it looks like it does. A constraints search that cannot tell you whether a listed building is on your site or forty metres away is not a constraints assessment, and the report is built to say so rather than to imply precision it does not have.

How findings are categorised

Every theme in the constraints overview carries a relevance category, which is what turns a list of returned records into something a project can act on.

Each theme row records a short summary, the engineering implication and a recommended action, so the reader gets from a returned record to a next step without an intervening interpretation exercise.

The findings and action registers

Priority findings are listed individually with a theme, the finding itself, the relevance category, the potential engineering impact, a recommended action and a confidence level. That last column is the one that matters most, because a low-confidence finding from a generalised national dataset should not carry the same weight as a high-confidence one, and without the column the report would flatten the two.

The action register then converts findings into owned work: an action ID, a priority, the theme, the action, a suggested owner or discipline, why it matters, and what validation is required to close it out. Owners are suggested by discipline rather than by name.

Data quality, as declared

The report carries its own limitations section, and it is not decorative. The standing caveats:

• Public datasets may be incomplete, out of date, generalised, duplicated or spatially inaccurate, and are screening information only.
• Records repeat across similar Historic England and Natural England layers, so counts indicate returned records rather than confirmed unique constraints.
• Coordinate systems are mixed in the harvested output, with layers recording both EPSG:27700 and EPSG:3857, so the coordinate basis requires validation rather than being taken on trust.
• Many returned fields are system, renderer, symbology or service fields and are not reproduced in the report.
• Features come from a buffer search, so they are not necessarily inside the site boundary.

What public data does not cover

The most useful part of the output is arguably the absences. Several datasets that a civil engineering desk study actually needs are not reliably present in public sources: detailed utilities and sewers, flood zones, ground investigation, contamination, mining, and adopted highway extents.

The report names these gaps explicitly rather than producing a clean-looking assessment with silent holes in it. A screening tool that quietly omits flood risk is worse than no screening tool, because the omission reads as an absence of constraint.

Identified improvements

The workflow's own output points at its next iteration: adding proximity flags so features are tagged on-site, adjoining or buffer-only at source rather than in review; de-duplicating repeated records across similar layers; stripping renderer and system fields from the reporting output; and carrying source dataset date, licence and confidence fields through where the service exposes them.