Can General-Purpose LLMs Perform Accurate Civil Estimates? Claude vs ChatGPT vs Civils.ai

We ran a controlled, blind quantity-takeoff test on a live landscaping, utilities and road-surfacing scheme. Two general-purpose LLMs (Claude Sonnet 4.5 and GPT-4-class ChatGPT) and Civils.ai were each given the same Issued-for-Construction (IFC) drawing package and the same blank Bill of Quantities templates. A third-party chartered quantity surveyor produced the ground-truth quantities. This article reports every line item, every error band, and the priced-bid impact.

Key Facts

Key Terms

1. Approach

The experiment was designed as a controlled, blind accuracy test. Each estimator received only the IFC drawing package and two blank BoQ templates — no hints, no reference quantities, and no human guidance on what to count or how to measure it.

Source Materials

Project: Northgate Link Road, Utilities & Public Realm Scheme — a ~1.1 km link road with associated storm drainage, wet and dry utilities, and a public-realm/landscape package (turf, planting, irrigation, hard landscape). Representative of a mid-scale civils and infrastructure project.

How the general-purpose LLMs worked — agentic file mode

Both Claude and ChatGPT ran in an agentic, sandboxed Linux environment with a Bash shell, file read/write and web search. Each followed broadly the same workflow:

1. Read the drawing register to build a page-number index.
2. Used pdftotext to extract machine-readable text from schedule pages (footing/pier-equivalent schedules, pipe schedules, manhole schedules, planting schedules).
3. Used pdftoppm to render plan, section and elevation pages as 200 DPI PNG images for visual scale reading.
4. Wrote a Python script using openpyxl to populate both BoQ templates while preserving existing formulas and formatting.

Each ran in a single coherent session. Estimated API cost: ~$4 (Claude) and ~$3 (ChatGPT).

How Civils.ai worked — purpose-built takeoff engine

Civils.ai ingested the same IFC package into its civil-specific takeoff engine. Rather than reading pages one at a time, it vectorised the plan and section sheets, auto-classified every drawing against CESMM4/CSI divisions, and — critically for this scope — reconstructed the earthwork surface from the grading contours and spot levels rather than assuming a uniform depth. It cross-referenced plan, section and schedule for every measured item (for example, reconciling pipe runs on the drainage layout against the pipe schedule and the manhole schedule), and resolved scope boundaries (single-side vs both-side kerb, net vs gross planting area) from the detail drawings.

2. Ground Truth — Chartered Quantity Surveyor

After all three estimators completed their takeoffs, an independent chartered quantity surveyor produced their own quantities from the same drawing package, working to CESMM4. Their figures were entered alongside the three machine sets, creating a direct line-by-line comparison across 45 non-lump-sum line items spanning earthwork, surfacing, landscape and utilities.

3. Results

3.1 Accuracy Distribution

The two general-purpose LLMs show a strongly bimodal distribution: schedule-driven counts cluster near zero error; derived volumes, distributed counts and scope-dependent linear items sit in the extreme tails. Civils.ai shows no items outside ±3%.

3.2 Per-Measurement-Unit Accuracy (mean absolute error)

For the LLMs, accuracy tracks how the quantity is derived, not the trade: anything read directly from a table (asphalt courses from the pavement build-up, chamber counts from a schedule) is close, while anything requiring derivation from geometry (m³ volumes, m² surfaces) or aggregation across sheets (LM runs) drifts badly.

3.3 Full Line-Item Comparison

Each machine cell shows the quantity and its error against the QS ground truth. Positive = overestimate, negative = underestimate.

Division 31 — Earthwork

Division 32 — Surfacing, Paving & Landscaping

Division 33 — Utilities

3.4 Cost Impact (Priced Bid)

Indicative mid-range unit rates (Spon's Civil Engineering and Highway Works Price Book, GBP) were applied to all four quantity sets, including a standard 30% addition for contingency, overheads and profit, and levies.

Civils.ai landed within 3% of the priced QS bid (97.0% accurate). Both general-purpose LLMs came in ≈40% low at the bid level — a gap of roughly £1.4M–£1.6M against the QS figure, which on a tender of this size is the difference between a credible bid and a non-compliant one.

3.5 Top Cost Misses — Claude

The five categories below account for roughly £540K of Claude's £1.43M total gap; the remainder is distributed across the other 30-plus line items.

4. Systematic Error Taxonomy

Every large LLM error in this test maps to one of six repeatable failure modes. Civils.ai's engine is built to neutralise all six.

The two over-estimating modes (kerb perimeter, gross area) partly mask the under-estimating modes at the total-cost level, which is why the LLM bids can look "only" 40% low despite most individual line items being far more wrong in one direction or the other. This netting-out is itself a hazard: it makes a badly constructed takeoff appear more plausible than it is.

5. Takeaways

Where general-purpose LLMs work well

• Schedule-driven counts. Standard and semi-mature trees, manholes, valve chambers and headwalls — anything read directly from a clean tabular schedule — came back exact or near-exact.
• Tonnage from an explicit build-up. Asphalt binder and surface courses, priced from the stated pavement build-up, were within 3%.
• Simple plan areas with a clear callout. Carriageway area and site clearing were close because they derive from a single dimensioned boundary.
• Lump sums. General-conditions items default to 1-each and are always correct.

Where general-purpose LLMs struggle today

• Derived volumes. Bulk earthwork is the single worst category. Both LLMs multiplied a plan area by an assumed uniform depth; neither reconstructed cut and fill from the grading surface. Result: 38–45% underestimate on the highest-value items in the bill.
• Networked linear work. Drainage and ducting were read as their primary runs only. Laterals, branch connections and multi-way duct banks — spread across multiple sheets — were missed, producing 30–46% underestimates.
• Distributed counts. Shrubs, gullies, draw pits and signs are enumerated across many plan sheets rather than in one schedule; the LLMs consistently under-counted them.
• Scope-boundary judgement. Kerb, edging and channel were pushed to a full-perimeter, both-sides default (+29% to +56%), while soft-landscape areas used gross site area rather than net planting zones (+41% to +55%).

Why Civils.ai reaches 97%

Civils.ai is purpose-built for civil takeoff, so the six failure modes above are handled by design rather than by luck:

• Earthwork from levels, not plan area. It reconstructs the cut/fill surface from contours and spot levels — the root fix for the largest category of LLM error.
• Plan + section + schedule reconciliation. Every measured item is cross-checked across all three view types, so pipe runs match the pipe schedule and chamber counts match the manhole schedule.
• Network tracing. It follows drainage and duct networks through laterals and branches instead of counting only the trunk run.
• CESMM/CSI-aware scope resolution. Kerb sides, net-vs-gross planting and pavement build-up thickness are resolved from the standard method of measurement and the detail drawings, not from a default.

Economic Assessment

A general-purpose LLM takeoff costs ~$3–$4 in API usage and takes under an hour. That is genuinely useful as a first-pass index — it names every line item and flags where the risk sits. But at a 40% bid-level underestimate, it is not a tender-ready quantity set, and the errors are systematic rather than random, so they will not "average out."

A chartered QS billing at £80–£150/hour would spend 20–45+ hours on a takeoff of this size — £1,600–£6,800 in labour. Civils.ai reproduced the QS bid to within 3% in a fraction of that time. The strongest workflow is: Civils.ai produces the measured takeoff, a QS reviews and signs off the high-value earthwork, drainage and surfacing lines — a review measured in hours, not days.

Recommended workflow

1. Use a purpose-built civil engine for the measured takeoff. General LLMs are best kept to a sanity-check / line-item-discovery role.
2. Always human-review the three highest-risk categories: bulk earthworks, networked drainage/ducting, and pavement/surfacing build-ups.
3. Derive earthwork from levels, never from plan area × assumed depth.
4. Resolve scope boundaries explicitly — kerb sides, net vs gross planting area, pavement layer thicknesses — before pricing.

Frequently Asked Questions