- Vendors sell three different jobs as one platform. Progress tracking asks what got built, safety monitoring asks who is at risk now, quality detection asks whether the work is correct. They fail in different ways.
- The published evidence is thinner than the marketing. A study of GPT-4 Vision on site imagery found it proficient at identifying stages, materials and machinery while facing challenges with precise localisation, and it reports no accuracy percentage at all.
- The detection papers report precision and recall on their own annotated datasets and say nothing about mud, rain, winter light or scaffold occlusion.
- The failure that costs most is the quiet one. A site camera that stops detecting in poor light and says nothing produces a clean safety report from a system that has gone blind.
- Deployments die of alert fatigue, not of model accuracy. Set an alert budget before the pilot starts and make progress data auditable before it touches a payment application.
The demo detects the missing hard hat in about a second. Clean footage, good light, one worker walking toward the camera. Everyone in the room nods, because that is the whole point of the demo. Then the system goes on a live site in February: rain on the lens, a stack of plasterboard blocking half the view, a hoist moving through frame, and nobody has told you what happens next.
AI in construction is a serious market and the pitch is unusually concrete, which makes it unusually testable. What is missing from almost every page a buyer finds is any account of what degrades these systems. This is that account, with the published evidence reported honestly, gaps included.
What are you actually buying?
Three distinct products, usually bundled under one subscription. Separating them is the single most useful thing a buyer can do, because each one has its own failure mode, its own tolerance for error and its own consequence when it goes wrong.
Progress tracking asks what has been built. It is a measurement problem, and its errors show up as a number that drifts from reality. Tolerable as a dashboard, serious when it feeds a valuation.
Safety monitoring asks who is at risk right now. It is a real time problem with a human in the loop, and its errors are either a false alarm that annoys a crew or a miss that nobody ever learns about.
Quality and defect detection asks whether what was built is correct. It is the hardest of the three, because the model needs to know what right looks like for this detail in this specification, and small deviations are precisely what matters.
What does the research actually establish?
That these systems recognise things in photographs. Not what they do across a wet winter on a live site, and the gap between those two statements is the story.
Take the large vision language model study first. A paper examining GPT-4 Vision applied to construction progress monitoring using high resolution aerial imagery found the model proficient at identifying construction stages, materials and machinery, while facing challenges with precise object localisation and segmentation. Read that sentence carefully, because the second half is the useful part: it can tell you roughly what it is looking at and struggles to say exactly where something is or draw its boundary. For progress measurement, where is most of the job.
The important methodological point is what that paper does not contain. It is a qualitative assessment and it reports no accuracy figures. If a vendor cites it to you alongside a percentage, the percentage came from somewhere else.
The purpose built detection work is more quantitative and narrower. A paper on automated construction progress monitoring using object detection built its own annotated dataset of building elements and reported precision, recall and F1 score, claiming significant improvement over existing methods. What its abstract does not discuss is dataset size limits, weather, lighting or occlusion. So the honest summary of the published position is this: detection of building elements in curated site images works and is improving, and nobody has published what it does through a British January.
Which site conditions break it, and does it tell you?
The conditions are predictable and the answer to the second question is usually no. This table is the one to take into the meeting. The last column is the one that decides whether a deployment is safe to rely on.
| Capability shown in the demo | What degrades it most | Visible or silent failure | What the operator has to do |
|---|---|---|---|
| PPE detection, hard hats and high visibility vests | Low winter light, rain on the lens, distance, workers turned away | Silent, and the report still looks clean | Check detection counts per camera per day, not just alerts |
| Fall and edge protection detection | Scaffold and mesh occluding the edge, changing geometry as work moves | Silent, since an undetected edge generates nothing | Re survey camera views whenever the structure changes |
| Progress percent complete by trade | Material stacked over finished work, interior work invisible from outside | Visible as drift, if somebody reconciles it | Reconcile against a manual check before it feeds billing |
| Material and delivery identification | Wrapped or palletised goods, mud covering identifying marks | Visible, because a missing delivery gets chased | Keep the paper trail as the source of truth |
| Equipment utilisation tracking | Plant parked out of view, dust, night shifts | Quietly understated | Spot check against operator timesheets |
| Defect and rework detection | Anything hidden behind a finish; tolerance smaller than the pixel size | Silent, and the earliest stage of the three products | Treat as a prompt to inspect, never as a sign off |
The silent against visible distinction deserves its own paragraph because it inverts how people assess risk. A noisy system is annoying and self correcting: the crew complains, somebody looks. A quiet system is comfortable and dangerous. Cameras that have effectively gone blind in low light still produce a report, and that report says no violations detected, which reads exactly like a safe site.
The same gap between curated evaluation and working conditions shows up wherever computer vision leaves a controlled setting. We documented the agricultural version, where light and canopy do the damage, in what camera based systems manage in a real field, and the robotics version in how to read a robot demo. The pattern is consistent enough to be a buying rule rather than an industry quirk.
Why is progress tracking the one with contractual teeth?
Because a percentage that nobody checks can end up in a payment application, and at that point a software error becomes a commercial dispute with your client.
Think about what percent complete means on a real trade. A plasterboard package is 60 percent complete when the boards are up on three floors of five, except the second floor was boarded first and then covered by stacked materials for the following trade, so from the air it looks finished and from inside it is not. The camera sees surfaces. The valuation needs volumes of work in place. Those are different quantities and only one of them is measurable from a photograph.
The vision language study above is directly relevant here, since its stated weakness is precise localisation and segmentation. Measuring how much of something exists is exactly a localisation and segmentation task. A model that identifies the stage of construction correctly and cannot delineate extents is useful for a narrative update and not yet for a quantity.
So the practical rule is a boundary rather than a rejection. Let progress vision drive attention, which means telling you where to look and what changed since last week. Keep the number that drives money coming from a measured, reconciled process until you have run both in parallel long enough to trust the divergence. If a vendor proposes going straight from install to valuations, they are asking you to carry a risk their own cited research does not support.
Why does safety monitoring fail even when the model works?
Alert fatigue. This is the mechanism by which these deployments actually die, and it has nothing to do with accuracy.
Run the arithmetic before the pilot. Suppose four cameras and a reasonable detection rate produce 60 flagged events a day across a site. Somebody has to open each one, look at the clip, decide whether it is real and act. At even 30 seconds each that is half an hour daily, and most of those events will be a worker who had a hard hat on two frames later. Within a month, nobody opens them. The system is now generating a compliance record that no human reads, which is worse than having no system, because the record exists and implies review.
So specify an alert budget first: how many events per day a named person will genuinely action, and then configure to that number rather than to maximum sensitivity. If the vendor cannot tune to a budget, the pilot will fail in week five regardless of how good the detector is. Whoever owns that review time needs it in their actual schedule too, which is a staffing question before it is a technology one, in the way we set out in scheduling a small team around work that has to happen.
Does any of this reduce your insurance or your liability?
Not automatically, and the honest answer is that it can cut both ways. A monitoring system creates a record, and a record is evidence. That evidence helps when it shows you identified a hazard and acted. It does not help when it shows a flagged event nobody opened.
This is the part of the business case that vendors are quietest about. Once a system is installed and generating alerts, the alerts exist whether or not anybody actioned them. A site that was previously operating on walk rounds and judgement now has a timestamped list of hazards it was told about. If the review process is real, that is a genuine improvement in both safety and defensibility. If the review process is a dashboard nobody opens, the installation has manufactured a paper trail of unactioned warnings.
Which brings the alert budget back round as a liability question rather than an efficiency one. The number of alerts a day you can genuinely action is not a comfort setting, it is the volume at which your record stays consistent with your conduct. Agree it, write down who owns it, and keep evidence that the review actually happens. A monthly note recording how many events were reviewed and what was done is tedious and it is the cheapest protection available.
What is the regulatory picture these systems are sold against?
Worth knowing precisely, because the sales pitch usually gestures at it. On fatalities, the US Bureau of Labor Statistics reported in its 9 May 2025 analysis of fatal falls in construction that about one in five workplace deaths in 2023, 20.8 percent, occurred in the construction industry, that 38.5 percent of those deaths were due to falls, slips and trips, and that construction accounted for 47.8 percent of all fatal falls, slips and trips that year. Roofing contractors alone accounted for 26.0 percent of those fall fatalities, with 110 deaths.
On enforcement, the picture is adjacent but distinct. OSHA's commonly used statistics page records 5,283 fatal work injuries across all industries in 2023, a rate of 3.5 per 100,000 full time equivalent workers, and lists the standards most frequently cited in fiscal year 2024. Several of those are exactly what a camera claims to watch: fall protection general requirements, ladders, fall protection training, scaffolding and eye and face protection. Those are citation counts rather than injury data, and the overlap is the real commercial argument for this technology.
Read together, the numbers justify attention to falls and edge protection above everything else, which is also the capability most likely to fail silently behind scaffold. That is an uncomfortable combination and it should shape where you point cameras and what you verify by walking the site.
What should you ask the vendor?
Seven questions. Three of them are rarely asked and they are the ones that determine whether you are still using the system next year.
- What was the model trained on, and does it include sites like mine? Climate, build type and site layout all matter, and a contractor working refurbishment is not the same case as new build. A model trained on dry weather high rise work is being asked to generalise.
- What happens when a camera view is blocked or the light drops? The answer you want is that the system reports degraded confidence rather than nothing. Ask to see that state in the interface.
- Who reviews a flagged safety event, and how fast? If the answer is the site manager, that is a real cost on a real person. Name them and agree the budget.
- What is the false alarm rate on a site like mine? Not on the benchmark. If they have never measured it on a comparable site, your pilot is the measurement.
- Does any of this feed a payment application? This is the question that turns software into a contractual exposure. An inaccurate percent complete in a dashboard is an argument. The same number in a valuation is a dispute.
- Can I export the raw detections, not just the dashboard? Without the underlying data you cannot audit a quiet failure, and you cannot leave.
- Who owns the footage, and what are the rules on workers appearing in it? Site cameras record people at work. Settle retention, access and any consultation obligations before cameras go up, not after somebody asks.
How should a pilot be structured?
On one site, for one of the three products, with the comparison recorded by hand. Trying all three at once guarantees you cannot tell which part worked.
Pick the job where you already have a manual baseline, because that is what makes the result interpretable. If a site manager currently walks the site each morning and notes PPE issues, you have a comparison. Run both for a month, count what the camera caught that the walk missed and what the walk caught that the camera missed, and keep the second list visible, since it is the one the vendor dashboard cannot show you.
Agree the stop condition in advance. A pilot with no failure criterion becomes a subscription by default. Write down what would make you decline: more than a stated number of alerts a day, a detection count that collapses in poor weather, a progress figure that diverges from the manual check by more than an agreed margin.
Price it honestly too. The licence is the visible cost and rarely the largest one. Add camera installation and moves as the structure changes, the connectivity, and the review hours. Usage based pricing at least makes AI spend legible rather than hiding it in a flat fee, which is why our own credit pricing follows actual consumption, and the same transparency is worth asking for from any site platform.
What is likely to improve, and what is not?
Detection in poor conditions keeps getting better, and more site specific training data exists every year, so the gap between demo and February narrows. Progress measurement will benefit most from being combined with the model of the building rather than from better image recognition alone, since knowing what should be there is most of knowing what is missing.
What will not change is the structure of the problem. A camera sees surfaces, so anything behind a finish or inside a wall stays invisible. A fixed view sees what it can see, so occlusion remains a scheduling problem rather than a software one. And a system that reports nothing when it fails will keep producing reassuring reports from blind cameras until somebody checks detection counts rather than alerts. That check costs nothing and almost nobody does it, which makes it the highest return habit in this entire subject.