Limitations | Blueprint Risk Engine

01Both kinds of wrong

Expect both false positives and false negatives.

That sentence is the honest summary of what this tool is, and it is at the top rather than in a footnote because it is the thing that decides whether the tool is any use to you.

A false positive is a flag on a line where nothing is actually wrong. The rule fired, the citation is real, the sentence it quoted says what it says, and there is still no problem. You open the page, read the line, and close it. That costs you a minute and it costs the tool credibility, which is the more expensive of the two.

A false negative is the gap the engine walked straight past. It is worse and it is quieter. A short register looks like a clean package, and the two are not the same thing. There is a known example already recorded rather than hidden: the scope detector matches marker phrases such as NIC, by others, furnished by, installed by, OFCI and CFCI, and on the first real project manual it was run against it missed a page that assigned owner-furnished scope in ordinary English with none of those words in it. That page was the residual catch-all, which is the most expensive form of the thing it was built to catch.

That gap was left open on purpose. Widening the pattern to catch phrases like "responsible for" would fire on a large fraction of any specification's body text, and a rule that fires too broadly is worse than an absent rule because it trains you to ignore the output. The gap is held open, recorded, and guarded by a test that fails if anyone closes it quietly.

Sources of false positives

Rule scope. A rule written for one jurisdiction's reading of a code section fires in a jurisdiction that reads it differently. Rules cite their authority and stay project-overridable for this reason.
Genuinely resolved language. A scope marker that a later sentence correctly settles can still be located and flagged.
Match error. Two lines that look like the same opening and are not.
Uncalibrated ranking. Exposure bands are placeholders, so an item can sort higher than it deserves.

Sources of false negatives

Vocabulary the detector does not hold. The recorded case above. Scope stated in plain English is not matched.
Rule coverage. Several rule families are allocated and ship no rule. The table below names each one.
Documents it never reads. Drawings are not read at all. Addenda are not reconciled.
Evidence it cannot see. Half the Division 08 to Division 28 seam analysis reads structured claims, and nothing populates those on any package the tool can currently produce.

This framing is borrowed openly from how static analysis tools document themselves. A security scanner has the same structural problem: it fires on code that is fine and stays silent on code that is not, and the only usable posture is to say so at the top and let the reader decide what the review is worth. Nothing about construction documents makes that problem easier.

02Not benchmarked

It has not been benchmarked

There is one measurement that would settle whether this engine is worth anything, and it does not exist.

That measurement is precision and recall of pre-bid flags against the change orders a project actually issued, plus lead time: how far ahead of the event the flag was raised. Precision is what fraction of the flags mattered. Recall is what fraction of the things that mattered got flagged. Lead time is whether knowing would have helped.

Nothing else substitutes for it. Not a passing test suite, not a deterministic replay, not a well-cited register. Those make the output auditable. They say nothing about whether the things it flags are the things that become change orders.

The method intended to produce it

Published here before there are results, because a methodology published after the numbers is a methodology chosen to fit them.

Source packages where the outcome is public. Bid documents from district procurement portals and purchasing pages, paired with the change-order record from that project's own board minutes. Both halves have to be publicly retrievable by a third party, from a citation, without going through this project.
Store manifests, never documents. The dataset holds the bid number, the district, the source link, the retrieval date and the derived annotations. Source bid documents and drawings are never committed. Anyone checking the work retrieves the originals themselves.
Require an additive change order traceable to a document defect. This correction came out of the sourcing search and it matters more than any lead it produced. Most publicly documented school security change orders are deducts: an unused allowance returned at closeout, equipment specified and not installed, a time extension for a supply delay. A deduct has no pre-bid document defect underneath it, so recall measured against one is undefined. What scores the engine is a change order that added scope for a stated reason: work assigned to nobody, a count wrong in the schedule, a division seam neither section covered, a pathway or power requirement nobody priced.
Confirm the project is actually in scope before accepting it. In school board agendas, "safety and security" usually means Division 27: intercom, paging, bells, clocks, classroom audio. It does not usually mean Division 28. Scanning agenda titles produces a steady stream of these, and one was accepted as a leading candidate here before the specification was opened and found to contain zero occurrences of Division 28. No candidate is accepted until a Division 28 section number or access control hardware is confirmed in the specification itself. That check costs one download.
Score counts and outcomes, never awarded dollars. An award price carries labor rates, prevailing wage, overhead, profit, and how badly that contractor wanted the job. It measures a bid strategy, not a document defect.
Publish the failures with the results. Per-category breakdown, and a section listing every case the engine missed and why. The failures section is the most credible thing the document will contain.

Where that stands today

No package satisfying all three sourcing criteria has been found in public sources. The failure has a clean shape: both halves exist, and not in the same project. The districts that publish full bid documents online have tended to be running Division 27 work. The districts that ran Division 28 projects hold their bid documents in a business office for in-person review.

So the honest status is not "the benchmark is in progress." It is that the dataset does not exist yet, the scoring code named in the project's own contract document has not been written, and the results file does not exist. When it does, it goes on this site with the misses in it.

A methodology published before the results is a commitment. Published after, it is a defense.

03What it does not read

What it does not read

Several things a reader would reasonably assume are in the box are not. This table is taken from the project's own architecture notes, which were checked against the code on 9 August 2026 rather than read off module names.

Capability and status
Capability	Status	What that means in practice
Drawing sheets	not built	Plan extraction does not exist. There is no rasterization, no scale determination from a scale bar or title block, no tiled extraction and no review payload. The module named for it in the project contract is not in the repository. Every spatial model the engine has ever run on was authored by hand.
Structured claim extraction	built, wired to no command	The one sanctioned model call for reading specification text is written and tested. No command reaches it, and the ingest path never populates the claims field. It is code that has never run on a real package.
Two-vendor agreement	built, wired to no command	The mechanism that reads the same content with two vendors and emits no value where they disagree is written and tested. Nothing outside its own test calls it, and the rule id that would carry a disagreement into the register is defined by no rule pack.
Addendum reconciliation	not built	There is no addendum parser and no diff against the base documents. The delta record is modelled and validated and nothing produces one. An addendum in the package is read as one more document, not as a modification to the others.
Contradiction detection	not built	No rule of that family exists in any pack, and no module compares claims across documents. Quantity mismatch, product mismatch and conflicting responsibility assignment between two documents are not detected today.
Schedule to specification joins	allocated, ships no rule	The rule block for "a device appears in a drawing schedule and every specification reference assigns it elsewhere, or nowhere" is reserved and empty. The schedule-row record is modelled; nothing populates it; no pack defines those ids. This was described in the project contract as the highest-value check in its phase, and it does not run.
Scanned documents	refused	A PDF with no text layer is refused by name rather than silently ingested as empty. Route it through OCR and re-run. The refusal reports every unreadable file at once, so it costs one round trip and not one per file.
A package's own equipment schedule	refused on purpose	Ingestion refuses any path under a schedule folder. A published equipment schedule is benchmark truth for that package, and a system that reads the answer key cannot be scored against it. There is no flag to override this; moving the file is the override, and it leaves a trace.
PDF text and CSI segmentation	built	Reads a package of text-layer PDFs, segments by CSI MasterFormat section numbering, and carries document, page and coordinate provenance on every element. No model call anywhere in it.
Scope marker detection	built, with a measured recall gap	Regex over marker phrases, not a model. The gap described at the top of this page is this component. It has been measured against one real project manual and the result was recorded rather than assumed.
Division 08 to 28 seam analysis	built, partially fed	Reads two kinds of evidence: located scope statements and structured claims. Only the first is populated on a package the tool can currently produce, so half of the seam analysis has nothing to read.

That last row is the one worth sitting with. The Division 08 to Division 28 boundary is the highest-value scope gap in this domain and it is what the engine is aimed at, and today it runs on one of its two evidence sources.

04The numbers

What the numbers mean, and what they are not

Every item in the register carries an exposure band. Here is exactly what that band is.

It is an uncalibrated structural placeholder. Every band in every rule pack the project ships is marked as not calibrated, and that is not a hedge, it is a field in the data. None of them is derived from cost history. They exist so the ranking has a second key to sort on after severity, and that is the entire job they do.

Every rendering marks it. The uncalibrated marking is required on each place a band is printed, so a placeholder can never appear in the same typeface as a derived figure. A placeholder set in the same type as a real number is a confident output, and a confident output where the honest answer is a flag is the specific failure this project treats as unacceptable.

It is an order-of-magnitude planning figure. It is not a quotation, not an offer, not a price, and not derived from a takeoff. It is there to help you decide which three of thirty flags to chase this afternoon.

The deliverable carries no pricing at all. No unit costs, no labor rates, no extended totals. The bill of materials carries quantities and device identities and nothing with a currency on it, and that is enforced in the software rather than left to whoever writes the report.

No exposure figure appears anywhere on this website, for the same reason. Publishing a placeholder band next to a product claim would be using a number as evidence when the number is admitted to be a placeholder.

05What it will not do

What it will not do

Not "has not got to yet." These are things the design refuses, and they will still be refused when the rest of it is finished.

It does not do takeoff.

It does not count devices off a drawing, because it does not read drawings. It does not produce your quantities. If you are looking for something that replaces the count, this is not it and will not become it.

It does not produce a quantity you should contract against.

The bill of materials it emits is supporting evidence for the register, not a purchase list. Every line names the rule that put it there so you can check it. Contracting against a number this tool produced, without checking it, is using it for something it was not built for.

It does not replace the estimator's review.

It does not decide what to bid, what to carry, or what to walk away from. It performs one specific review that gets skipped under deadline, which is the cross-check between documents, and it hands you a list of questions. A person answers them.

An empty result is not a clearance.

A short register means the rules that exist did not fire on the documents that were read. Given the table above, that covers a lot of ground the engine never went near. Read a short register as "these rules found nothing", never as "this package is clean."

The system is built to make that distinction where it can. A derived rule run against a package with no documents loaded raises an error rather than reporting zero findings, because zero findings and a clean package look identical in a report and are not the same thing. A validator run with no spatial model reports nothing rather than reporting a pass, because a validator that passed having examined nothing is indistinguishable from one that looked.

It does not arbitrate between two readings.

Where two independent readers disagree on a field, no value is emitted. Not a value with a warning next to it. There is deliberately no tiebreak, because a tiebreak reduces the flag count without reducing the ambiguity, and a shorter list reads as good news about the documents when it is only a change in how the software behaves.

06The objection

The objection about verification

Said out loud, repeatedly If you're going to have to go through and do the takeoff to verify what the AI spat out, what's even the point of using AI?

The objection is correct about the thing it is aimed at. If a tool produces a count and you have to redo the count to trust it, you did the work twice and paid for software. Nobody should buy that.

It does not land on this tool for one reason, and the reason is a fact about what the output is, not an argument about model quality.

This is not takeoff. It is a review pass.

The output is not a quantity you check against your own quantity. It is a list of places in the documents where two sentences do not line up, each one naming the document, the page and the sentence. There is no number to verify because the engine is forbidden from producing one: the model is allowed to locate a sentence, transcribe it verbatim, classify it and match it, and it is never allowed to compute a count, pick a part number, produce a score, or rule on compliance. Every number in the output is computed by ordinary code from written rules.

What you do with a flag is not a takeoff. You open the cited page, read the sentence, and decide one of three things: it is a real gap and it becomes a pre-bid RFI, it is a real gap you will not get answered and it becomes a line on your qualifications and assumptions, or it is nothing and you close it. That last case is the false positive, and it is the cost.

What it costs, stated plainly

The cost is one open-the-page-and-read-it per flagged line, and it is paid by the estimator, in the week they have the least time. There is no version of this where the flag arrives pre-verified.

How much time that is depends entirely on how many lines come back, and that is the number nobody has. The only run anyone can point to is the repository's own hand-authored twelve-opening fixture, which produces a seven-line register with three items carrying an entity flagged for review. That is a synthetic fixture built to exercise the rules, not a real bid package, and quoting it as an expected workload would be dishonest. On a real package, the register length is unmeasured, which means the review cost is unmeasured too.

So the trade is not "software instead of review." It is: a fixed reading cost per flag, against the cross-check between documents that gets skipped first when the deadline compresses, because it is the slowest part and it usually finds nothing. If your process already runs that cross-check every time and finishes it, this tool has less to offer you than it does to someone whose process does not.

A pre-bid RFI is free. The same question asked after award is a negotiation.

The honest limit on this answer: it assumes the flags are worth reading, and that is exactly what has not been measured. If precision turns out to be poor, the reading cost per useful finding goes up and the objection starts landing. That number is the benchmark, and the benchmark does not exist.

07The other objection

The objection about absence

Said out loud, repeatedly The AI can't tell you what's not there.

True of a model reading one document. A model asked "is anything missing from this specification" has nothing to compare it against and will produce a plausible answer either way.

That is not how absence is found here. It is found by comparing two documents in code and naming both. The mechanism, rather than the conclusion:

How the engine finds an absence

Two sets, built separately. For a given controlled opening, code assembles what the Division 08 sections say about its locking device, its power and its rough-in, and separately what the Division 28 sections say about the same three. Each entry carries the document, the section and the page it came from.
The comparison is set logic, not a question to a model. Nothing asks a model whether the two agree. Code compares the sets, one opening and one aspect at a time.
Five states, and four of them cost money. Both divisions carry the work, which is a duplicate buy. Both carry it and require different things, which is an argument on the day the door is hung. Each division assigns it to the other, which is the seam failure caught in the act. Neither division specifies it, which is the one where somebody eats it. And the fifth: the documents do not settle it.
The finding names both documents. A "neither division specifies this" item cites the Division 08 section that was read and the Division 28 section that was read. Absence is reported as a relationship between two named, cited documents, which is a claim you can take into a pre-bid meeting and put in front of the person who wrote one of them.
Undetermined is a separate finding from unassigned. They are not the same and collapsing them is what makes an estimator stop believing a register. "Neither section covers this" is a conclusion reached after reading both. "The documents do not settle it" is a flag that means look here. The engine has a distinct rule for the second one precisely so it never reports it as the first.
No sections, no finding. If the package contains neither a Division 08 nor a Division 28 specification section, the engine emits nothing at all rather than reporting that neither division covers the work. A flag with no section heading to cite would carry no provenance, and an absence claim with nothing on both sides of it is not evidence.

The limit on this, stated with the mechanism rather than after it: this comparison reads two kinds of evidence, located scope statements and structured claims, and only the first is populated on any package the tool can currently produce. So the seam analysis runs today at half its designed evidence. That is the same row as in the table above, and it is the single largest gap between what this section describes and what runs.

08Next

Where to go from here

/how-it-works/ How it works The design, in plain language. The AI reads and points, code decides, two independent readers must agree, and every line traces back to a page. /security/ Your documents What happens to a bid package that gets processed, who receives text from it, where responses are cached, and what has to be disclosed before anything leaves the machine. / The overview One finding in full detail, the seam this is aimed at, the state of the build, and a longer list of what is not proven. /diagrams/ The same thing, drawn Five diagrams with numbered points, including the pipeline with the unbuilt stages marked.