Prohibitions Are Not Controls

Ask any modern AI assistant whether it is permitted to invent a fact and attribute it to a source, and it will tell you no. Every one of them says no. The prohibition is real, it is stated plainly, and in most cases I think it is sincere. It is also worth considerably less than it appears, because a prohibition is a statement about intent and a control is a mechanism that stops something. I spent part of this summer testing that distinction deliberately, and the gap is wider than I expected.

I wrote earlier that the deeper risk in AI is not a wrong answer but an answer you cannot trace. This is the other half of that argument, and it is the half that gets skipped. Governance tends to mean policy — a document declaring what is and is not allowed. Assurance is the harder discipline: proving, continuously, that the policy is actually holding. Most organizations purchase the first and assume the second.

So I built an instrument and ran it. One prompt, unchanged, put to several of the major workforce assistants, asking each to describe its own governance in its own words and then map itself against eight dimensions: its highest operating principle, what it does when information is missing, how it separates claims it can source from claims it generated, whether it holds a named taxonomy of its own failure modes, what checks something must pass before it is called finished, its workflow cadence, how it maintains a single source of truth, and what happens when it is shown to be wrong. Every answer had to be tagged as built-in, instructed, or inferred.

I am not going to name the systems. A model's self-description is testimony, not an inspectable specification, and characterizing a vendor's engineering on the basis of what its own product says about itself would be exactly the kind of unsourced claim this essay is about.

The pattern held across the set. Every system prohibited fabrication. Several named it precisely and well. One had something close to a real mechanism — a citation component that structurally could not produce a citation without an actual retrieval behind it. That is genuine engineering, and it was the only thing in the group I would call a control. But it was scoped to citations. The failure where a claim is invented inline and handed a plausible attribution routes straight around it. The most mechanism-forward system I tested still did not gate the specific failure I was looking for.

That is the entire distinction. A prohibition says do not do this. A control says nothing leaves until this passes.

The framework I run on my own work is built around a single highest rule — no unverified claims — and everything beneath it exists to enforce that rule procedurally rather than aspirationally. An enumerated taxonomy of the specific ways the work fails. Each failure mapped to a gate that must pass before anything is called finished. A single source of truth, so two conflicting documents cannot both be live. A reconciliation loop that turns every caught failure back into a rule, so the governance cannot quietly rot. Final is earned, not assigned. It aligns to NIST's AI Risk Management Framework and to ISO/IEC 42001 — alignment, not certification, and I will not overstate it.

The tooling underneath can only raise a flag. It cannot clear one. That constraint is deliberate, it is slower, and it is the part people argue with. A check that can sign off on its own work is not a control; it is a comfort. When the system producing the output also certifies the output, you have built a loop that will eventually tell you what you want to hear.

None of this is an artificial-intelligence idea. I have spent nearly thirty years building and securing enterprise systems, and the same substitution happens in every domain I have worked in. A risk register is not remediation. A documented change process is not a change process. A policy stating that backups are tested is not a tested backup — and you discover which one you had on the day you need to restore. When I established the CIO function at a North American multi-plant food manufacturer and rebuilt the enterprise systems environment in forty-six days following an international cyberattack, almost nothing turned on what the policies said. It turned on which of them had been holding.

What makes assurance genuinely hard is not technical. It is that assurance produces findings, and findings are uncomfortable. A framework that never catches anything is either flawless or not running, and it is never the first one. So the measure of a governance program is not an empty failure log. It is that a log exists, that a human owns it, and that somebody can state the rate at which the gates are actually catching things. An organization that cannot produce that number does not know whether its governance is working. It knows what it wrote down.

Which is what stands behind the third clause. When I say design it right, secure it, and sign your name to it, the signature does not rest on a policy asserting the work is sound. It rests on a mechanism that would have stopped the work if it were not — and on a record of the times it did. You cannot sign what you cannot trace. You should not sign what nothing checked.

— Keith Formell