Provenance Is the Point

Ask most people what makes artificial intelligence risky and they will describe a wrong answer — the confident, fluent, plausible response that turns out to be false. That is the visible failure, and it is the wrong one to fear most. The deeper risk is quieter: an answer you cannot trace. A system that hands you something useful and leaves you no way to know where it came from, what it was built on, or whether it can be trusted enough to act on. A wrong answer you can catch. An unaccountable one you cannot.

Governance is what closes that gap — not as a brake on the technology, but as the thing that lets you put weight on it.

I did not arrive at this through AI. I arrived at it through more than twenty-five years of building and securing enterprise systems — across public-sector law enforcement, national manufacturing, SMB consulting, and my own ventures — where the unglamorous center of the work was almost always the same question: where did this come from, and which system is the source of truth? Reconciling conflicting records into one architecture you could stand behind was the job long before anyone called it AI governance. The discipline is old. Only the system is new.

So I don't treat AI governance as a slide I present. I treat it as a system I run — on my own work, first. Every significant output I produce with AI passes the same discipline I would install for anyone else: inputs are gated, claims are checked against their sources, and nothing is allowed to be called finished on confidence alone. I keep a standing review that turns the same scrutiny back on the rules themselves, so the governance cannot quietly rot. And the tooling I trust is the kind that can only raise a flag — never clear one. A check that can sign off on its own work isn't a control; it's a comfort.

That posture begins by refusing the most common mistake: treating AI as a novelty, a clever assistant off to the side, exempt from the rules every other production system has to follow. The moment its output informs a decision, it is a production system, and it earns the same scrutiny — documented inputs, a known source of truth, version control, and a record of how it reached what it reached.

This is the half that gets skipped. Governance tends to mean policy — a document declaring what is and isn't allowed. Assurance is the harder part: proving, continuously, that the policy is actually holding. It means keeping two questions apart that people love to collapse into one. How much do I trust this particular answer? is not the same as how reliable is the process that produced it? — and a single “confidence” number hides both. Pull them apart and you can see what you are standing on. Collapse them and you are guessing with a decimal point.

The failure modes are not exotic. A fabricated citation that reads exactly like a real one. Retrieval or training data nobody governed, quietly shaping every answer. A model whose behavior drifts while everyone assumes it is steady. None of these announce themselves; they surface as confident output — which is precisely why provenance has to be built in rather than bolted on afterward. You cannot inspect for a problem you have given yourself no way to see.

And like every security decision, this one is finally about appetite, not architecture. How much an organization is willing to trust a system it cannot fully audit is a choice leadership makes — consciously, or by default. The governance follows the answer. My only argument is that the answer should be made out loud, by someone willing to own it.

Which is the whole of it, really. I use AI every day, in my own practice and across my ventures. I don't let it sign. The judgment, the accountability, the name on the work — those stay human, because they are the part no system can hold for you. The standard I keep returning to is the same one I have applied to every system I have ever built: design it right, secure it, and sign your name to it. AI doesn't change the standard. It only raises the cost of skipping the signature — and you can only sign what you can trace.

— Keith Formell