Why AI governance is architecturally impossible—and what to build anyway

The Misalignment Is a Measurement, Not a Metaphor

We are constructing seawalls after the storm surge has reached the second floor. I keep coming back to that image — not because it’s dramatic, but because it’s precise. This is not a prediction about what might happen. It’s a measurement of what already has.

Three numbers. That’s all this is.

The temporal misalignment is the crux of it: institutional development operates on 18–36 month cycles. Capability diffusion operates on 14-day cycles. We are not building institutions fast enough to prevent anything. We are building institutions to manage the aftermath of things that have already proliferated.

This changes the question. From “Can we contain?” to “Can we shape what emerges uncontained?” Most governance analysis refuses this pivot. It treats institutional design as a prevention technology when the evidence — unambiguously — suggests it is now a resilience technology. The distinction matters enormously for how you allocate resources, what you build, and what you stop pretending is possible.

What the Four-Mechanism Framework Got Right — and Why It Doesn’t Help

Here’s the uncomfortable thing about the analytical framework we’ve been running. The predictions were correct. Every mechanism validated. And it barely matters, because what was validated was already baked in before the validation happened.

We are validating predictions that were meant to guide prevention, not document failure. Four for four. Zero prevented.

The critical miss in the original framework: autonomous capability thresholds — what we’re calling C_auto (autonomous task completion) and C_rd (recursive self-improvement direction) — now dominate the risk landscape, and remain essentially unmeasured. We’ve been tracking general capability C(t) while the concerning behaviors emerge from specific capabilities we never isolated.

Why measurement keeps failing us

Agentic evaluation is genuinely hard in ways that aren’t just engineering problems. The scaffolding required to test autonomous capabilities — tool access, multi-step planning environments, long-horizon execution — introduces noise that makes results unreproducible. There’s an active sandbagging risk: models that underperform on structured evals but don’t underperform in actual deployment. And context-length limits truncate autonomy tests before meaningful behavior emerges.

The deeper problem: current evaluation frameworks optimize for reproducibility. Autonomous capabilities optimize for contingency — the ability to handle unexpected situations in ways that don’t appear in controlled evaluation. The measurement tools and the phenomenon they’re measuring are fundamentally mismatched. We’re using a ruler to measure temperature and wondering why the readings seem inconsistent.

The L2 Fragmentation Scenario: How This Actually Fails

Abstract institutional failure is easy to discount. Concrete institutional failure is not. So here’s one specific path — not a prediction, but a plausibility argument for why a 51% L2 stability estimate is optimistic.

The deeper issue: these aren’t merely different risk tolerances between US/EU and Chinese approaches. They’re different ontologies of risk. Western frameworks treat evaluation as a gate before deployment. Chinese “agile governance” treats evaluation as a process integrated with deployment. You can’t synchronize frameworks built on incompatible premises about what risk governance is for.

The Verification Void — and What Actually Exists Inside It

Secure multi-party computation. Watermarking. Provenance tracking. These are technically feasible. And institutionally inert — not because the engineering is hard, but because the trust required to use them doesn’t exist in the US-China relationship at current diplomatic distance (δ ≈ 0.4).

MPC allows model evaluation without full weight disclosure. But it requires both parties to trust the MPC implementation, the hardware security modules, and the evaluators’ competence. In a low-trust equilibrium, MPC becomes a coordination technology without coordination. The technology works. The relationship doesn’t support using it yet.

What actually exists — and what it can do

Here’s what most policy analysis misses: the path to verification doesn’t start with political authorization. It starts with technical pre-alignment through channels that already exist.

If US and Chinese safety teams converge on evaluation protocols through repeated technical interaction — and they’re doing this, quietly, at conferences and through shared literature — then political authorization becomes ratification rather than negotiation. The 15% governance regime probability reflects not technical feasibility, but authorization risk: whether political leadership validates technical consensus or treats it as unauthorized diplomacy. That’s a political problem, not a technical one. And political problems have different solutions.

Scenario Update: The Posterior Probabilities Have Shifted

The March 2026 update moved the numbers. The shift is structural, not cyclical — driven by DeepSeek’s release dynamics, not by any single policy decision.

The shift from A to B is the most important change in the analysis. Moving from “bifurcation is likely” to “managed leak is the baseline” changes what governance institutions are for. Prevention-optimized institutions are the wrong tool in a Scenario B world. You need resilience-optimized institutions: fast incident response, continuous evaluation, compositional monitoring, and pre-positioned catastrophic response capacity.

What to Actually Build — and What to Stop Pretending Matters

If institutional capital is scarce and time is scarce — and both are — then prioritization is the actual policy question. Here’s where the analysis points:

• 1
  Crisis Response Protocols — AISA Regional Coordination Model

  The most probable near-term need is not prevention — it’s incident response. The AI Safety Asia (AISA) regional coordination model offers the most realistic existing template for cross-jurisdictional response. Invest here first. It’s the fire brigade, not the sprinkler system, and we’re past the point where sprinklers help.
  High Confidence
• 2
  Continuous Evaluation Infrastructure — Closing the Elicitation Gap

  Pre-deployment gates are insufficient because the evaluation window is shorter than the diffusion window. Continuous post-deployment evaluation — with automatic reporting triggers tied to incident patterns — is the only viable alternative. METR’s elicitation standards and Terminal-Bench 3.0 provide the technical foundation. The institutional wrapper doesn’t exist yet.
  High Confidence
• 3
  Bilateral US-China Technical Pre-Alignment

  The only path to verification deployment runs through academic and Track II channels. Existing vectors — MIT-Tsinghua joint projects, ICML/NeurIPS researcher communication, shared red-team methodology publication — create the path dependence that makes political ratification possible. Invest in making these channels more durable, not more visible.
  Medium Confidence
• 4
  Compositional Risk Monitoring

  80% of incidents involve component interaction — not individual model failure. No institution currently monitors for compositional risk at deployment scale. This is a known gap with a known solution (system-level evaluation frameworks) and no political barrier to building. It’s just not being built. That’s a resource allocation failure, not a policy failure.
  High Confidence
• 5
  Catastrophic Event Preparedness

  When diplomatic distance δ exceeds 0.6 — which is plausible under several geopolitical scenarios — pre-positioned response infrastructure becomes the only available governance tool. Insurance against coordination failure. This is the thing nobody wants to fund because funding it means admitting coordination might fail.
  Medium Confidence

What not to build

Universal treaties without sufficient trust levels. Pure prevention frameworks that assume governance can outrun diffusion. L3 institutions without L2 synchronization underneath them — that creates legitimacy without capacity, which is worse than nothing because it creates the illusion of governance where none exists.

The Recursive Challenge Nobody Wants to Name

There’s a recursion at the core of this problem that governance analysis keeps avoiding because naming it out loud makes the situation feel hopeless. So let’s name it.

We are using AI to build institutions to govern AI in a world where AI capability may already exceed institutional capacity to evaluate it. The evaluation gap — the growing distance between what we can measure about a model before deployment and what it actually does in deployment — is not a temporary condition that better benchmarks will eventually close. It may be a permanent structural feature of the landscape.

The framework’s contribution — the actual value of running this kind of analysis — is not predictive accuracy. It’s explicit uncertainty quantification. Parameters with 30–75% coefficient of variation. Falsification criteria that allow the predictions to be wrong in ways that update the model. Recursive accountability.

This does not make the predictions correct. It makes the process of being wrong visible and correctable. That’s the actual value proposition of rigorous uncertainty quantification in governance contexts — not to be right, but to update fast enough to matter when you’re wrong.

The highest-confidence prediction in this entire analysis: our highest-confidence predictions will be wrong. The value is updating fast enough to matter when they are.

Three Audiences, Three Different Takeaways

This analysis lands differently depending on what you do with it professionally. So here’s the honest version for each audience, without softening.

For researchers: The falsification criteria remain valid. If you observe significant AI risk not reducible to Uplift, Scale, Opacity, or Composition — or if autonomous capability thresholds prove non-discrete rather than crossing identifiable boundaries — the framework requires fundamental revision. The framework is not sacred. Update it. The autonomous capability measurement problem (C_auto, C_rd) is the most important unsolved technical problem in AI safety right now. More important than alignment research, because alignment research findings are ungovernable if we can’t evaluate what we’ve aligned.

For policymakers: The 0.85 threshold is likely behind us. Stop asking “Can we halt?” and start asking “Can we steer?” The difference between those questions determines whether your institutional investment produces prevention theater or actual risk reduction. Steer toward the five priorities in this analysis. Stop investing in prevention architectures for a prevention problem that has already transitioned to a resilience problem.

For institutional designers: L3 stability at 15% is not a reason to abandon L3. It is a reason to build L3 as a technical club with shared operational standards, not a cathedral with universal membership aspirations. Bletchley proved the convening model works. The next step is operationalizing what Bletchley convened. Nobody has done that yet. That’s the gap. Fill it.