Who Decides What AI Should Never Do? The Ultimate 2026 Ethical AI Governance Masterclass

The question nobody answers clearly

Ask who enforces speed limits and you get a clean answer. Ask who decides what AI should never do, and you get five different institutions talking past each other while the technology ships anyway.

That’s not cynicism — it’s the structural reality of 2025–2026. AI governance is genuinely multi-polar. Governments write laws on different timelines. International bodies publish recommendations that member states ignore. Corporations sign ethical pledges and quietly walk them back. Academics advise. Civil society litigates. And the systems keep deploying.

Understanding who actually holds which lever matters more than knowing what any single document says. Let’s map it.

The five power centers — and what each one actually controls

Let’s be concrete about what each center actually does — and where it stops.

Governments: The only ones who can fine you

The EU AI Act, which began phased enforcement in February 2025, is the only framework with binding prohibitions backed by real penalties. Banned outright: social scoring by public authorities, real-time remote biometric surveillance in public spaces (with narrow exceptions), AI that exploits psychological vulnerabilities, and systems that infer protected characteristics from biometrics for discriminatory purposes. Established

The US remains fragmented. Executive Order 14110 established AI safety standards for federal agencies but left private deployment largely unregulated. Sector-specific rules (CFPB on credit scoring, FDA on medical AI) create patchwork coverage. State-level laws — Colorado, California, Illinois — are filling gaps the federal government hasn’t touched. It’s messy by design.

UNESCO and ITU: Norm-setters with no enforcement teeth

UNESCO’s 2021 Recommendation on the Ethics of AI — signed by all 194 member states — outlines principles including human oversight, transparency, sustainability, and equity. It’s influential in shaping national frameworks. What it can’t do is sanction anyone. Probable that several signatory governments will incorporate UNESCO principles into binding law by 2027; most haven’t yet.

“For anybody who’s using AI… do they have enough judgment to know when to listen to it and when not to?” — Rembrand Koning, Assistant Professor, Harvard Business School (2025 field study)

Corporations: Self-governance that’s only as strong as the incentive

Google’s original “AI Principles” prohibited weapons and surveillance technology. Then in early 2024 — quietly — that prohibition was removed from public documentation. OpenAI’s usage policies prohibit weapons of mass destruction and CSAM; they also explicitly restrict use for “undermining oversight of AI systems.” Anthropic’s Constitutional AI approach bakes principles into the model’s training rather than just its policies.

These aren’t meaningless. But they’re voluntary, revision-prone, and enforced by the same companies that profit from relaxing them. The incentive structure is the problem. Established

Academia: The best diagnosticians, rarely the decision-makers

Stanford’s AI Index, MIT’s research labs, and Harvard’s AI ethics faculty produce the data that shapes policy debates. HBS’s 2025 study on Kenyan entrepreneurs — covered below — is a good example: rigorous empirical work that reveals structural inequities the governance frameworks haven’t caught up to. Academics advise regulatory bodies, publish widely, and testify in legislatures. They don’t vote on regulations.

Civil society: The alarm system

The ACLU’s challenges to facial recognition in U.S. law enforcement, AlgorithmWatch’s audits of European hiring AI, Timnit Gebru’s Distributed AI Research Institute — these organizations do something the others don’t: they name specific systems, document specific harms, and create political pressure. Most AI regulatory breakthroughs follow a predictable sequence: civil society documents the harm → media amplifies it → legislators act. The Apple Card gender bias case followed exactly this pattern.

Hard bans vs. soft norms — the enforcement gap visualized

This gap is why “51% of AI projects show ethical harms” (McKinsey 2025) and “21.3% growth in AI legislation” (Stanford AI Index) can both be true simultaneously. More laws don’t automatically mean more compliance — especially when the laws with teeth only cover the EU’s jurisdiction.

The judgment gap — the problem nobody’s legislating

Here’s the finding that stuck with me most from the 2025 research cycle. Harvard Business School ran a field experiment with Kenyan entrepreneurs using AI business assistants. The results split cleanly along one variable: prior judgment.

The AI wasn’t broken. The same system, giving roughly the same advice, produced opposite outcomes based on whether the human using it had enough expertise to know when the advice was wrong.

Professor Rembrand Koning’s framing is precise: “Do they have enough judgment to know when to listen to it and when not to?” That question doesn’t appear anywhere in the EU AI Act, UNESCO’s recommendation, or any corporate ethics pledge I’ve read. They focus on what AI does. The HBS study shows that equally important — and harder to regulate — is what the human does with what AI says.

HBS’s Professor Colleen Ammerman raised a related concern: women in the sample used AI tools 10–40% less than men, compounding existing gaps. If the people with least access to AI are also least likely to develop the judgment to use it critically, the equity problem compounds itself quietly. Probable that current governance frameworks underweight this dynamic entirely.

Gray zones where nobody agrees

Hard bans are actually the easy part. Everyone agrees you shouldn’t use AI to run a social credit system in a liberal democracy. The fights that matter happen in the gray zones — applications where reasonable people disagree about whether prohibition is warranted.

Philosopher Michael Sandel’s argument here is worth taking seriously: some decisions require empathy and contextual moral judgment that AI fundamentally cannot provide. Parole decisions, medical triage, child welfare determinations — these aren’t computational problems. When you automate them, you don’t eliminate bias; you crystallize it, make it faster, and strip out the human capacity for mercy. Probable

“The problem with AI in consequential decisions isn’t that the algorithm is wrong. It’s that an algorithm being right 80% of the time means someone — a specific person — is wrongly imprisoned or wrongly denied parole.” — Paraphrase of arguments made by legal scholars including Sandel and others at Harvard Law’s AI and Justice symposia

What this actually means if you’re making decisions about AI

Three things matter more than reading another ethics framework document:

1. Know which jurisdiction’s rules apply to you

If you’re deploying AI that touches EU residents, you’re under the AI Act’s scope whether you’re headquartered in Brussels or Austin. The territorial reach of EU regulation is a fact many non-EU companies are still processing. The highest-risk categories (biometric systems, employment decisions, credit scoring, critical infrastructure) have the strictest requirements and shortest compliance windows.

2. Build judgment into your deployment — not just guardrails

The HBS finding has a practical implication most governance frameworks miss: it’s not enough to have ethical AI if the humans using it can’t critically evaluate its outputs. The question “does our team have enough judgment to know when not to follow this system?” should precede “have we implemented bias auditing?” Both matter, but you can audit your way into false confidence.

3. Assume voluntary pledges will drift toward convenience

Google’s pledge removal. Microsoft’s gradual shift on facial recognition policies. OpenAI’s evolving usage policy language. These aren’t scandals — they’re incentive structures behaving predictably. If your ethics program relies on a vendor’s voluntary commitments as a control, you have a gap. Document which ethical decisions are yours and which you’ve outsourced to a supplier whose interests aren’t perfectly aligned with yours.

FAQ