AI’s Missing Black Box: Why Failed Experiments Vanish — and Who Keeps It Quiet

Failed AI experiments rarely die with a press release. They die by demo shutdown, reassignment, and contract — an ordinary, decentralized machine, not a coordinated one, that leaves no postmortem and no public record. It mostly works. It doesn’t always: whistleblowers, leaked documents, and an SEC complaint are the reason we know any of this at all.

TL;DR

• The mechanism isn’t a conspiracy — it’s decentralized incentives, publication bias, and ordinary contract law, which is exactly what makes it durable and hard to fix.
• Until 2024, OpenAI asked departing employees to sign lifetime non-disparagement agreements enforced through already-vested equity. Vox exposed it. OpenAI’s 2024 “we removed the clauses” statement came with an asterisk that most coverage missed: obligations were lifted “except where the provision was mutual” — meaning some agreements simply weren’t released.
• Meta’s Galactica lived online for three days in November 2022. No public postmortem ever followed — the ordinary fate of a killed AI project; this one just briefly happened in public.
• The UK’s AI Security Institute found universal jailbreaks in every frontier system it tested — yet it cannot compel access to a model a lab doesn’t want to show it.
• Even the best-documented lab, per SaferAI’s 2025–2026 audit, scores 34% on risk-management transparency. The median across twelve major labs is 18%.

---

On November 15, 2022, Meta AI released Galactica, a 120-billion-parameter model trained on 48 million scientific papers, pitched as a new interface to human knowledge. Within two days, researchers found it fabricating citations, inventing plausible-sounding falsehoods, and generating wiki articles on nonsense topics with the same fluent confidence as real science.

By November 17, the demo was offline — Meta’s chief scientist Yann LeCun said publicly it was pulled because people were “casually misusing it.” The paper stayed on arXiv. The internal review that had cleared the launch, and any postmortem explaining what it missed, did not.

Galactica is the rare case where a failure happened loudly enough to leave a paper trail. This article is about the far larger set of cases that don’t: not what AI labs build in secret, but what they kill in secret, and the specific, legal, almost boring machinery that makes those deaths invisible.

It’s not a conspiracy. It’s worse.

The instinct, faced with a pattern like this, is to reach for “secret lab” framing — some coordinated effort to hide what AI companies are really doing. That’s wrong, and it’s worth saying why before the case studies, because the case studies only make sense once you see the mechanism they’re evidence for.

There is no smoky room. No one is steering this industry-wide. Three ordinary forces do the work, running independently at every lab at once:

Survivorship bias at industrial scale. Science already rewards positive results over negative ones. Frontier labs amplify that by orders of magnitude: a failed experiment consumed real compute, involved researchers who could’ve been shipping something else, and — if disclosed — hands competitors a map of dead ends to avoid. Every incentive points toward burying the failure and publishing the paper about what worked. Galactica’s paper is still live. Its postmortem never existed publicly.

Contractual capture of human memory. Researchers at frontier labs disperse constantly — into rival labs, government institutes, safety nonprofits — carrying knowledge that exit paperwork discourages them from sharing. The next section covers this in detail, including the asterisk in that last TL;DR bullet.

Voluntary-only oversight. The institutions best positioned to reconstruct the hidden picture — government evaluators — can only see what labs choose to show them, and those grants of access increasingly come bundled with partnerships. The section on regulators below covers what this looks like in practice, numbers included.

A conspiracy would have an address you could subpoena. This has none — each lab’s silence is individually rational, and the aggregate is a field flying without a black box. Aviation made the opposite bet a century ago: the NTSB gets the wreckage, and every crash makes the next flight safer. AI’s crashed projects are, contractually, trade secrets.

The strongest counterargument deserves to be stated plainly, not waved off: openness has real costs. Model weights and training methods are legitimately protectable IP. Full publication of failure data would inform bad actors along with good ones. Labs’ caution about publishing biosecurity-relevant findings — AISI itself redacts dangerous detail before release — is arguably responsible, not sinister. That argument wins for capability information. It loses for governance information: there is no competitive rationale for gagging a departed employee’s safety criticism, and no rationale at all for never publishing a postmortem on a demo that already failed in public. Protect the code, free the criticism — the line the 2024 whistleblowers drew is the right one, and the industry crossed it for years.

The anatomy of a vanishing

When a lab abandons a project, there’s no funeral — just a predictable, five-stage sequence at minimizing what leaks:

1. Internal evaluation. Benchmarks and red-team results stay proprietary.
2. The decision gate. Competitive pressure usually decides “not ready” versus “dead” — Galactica’s failure landed exactly when Meta could least afford a credibility gap in LLMs.
3. The shutdown. Demos come down, access is revoked, blog posts are quietly edited or left standing as fossils.
4. The dispersal. Researchers are reassigned; Slack threads and internal wikis become orphaned institutional memory.
5. The seal. Employment agreements ensure whatever the team learned — including why it failed — stays inside the company, covered next.

(This diagram renders as a self-contained dark card with its own background, so it won’t clash with a light page theme — but preview it against your actual template before publishing, same as any embedded graphic.)

Machine learning, like any experimental science, advances on postmortems. The field’s knowledge of what doesn’t work lives disproportionately inside companies with every incentive never to publish it. Galactica is instructive precisely because the world got three days of access to a failure — and Meta still never published why its internal review cleared a model that public testers broke within hours.

The contract layer: how departures become permanent silence

This is the most thoroughly documented piece of the machinery, because the documentation comes from the labs’ own former employees — and it contains the article’s single most-missed detail.

For years, OpenAI asked departing staff to sign agreements binding them — for life — never to criticize the company, with the agreement itself covered by a non-disclosure clause: acknowledging it existed was a violation. The leverage was already-vested equity, often worth millions. Kelsey Piper’s May 2024 reporting at Vox exposed the documents. OpenAI’s response was to state it was removing the clauses from standard paperwork and releasing former employees from existing ones — except where the provision was mutual. That carve-out matters: a mutual non-disparagement clause (both parties agree not to criticize each other) sounds symmetrical and reasonable, but it still means the employee can’t speak, and OpenAI’s own statement doesn’t say how many of the estimated 500-plus signed agreements were mutual versus one-sided. Most coverage of the 2024 “fix” repeated the headline and skipped this line.

An independent LessWrong tracking effort put that signatory estimate at over 500 former staff, with only a handful publicly confirmed released as of mid-2024 — an estimate the tracker itself flags as necessarily approximate, since a self-concealing gag is unfalsifiable by design.

Two employees put faces on it. William Saunders, a researcher on OpenAI’s superalignment team, resigned in February 2024, signed to preserve his equity, then spoke to TIME anyway, saying he risked losing access to stock worth millions by doing so. Daniel Kokotajlo quit in April and refused to sign, forfeiting an estimated $1.7 million. In June 2024, they and eleven other current and former OpenAI and DeepMind employees published a right-to-warn open letter, asking labs for protected channels to warn regulators, boards, and the public — endorsed by Yoshua Bengio, Geoffrey Hinton, and Stuart Russell.

Two months later, whistleblowers filed an SEC complaint alleging OpenAI’s agreements illegally restricted employees from warning regulators and forced them to waive federal whistleblower compensation rights.

Anthropic ran its own version. According to AI Lab Watch — a lower-authority secondary tracker, flagged as such because it isn’t a primary document — some Anthropic severance agreements included a non-disclosure clause covering the non-disparagement clause itself. Anthropic has since publicly stated it never tied non-disparagement to vested equity and has removed the practice; the point here isn’t that Anthropic’s version was as severe as OpenAI’s, but that the reflex to legally seal a departure is industry-wide, including at the lab built on safety credibility.

None of this is exotic to AI. Business Insider’s 2021 review of 36 tech-worker NDAs — spanning Fortune 500 firms and startups — found every separation agreement examined contained both nondisclosure and non-disparagement clauses. AI labs didn’t invent this machinery; they aimed it at a category of work whose failures can carry public-safety weight, which is the important distinction. Trade-secret protection for weights and training methods is a legitimate interest. Gag clauses attached to criticism, enforced through equity employees had already earned, are a different thing — and that’s what the 2024 reporting actually exposed.

The projects that died — mostly — in the dark

The scope here isn’t limited to technically failed experiments — a scrapped defense contract or a canceled censored-search product got killed by the same organizational reflex to seal internal deliberations that kills a failed model, so they belong in the same pattern of institutional secrecy, even though the mechanism triggering the kill differs. The table below is deliberately modest — strong reporting only, not an exhaustive graveyard.

Project / program	What happened	What was never publicly released
Meta Galactica (Nov 2022)	Public demo pulled after 3 days amid hallucination and bias findings	The internal launch review, and why it cleared a model public testers broke in hours
Google Project Maven (2018)	Pentagon contract not renewed after ~4,000 employees petitioned and a dozen engineers resigned	The internal deliberations behind the reversal — a contract decision, not a technical postmortem, but sealed the same way
Google Dragonfly (2018–19)	Censored-search project for China scrapped after leaks; employees were asked to accept verbal NDAs before being briefed	The project’s internal review documents and the deliberations behind them
Amazon Rekognition (2020–)	Police-use moratorium announced June 2020, still in effect in 2021 after two years of bias research	Demographic error-rate data; why internal review didn’t flag it sooner
OpenAI Superalignment (May 2024)	Team dissolved after Ilya Sutskever and Jan Leike resigned the same week; had been promised 20% of compute	Whether that compute commitment was actually delivered

Dragonfly is worth a closer look because the secrecy mechanics were documented by the employees living through them. According to BuzzFeed News’ reporting on internal posts, one engineer was asked to accept a verbal NDA before his own VP would brief him on a project he might be reassigned to; he transferred out instead. A colleague wrote that he planned to leave Google altogether, in part because he’d been asked to keep Dragonfly secret — not from the public, but from his own coworkers. Managers restricted internal code access, a sharp break from Google’s famous day-one codebase policy.

Then there’s the case where it wasn’t a project that vanished but the research about one. In December 2020, Timnit Gebru, co-lead of Google’s ethical AI team, left after conflict over a draft paper — later published as “On the Dangers of Stochastic Parrots” — examining risks in the large language models Google was racing to build; Gebru said Google forced her out, while Google’s head of AI maintained internally that she had set conditions for staying that the company wasn’t willing to meet. What’s undisputed: the paper exists publicly only because Gebru’s co-authors published it after she left. The far more common outcome — an internal review that simply shelves a paper questioning a flagship direction, with no controversy loud enough for anyone outside to hear about it — is the one this article is actually about.

Amazon Rekognition shows the pattern from the outside in. Researchers including Joy Buolamwini and Inioluwa Deborah Raji published findings of dramatically higher error rates on darker-skinned women than on lighter-skinned men; NIST later found similar demographic differentials industry-wide and noted Amazon hadn’t submitted its algorithms for testing. IBM quit facial recognition outright in June 2020; Amazon announced a one-year police moratorium, later extended indefinitely. The error-rate data advocates spent two years demanding remains unpublished by Amazon to this day. The experiment didn’t vanish. Its accountability did.

What the regulator can see

If contracts seal failures internally, oversight bodies exist to look from the outside — and the numbers here cut both ways.

The UK’s AI Security Institute (renamed from AI Safety Institute in February 2025) has evaluated frontier models since November 2023. Its first Frontier AI Trends Report, published December 18, 2025, aggregates two years of testing across more than thirty systems:

• Every frontier system tested had at least one universal jailbreak — no model offered categorical protection against a determined attacker, though jailbreaking one model for biological-misuse content took roughly 40 times more expert effort than jailbreaking its predecessor six months earlier.
• Self-replication capability climbed sharply on AISI’s RepliBench suite — still unlikely to succeed in real-world conditions, since models struggle to acquire and hold new compute access, but improving quickly. AISI is explicit that there’s “not yet evidence” of models attempting this spontaneously, as opposed to when prompted.
• Autonomous cyber-task length has roughly doubled every eight months — a trend echoed in independent METR research on software-development tasks.

The catch the report can’t fix: all of it is voluntary. As the Ada Lovelace Institute documents, AISI has no legal route to compel access to an unreleased model, cannot require pre-release safety standards, and cannot fine or suspend a harmful model — visibility rests on labs maintaining, in the Institute’s own words, “warm relationships” with the evaluator. AISI’s independence gets more complicated as its partnerships deepen: a December 2025 memorandum with Google DeepMind bundles evaluation access into a broader government collaboration, raising a real question about how sharply an evaluator can assess a company it’s simultaneously partnering with.

The clearest evidence of what stays hidden comes from outside auditors, not regulators. SaferAI’s 2025–2026 assessment scored twelve frontier labs’ published safety frameworks against 65 weighted criteria. Overall transparency scores ranged from 8% to 34%, with a median of just 18%. Anthropic led at 34%, OpenAI close behind at 33%, then G42 at 24% and Meta at 21%; eight of twelve providers scored 20% or below, and Cohere scored lowest at 8%. No provider in the assessment publishes severity or probability scores for its own risk models. The authors are careful about the limitation: they assessed documents, not practice, so a lab’s actual internal process could be stronger — or weaker — than what it discloses.

It’s worth sitting with what even the top score means. Anthropic, the highest-scoring lab, still fully or partially meets barely a third of the criteria SaferAI considers best practice for its own published risk framework. That’s not evidence the mechanism described in this article is airtight — if anything, a 34% ceiling under external pressure to score well suggests the system is porous even at the top, where the incentive to disclose is strongest. It’s evidence that “porous” and “hidden by design” aren’t opposites; the same lab can be the industry’s most transparent and still fail two-thirds of an external transparency rubric.

What would actually change this

The fixes are known and unglamorous, and none require labs to become charities:

1. Mandatory incident reporting, aviation-style. Labs training above a compute threshold report capability-overshoot events, severe pre-release jailbreaks, and killed-deployment postmortems to a regulator — redacted for biosecurity, but on the record. Attaching the duty to the company, not the employee, neuters the NDA problem at the source.
2. Non-waivable whistleblower protections. US law already protects disclosure to regulators; making that non-waivable in employment agreements, as the 2024 SEC complaint effectively demanded, converts “right to warn” from a request into a default, and would close the mutual-clause carve-out described above.
3. Pre-registered evaluations. Labs registering safety thresholds before training runs would make a quiet shift in those thresholds — the superalignment team’s undelivered compute promise being the canonical grievance — externally detectable instead of reconstructable only from departed employees’ public statements.
4. Independent evaluation with statutory teeth, funded at arm’s length from the labs being evaluated, so the December 2025 pattern — evaluation access bundled into a commercial partnership — becomes a conflict of interest to disclose rather than a business model.

None of this is close to guaranteed, and the near-term regulatory picture is uneven: the US remains voluntary-framework dependent; the EU AI Act’s obligations phase in by risk tier; compute-threshold incident reporting doesn’t exist anywhere in force yet. The realistic path is incremental — each public whistleblower event moves the Overton window a few feet, and the window is where regulation eventually gets written.

If you want to track that motion directly rather than wait for the next viral departure, three things update on a predictable cadence: AISI’s Frontier AI Trends Report, SaferAI’s framework assessment (labs revise roughly annually), and the LessWrong non-disparagement canary list, which grows by exactly one name every time a gagged employee decides equity isn’t worth the silence. The secret labs were never the story. The paperwork was.

Myth vs. fact

Myth	Fact
“There’s a secret lab hiding a rogue AI.”	No evidence supports this. What’s documented is mundane: NDAs, reassignments, and unpublished internal reviews — legal secrecy, not a hidden facility.
“OpenAI removed its non-disparagement agreements in 2024, full stop.”	It removed them “except where the provision was mutual” — a carve-out its own 2024 statement included and most coverage didn’t mention.
“Government AI safety institutes can force labs to show them anything.”	AISI evaluates only what labs voluntarily provide. It cannot compel access to unreleased models, mandate pre-release standards, or fine or suspend a harmful model.
“The best-scoring lab on safety transparency is basically transparent.”	Anthropic’s top SaferAI score is 34% — meaning even the industry leader fails roughly two-thirds of the external transparency criteria assessed.
“AI jailbreak defenses are basically solved now.”	AISI found a universal jailbreak for every frontier system tested as of December 2025. Defenses have improved unevenly, not disappeared.

Glossary

• Non-disparagement agreement — A contract clause barring criticism of a company, sometimes for life, often enforced through financial leverage like already-vested equity.
• Mutual non-disparagement clause — A version where both parties agree not to criticize each other. Sounds symmetrical; in practice it still silences the employee, who typically has far less to say about the company than vice versa.
• Canary tracking — An informal method, used by trackers like the LessWrong non-disparagement project, of inferring how many people are bound by a gag clause by counting how few have publicly confirmed being released from one.
• Universal jailbreak — A technique that reliably bypasses an AI system’s safety filters across a wide range of harmful requests, rather than working only in narrow cases.
• RepliBench — AISI’s evaluation suite testing whether a model can autonomously replicate — acquiring compute or copying its own weights — without being explicitly instructed to.
• Frontier AI Trends Report — AISI’s first public aggregation of two years of frontier-model evaluation data, published December 18, 2025.
• Safety framework transparency score — A rating, like SaferAI’s, of how much of a lab’s published risk-management documentation is specific and public — not a measure of how safe its models actually are in practice.

FAQ

Did OpenAI actually stop using non-disparagement agreements?
It stated in 2024 it was removing the clauses from standard departure paperwork and releasing former employees from existing obligations — except where the clause was mutual, a carve-out its own statement included.

Can a government AI safety institute force a lab to hand over an unreleased model?
No. AISI evaluates models labs voluntarily share. It cannot compel disclosure of unreleased systems, mandate pre-release standards, or fine or suspend a harmful model.

Is there a jailbreak that works on every current AI model?
As of AISI’s December 2025 report, every frontier system tested had at least one universal jailbreak, though the effort required varies enormously by model and harm category.

How transparent is the best AI lab, really, on safety risk management?
Per SaferAI’s 2025–2026 assessment, Anthropic scores highest of twelve labs at 34% — the median is 18%. Even industry-leading transparency means most best-practice criteria go unmet in what’s published.

Did any AI lab’s killed project get a public postmortem?
Not in the documented cases here. Meta never explained why Galactica’s internal review cleared a model public testers broke within hours; Google hasn’t published the Maven or Dragonfly deliberations; Amazon hasn’t published the demographic error-rate data behind Rekognition’s ongoing moratorium.

Are AI labs legally required to disclose failed or dangerous experiments?
No binding requirement exists yet in the US or UK. The EU AI Act imposes phased, risk-tier obligations, but a general duty to report killed deployments or near-miss safety incidents doesn’t currently exist anywhere in force.

---

---