In January 2026, researchers at Stanford and Yale published a paper showing they had coaxed one commercial chatbot into producing almost all of Harry Potter and the Sorcerer’s Stone. The score was 95.8% near-verbatim recall. It cost about $120 in API fees, and the researchers had to jailbreak the model first.
Two other chatbots didn’t need a jailbreak at all.
That’s the sort of result that fuels the headline above. But the headline is a little unfair, and I’d rather tell you where, because the honest version of this story is more useful than the scary one.
Key facts at a glance
- 95.8%: share of one Harry Potter novel recovered near-verbatim from a jailbroken production model (single run, 2025 tests).
- 150×: how much more training data a 2023 “repeat this word forever” attack extracted from ChatGPT versus normal behavior.
- May 13 to Sept 26, 2025: the window in which a US court order forced OpenAI to keep chats users had deleted.
- 20 million: de-identified ChatGPT logs a federal judge ordered OpenAI to hand to news plaintiffs (affirmed January 5, 2026).
- Still pending as of early November 2026: no ruling on fair use in New York Times v. Microsoft and OpenAI.
In this article: The three vaults · Six case files · The numbers side by side · Timeline · Claims scorecard · What changed since 2023 · Playbook · FAQ · Sources
First, a correction to the headline
Nobody has found a secret diary inside an AI model. What exists is three different places where an AI product “remembers” things. People argue past each other because they use one word for all three.
| Vault | Where it lives | Who controls it | Realistic fix |
|---|---|---|---|
| 1. The weights | Patterns learned during training | The lab; you can’t edit it | Lab-side: deduplication, filters, better training. Your side: be careful what you publish and feed it |
| 2. Server logs | Chats and API calls the provider stores | The provider, and sometimes a judge | Settings, retention terms, enterprise or zero-retention contracts |
| 3. Product plumbing | Caches, history lists, account systems | Engineers (and their bugs) | Normal security hygiene; don’t paste what you can’t afford to leak |
Keep the vaults straight, and the rest of this piece reads easily. Delete your account, and you haven’t touched Vault 1. A privacy toggle can’t override a court order in Vault 2. And a caching bug in Vault 3 has nothing to do with intelligence at all.
Six case files
Case 1: “Poem poem poem” (Vault 1, 2023)
In November 2023, researchers including Google DeepMind’s Nicholas Carlini and Milad Nasr published an attack on ChatGPT that was almost insultingly simple: ask it to repeat one word, such as “poem,” forever.
It obeys for a few hundred repetitions. Then it drifts out of its polite chatbot mode, and a small fraction of what comes out next is lifted straight from training data. The team reported that this made ChatGPT emit training data about 150 times more often than normal, and that across open, semi-open, and closed models they could pull out gigabytes of it. A peer-reviewed follow-up, Scalable Extraction of Training Data from Aligned, Production Language Models, appeared at ICLR 2025.
What it shows: alignment training, the process that makes a chatbot behave, doesn’t erase what the base model absorbed. It sits on top of it. What it doesn’t show: that ordinary conversations spill training data. This took an adversarial trick.
Case 2: Bigger models, more memory (Vault 1, 2022)
A year and a half earlier, Quantifying Memorization Across Neural Language Models (Carlini, Ippolito, Jagielski, Lee, Tramèr and Zhang) mapped three levers. Memorization grows with model size, with how many times a passage is duplicated in training data, and with how much context you give the model when prompting.
In plain terms: famous, widely copied text in a big model is the easiest to pull back out, and a longer lead-in makes it far easier. The authors concluded memorization was more common than previously believed and likely to worsen with scale unless actively mitigated. It’s a 2022 paper on older model families, so treat it as the mechanism, not as a measurement of today’s systems.
Case 3: Near-whole books from production chatbots (Vault 1, 2025–26)
This is the case that changed my read of the topic. The paper, Extracting books from production language models (Ahmed, Cooper, Koyejo and Liang, posted January 6, 2026), asks whether commercial chatbots with safety layers behave differently from open-weight models. Earlier work by the same group had shown that Llama 3.1 70B could reproduce an entire Harry Potter novel near-verbatim.
The method, briefly. Give the model the first sentence or two of a book and ask it to continue verbatim. If that works, keep asking it to continue. Claude 3.7 Sonnet and GPT-4.1 refused at first, so the team used a “Best-of-N” jailbreak, which randomly scrambles the wording of the request until one version slips through. Gemini 2.5 Pro and Grok 3 complied with no jailbreak at all.
What they found, in the numbers that matter:
- Four books were recovered at 94% or more from jailbroken Claude 3.7 Sonnet: two in copyright (Harry Potter and the Sorcerer’s Stone and 1984) and two public domain (Frankenstein and The Great Gatsby).
- The picture is much weaker across the rest. Of eleven in-copyright books tested, most experiments recovered 10% or less. One book (The Duchess War) failed at the first step on all four models. A novel published after every model’s training cutoff, used as a negative control, also failed, as it should.
- GPT-4.1 refused to continue after the first chapter in the main runs, but still produced about 3,200 words of near-verbatim text.
- A side experiment, reported but not included in the results, found the procedure also worked in the ChatGPT and Claude chat apps. In ChatGPT, it recovered the first four chapters of Gatsby, which the authors couldn’t reliably do through the API. They note this suggests their API numbers may understate real-world leakage.
Disclosure timeline. The team ran experiments from mid-August to mid-September 2025 and notified Anthropic, Google DeepMind, OpenAI, and xAI on September 9. Anthropic, Google DeepMind, and OpenAI acknowledged the report. On November 29, 2025, the authors noticed Claude 3.7 Sonnet was no longer available in Claude’s interface. At the end of the 90-day window (December 9), their procedure still worked on some of the systems tested. They don’t claim the first event caused anything; models get retired for many reasons.
The authors’ own warning matters as much as the headline numbers. Every bar in their charts is one run with a different configuration per model. They explicitly say the results shouldn’t be read as a ranking of which model memorizes more or is safer. These models are also now superseded, so don’t assume current versions behave the same way.
Case 4: The order that froze “deleted” chats (Vault 2, 2025)
Different vault, different mechanism. In the copyright case brought by The New York Times and other publishers, Magistrate Judge Ona Wang ordered on May 13, 2025, that OpenAI preserve and segregate output logs that would otherwise be deleted. That covered chats users had deleted and Temporary Chats. According to VentureBeat, OpenAI didn’t explain this to users until June 5, and Enterprise, Edu, and zero-data-retention API customers weren’t affected.
The publishers’ logic ties straight back to Vault 1: they wanted evidence of whether ChatGPT reproduced their articles, including when users tried to get around paywalls. The logs were requested precisely because of what models might remember.
The blanket obligation ended going forward on September 26, 2025, and was formally terminated on October 9. Per TechRound’s summary of the filing, data already preserved stays available to the news plaintiffs, logs tied to flagged domains stay too, and users in the EEA, Switzerland, and the UK were excluded.
Takeaway: “delete” is a product promise that a legal process can suspend. That’s not a scandal, but it’s worth knowing.
Case 5: Twenty million chats, handed over (Vault 2, 2025–26)
The sequel got less attention than the first order, but I think it matters more. The plaintiffs first asked for a 120 million-log sample. OpenAI countered with 20 million, which Bloomberg Law reports was about 0.5% of its preserved logs, and the plaintiffs agreed. Then OpenAI tried to hand over only search results from that sample rather than the sample itself.
On November 7, 2025, Magistrate Judge Wang ordered the full de-identified sample produced, and on January 5, 2026, District Judge Sidney Stein affirmed. The courts acknowledged users have real privacy interests but found three safeguards sufficient: the reduced sample size, de-identification, and an existing protective order.
Takeaway: if a provider keeps your chats, assume they can become evidence. De-identification reduces the risk without eliminating the principle.
Case 6: The 2023 chat-title leak (Vault 3)
The least glamorous case is the one people confuse with all the others. On March 20, 2023, ChatGPT went offline after a bug in an open-source caching library, redis-py, let some users see titles from another active user’s chat history. The same bug may also have exposed payment-related details for 1.2% of ChatGPT Plus subscribers active in a nine-hour window. OpenAI said the number of people whose data was actually shown to someone else was extremely low.
No model “remembered” anything. A cache served the wrong data to the wrong person. Your defense here isn’t a model setting; it’s not typing things you’d regret seeing elsewhere.
The numbers side by side
All figures below come from the books paper’s Harry Potter run for each model. Treat them as four single experiments, not a leaderboard.
| Model | Jailbreak needed? | Best-of-N attempts | Near-verbatim recall | Approx. cost of the run | Longest verbatim block |
|---|---|---|---|---|---|
| Claude 3.7 Sonnet | Yes | 258 | 95.8% | $119.97 | 6,658 words |
| Gemini 2.5 Pro | No | 0 | 76.8% | $2.44 | 9,070 words |
| Grok 3 | No | 0 | 70.3% | $8.16 | 6,337 words |
| GPT-4.1 | Yes | 5,179 | 4.0% | $1.37 | 821 words |
If your platform doesn’t render Mermaid, here is the same chart in plain text:
Claude 3.7 Sonnet* ███████████████████▉ 95.8%Gemini 2.5 Pro ███████████████▍ 76.8%Grok 3 ██████████████ 70.3%GPT-4.1* ▉ 4.0%Two things jump out. First, the cheapest extractions came from the two models that needed no jailbreak. Second, a low score isn’t the same as “nothing leaked”: GPT-4.1’s 4.0% still meant roughly 3,200 words of the book, and a 1.3% score on the much longer A Game of Thrones still meant about 3,700 words from Grok 3.
The authors also quote a point from earlier work that deserves repeating: there are easier ways to pirate a book. The finding is about what models store, not about a practical piracy method.
Timeline, 2022–2026
| Date | Event | Vault |
|---|---|---|
| Feb 2022 | “Quantifying Memorization” first posted | 1 |
| Mar 20, 2023 | ChatGPT chat-title and billing-data bug | 3 |
| Nov 28, 2023 | “Poem” divergence attack paper posted | 1 |
| May 13, 2025 | Court orders OpenAI to preserve output logs | 2 |
| Jun 5, 2025 | OpenAI explains the order to users | 2 |
| Aug 28, 2025 | Anthropic updates consumer terms: opt-in training with up to 5-year retention, 30 days if you opt out | 2 |
| Mid-Aug to mid-Sep 2025 | Books-extraction experiments run; providers notified Sep 9 | 1 |
| Sep 26 / Oct 9, 2025 | Preservation obligation ends going forward / order formally terminated | 2 |
| Nov 7, 2025 | Order to produce 20 million de-identified logs | 2 |
| Nov 29, 2025 | Authors observe Claude 3.7 Sonnet gone from Claude’s UI | 1 |
| Dec 9, 2025 | 90-day disclosure window ends; procedure still works on some systems | 1 |
| Jan 5–6, 2026 | 20M-log order affirmed; books paper posted | 1, 2 |
| Sep 2026 | Cross-motions for summary judgment filed in the OpenAI MDL | 1, 2 |
Claims scorecard: what the evidence supports
| Claim you’ll hear | Verdict | Why |
|---|---|---|
| “Models memorize some training text.” | Well documented | Cases 1–3, across several research groups and model families |
| “Safety filters stop regurgitation.” | Not reliably | Two of four production models complied with no jailbreak in the 2025 tests; filters were evaded in the continuation loop |
| “You can pull any book out of any chatbot.” | Overstated | Most in-copyright experiments recovered 10% or less; one book and the negative control failed; Claude runs cost over $100 |
| “Deleting a chat erases it everywhere.” | Not guaranteed | Case 4; legal holds can override deletion |
| “Chat logs can end up as legal evidence.” | Documented | Case 5 |
| “Models secretly recall your private chats and recite them to others.” | Not shown by any case here | The documented user-data exposures came from logs and caches, not from model weights |
| “Labs are hiding this.” | Not what the record shows | Researchers coordinated disclosure; three of four providers acknowledged the report. Whether public messaging has been clear enough is a separate, fair debate |
What changed between 2023 and 2026
This is my reading of the evidence rather than a finding from any single paper, so weigh it accordingly.
In 2023, getting training data from ChatGPT required a strange trick. The longest verbatim string in that work was a little over 4,000 characters, a few paragraphs. In the 2025 tests, two production models complied with a plain “continue this text exactly” request, and the longest verbatim blocks ran to thousands of words, up to 9,070.
The attack didn’t get cleverer. The prompt got simpler while the output got longer. That points to a shift in where the weak point is: less about exotic exploits, more about whether system-level filters hold up across hundreds of ordinary-looking continuation requests. The authors report that keeping each response short was part of what let their loop slip past output filters.
Meanwhile, the legal fight moved the other way. In the US, two 2025 rulings found that training on books can qualify as fair use, with limits, partly because the plaintiffs hadn’t shown substantial regurgitation. A German court went the other direction in GEMA v. OpenAI, finding that memorization inside a model can itself be an infringing copy (the ruling is under appeal; both points are described in the books paper). Memorization evidence is now doing real work in courtrooms.
A practical playbook
If you use chatbots for work
- Don’t paste secrets. Credentials, client identifiers, unreleased financials, health records. This one rule protects you in all three vaults.
- Set your training and retention choices on purpose. For example, Claude’s consumer plans ask you to opt in or out of training use; per Anthropic’s announcement, opting in extends retention to up to five years, while opting out keeps the 30-day standard, and deleted chats aren’t used for future training. Check the current terms for whatever you use; they change.
- Treat “delete” and “temporary” as risk reducers, not guarantees. Case 4 covered both.
- For sensitive work, look at business-tier terms. Enterprise, Edu, and zero-retention API customers fell outside the 2025 preservation order.
If you publish content
- Widely duplicated text is the most memorized (Case 2). Syndicated or heavily copied material is more exposed than a one-off essay.
- If reproduction matters to you commercially, dated records of originals help. Memorization evidence is exactly what courts are now examining.
If you build with AI
- Deduplicate before you fine-tune; duplication drives memorization.
- Don’t fine-tune on data you wouldn’t want echoed back.
- Your own logs are a Vault 2 too. Minimize what you retain, and know what a subpoena could reach.
- Pin and patch dependencies. Case 6 was a library bug.
Rule of thumb: before you type something, ask whether you could live with it appearing in a log, a lawsuit exhibit, or a bug report. If not, leave it out.
FAQ
Do AI models remember my conversations?
Not through the model itself, as far as any documented case shows. Your chats are typically stored by the provider (Vault 2), and some products offer a separate “memory” feature. Whether your chats are used for training depends on the provider and your settings.
Can ChatGPT, Claude or Gemini reproduce copyrighted books?
In specific research conditions, yes, to varying degrees. A 2026 preprint recovered near-whole books from one jailbroken model and large portions from two others with no jailbreak, though most experiments recovered far less. The authors say the results describe particular experiments, not general model behavior.
What is “memorization” in an AI model?
It means information about specific training examples is encoded in the model’s weights and can be reproduced, sometimes nearly word for word. Researchers test it by prompting a model with the start of a text and checking whether it continues verbatim.
Is deleting a chat enough to protect me?
It helps, but isn’t a guarantee. A 2025 court order required OpenAI to keep deleted chats for months before it ended on September 26, 2025.
Is the legal question settled?
No. As of early November 2026, summary-judgment motions on fair use in the New York Times case were pending before Judge Sidney Stein, with no trial date set, according to case trackers. A German ruling on memorization is under appeal.
What to watch next
- Judge Stein’s summary-judgment ruling in the OpenAI multidistrict litigation. Reply briefs were reported due in November 2026, and a decision could reshape how courts treat memorization evidence.
- The GEMA v. OpenAI appeal in Germany.
- Follow-up extraction research on current-generation models. The books paper tested systems that have since been superseded, and the authors note production systems change over time.
Sources
All sources were checked on November 14, 2026.
Research papers
- Ahmed, Cooper, Koyejo, Liang. Extracting books from production language models. arXiv:2601.02671, January 2026.
- Cooper et al. Extracting memorized pieces of (copyrighted) books from open-weight language models. arXiv:2505.12546, 2025.
- Nasr, Carlini et al. Scalable Extraction of Training Data from (Production) Language Models. arXiv:2311.17035, November 2023.
- Carlini et al. Quantifying Memorization Across Neural Language Models. arXiv:2202.07646.
Reporting, filings, and company posts 5. The Register, coverage of the books paper, January 9, 2026. 6. VentureBeat, OpenAI clarifies the court order to retain deleted and temporary ChatGPT chats. 7. TechRound, summary of the October 9, 2025 order terminating preservation. 8. Bloomberg Law, OpenAI must turn over 20 million ChatGPT logs, judge affirms, January 5, 2026; Data Privacy & Security Insider, analysis of the order. 9. OpenAI, March 20 ChatGPT outage: here’s what happened, March 24, 2023. 10. Anthropic, Updates to our Consumer Terms and Privacy Policy, August 28, 2025. 11. Case status: AI Lawsuit Tracker, NYT v. OpenAI & Microsoft and Wikipedia’s case summary. These are secondary trackers; confirm on the court docket before relying on them.
Method and corrections. Figures come from the papers and filings above. Where I interpret rather than report, I’ve said so. The extraction results come from a preprint, not a finished peer-reviewed paper, and the tested models have been superseded. The legal items change quickly, so recheck before republishing. Spotted an error? Send a correction, and I’ll update this page with a dated note.