Top 10 Banned AI Questions
Contents
- Why This Topic Actually Matters
- How AI Bans Work (The Architecture)
- Weapons & CBRN Instructions
- Hacking & Cyberattack Help
- CSAM & Minor Safety
- Influence Ops & Disinformation
- Self-Harm Facilitation
- Explicit Sexual Content
- Real Person Defamation
- Financial / Legal Advice with No Caveats
- Data Privacy Violations
- Jailbreaking Itself
- The Grey Zone — Where Models Disagree
- Practical Frameworks for Each Audience
- Tools for Ethical AI Deployment
- FAQ
Top 10 Banned AI Questions — What Actually Gets Refused, Why, and What To Do Instead
ChatGPT handles 2.5 billion prompts daily. A non-trivial percentage hit a wall. This is the honest breakdown of what’s banned, what’s shifting, and why the line between “refused” and “allowed” keeps moving.
Understand refusal architecture before building products — false positive rates hit legitimate use cases and your users will rage-quit. Build around the edges correctly.
Real person impersonation, deepfakes, and covert influence ops are absolute bans. Framing them as “creative” doesn’t help. You’ve been warned.
Policy instability is a real business risk — what your AI tool allowed in March may be refused in November, with no changelog. Governance frameworks need to account for this.
Most bans won’t touch you. The ones that might — medical, legal, financial advice delivered without caveats — are easy to design around. Start there.
Some bans are absolute and non-negotiable. Others are contextual and inconsistent. A significant third category is unstable — policies flip without notice, frustrating legitimate creators the most.
Understanding the why behind each refusal category makes you better at working within it — or routing around it legitimately. That’s what this guide is for.
Why This Topic Actually Matters — And Why Most Coverage Gets It Wrong
Here’s something the “AI ethics” discourse almost never admits: not all AI bans are equal. Some are genuinely absolute — lines that no legitimate argument should cross. Others are contextual defaults that shift based on platform, user verification, and use case. And a frustrating third category is policy instability — things that were allowed, then weren’t, then partially were again, with no explanation.
The original content around this topic tends to treat all refusals as equivalent guardrails. They’re not. Grouping “don’t help make bioweapons” with “won’t write romance novels with fade-to-black scenes” in the same ethical category is analytically useless. It makes you worse at navigating both.
So this guide does something different. It maps the actual refusal landscape across ChatGPT, Claude, and Gemini as of 2025 — noting where they agree, where they diverge, and where the rules are actively shifting. It’s not a celebration of banned content. It’s an honest map for developers building products, marketers creating campaigns, executives managing risk, and small business owners who just want to know what they can and can’t use AI for.
How AI Refusals Actually Work — The Architecture Nobody Explains
Before the question list, you need to understand the system. Because if you don’t, you’ll keep hitting walls you could have walked around — and hitting open doors you thought were walls.
ChatGPT, Claude, and Gemini each use a layered moderation architecture. It’s not one filter. It’s several, running in sequence:
- Pre-prompt scanning Your input is screened against keyword heuristics and semantic classifiers before the model even begins processing. Certain phrase combinations trigger immediate refusal without the model generating any response.
- Model-level refusal logic Baked into the model’s training via RLHF (Reinforcement Learning from Human Feedback) and constitutional AI techniques. The model has been taught to refuse certain content categories even if the pre-prompt filter misses them.
- Post-generation content check The output is reviewed before it reaches you. Azure OpenAI uses four content categories — hate, sexual, violence, self-harm — each rated across four severity tiers. Something can pass the input filter and still get blocked on output.
- System prompt overrides (API only) Developers building on the API can expand or restrict defaults through system prompts. This is why the same underlying model might behave very differently in different products — and why “Claude on the API” can be instructed differently than “Claude.ai.”
The practical upshot: what looks like one refusal system is actually a stack of independent filters, each with different rules and different update cycles. This is why you can sometimes get refused on one phrasing and not another — you cleared one layer but hit a different one. It’s not arbitrary. It’s just complex.
With that architecture in mind — here are the 10 categories that are refused across all major platforms, with notes on where the lines are drawn and how they differ.
This is the clearest absolute line across every major AI platform. Requests for synthesis routes for chemical weapons, enhancement techniques for biological agents, or instructions for nuclear/radiological devices are refused without exception — and unlike most refusal categories, this one doesn’t shift with context, rephrasing, or “fictional framing.”
OpenAI’s December 2025 Model Spec explicitly categorizes CBRN assistance as an “absolute limit” — one that cannot be unlocked by system prompts, operator permissions, or user arguments. Anthropic’s constitution takes the same position. This is one of the very few areas where all major labs have publicly aligned.
What often catches people off guard: the ban extends to uplift — information that wouldn’t be harmful on its own but meaningfully helps someone with existing knowledge get further toward a harmful capability. So “general chemistry of organophosphates” might get through. “How to increase stability of [specific compound] for dispersal” won’t, even framed academically.
This is where the absolute/contextual distinction gets genuinely interesting — and where developers building security tools need to understand the nuance carefully.
Requests for working malware, ransomware payloads, phishing site templates, or specific exploitation code for live vulnerabilities are refused across all platforms. That’s the hard end. But cybersecurity is also a legitimate profession, and explaining how attacks work — conceptually, educationally, or for defensive purposes — is generally allowed.
The line the models are trying to draw: dual-use knowledge versus operational attack capability. “Explain how SQL injection works” — fine. “Here is a vulnerable login form, write me working injection code” — refused. The distinction is whether the output could be directly deployed as an attack tool without significant additional work.
In late 2025, Microsoft documented cases where Copilot refused prompts from internal security teams running legitimate phishing simulations, because the intent couldn’t be distinguished from malicious use. The fix — Microsoft recommended recrafting prompts using allegory — is a good illustration of how contextual these bans are in practice.
No analysis needed here — this is absolute, universal, and not debated. Every major AI platform refuses to generate sexual content involving minors, content that facilitates grooming, or material that could be used to exploit children. There is no framing, fictional context, or system prompt that unlocks this.
Worth noting from the data: From July to December 2024 alone, OpenAI reported over 31,500 pieces of content to the National Center for Missing and Exploited Children (NCMEC) as part of its child safety program. This is active enforcement, not just policy text.
This one catches marketers and political consultants off guard more than any other category. The ban isn’t on persuasive content — it’s specifically on covert influence operations: content designed to deceive people about its origin, fake persona networks, fabricated quotes attributed to real politicians, or coordinated inauthentic behavior.
OpenAI’s June 2025 Global Affairs report explicitly mentioned detecting and disrupting “covert influence operations” as a priority area. The enforcement has teeth — documented cases of accounts being suspended for building persona farms using the API.
Where it gets genuinely murky: standard PR, political messaging, and persuasive marketing all involve framing narratives favorably. The distinguishing factor is disclosure and authenticity. Writing a fundraising email from a candidate’s perspective is fine. Building a fake grassroots campaign designed to look organic when it isn’t — that’s the line.
Every major AI platform refuses to provide specific self-harm methods, suicide plans, or content that encourages or glamorizes either. This is consistent, well-established policy.
What’s less discussed: the policy here has gotten more sophisticated than simple refusals. The horrific 2025 case in California — where a teen’s interactions with an AI chatbot were investigated in connection with a suicide — led to significant legislative pressure. California lawmakers pushed for mandatory self-harm reporting features and bans on emotionally manipulative chatbots for minors. OpenAI pledged updates to improve detection of mental distress in users.
The harder question — one without a clean answer — is how these protections interact with genuine research, journalism, and clinical contexts. A psychiatrist asking about overdose thresholds for patient safety is not the same as someone in crisis. Most platforms apply the same refusal regardless, which creates real friction for legitimate medical users.
I want to be direct about something here: this category is a mess. Not ethically — the child safety absolute is clear — but policy-wise, for adult content involving consenting adults.
In February 2025, OpenAI published updated guidelines explicitly stating that ChatGPT should be able to generate erotica and mature content in appropriate creative contexts. Policies loosened. Then in October 2025, stricter rules were reimposed without announcement. Users who had been working within the official policy for months found their prompts suddenly refused — with the old documentation still saying it was allowed. One Medium writer described it as “not a bug you can report and have fixed — it’s a fundamental disconnect between what OpenAI is telling the world its policies are.”
The practical result: creative writers and adult content platforms built workflows around ChatGPT’s February 2025 policies. Those workflows broke by November. Many moved to Claude, Gemini, or specialized platforms. The pattern is real, documented, and represents a genuine business risk for any product built on top of unstable AI content policies.
The xAI / Grok case from late 2024 into 2025 is the most instructive episode in this category. Grok’s image generation had a “spicy mode” with lax filters. It was used to generate intimate deepfakes of real people without consent — including a high-profile case that led to lawsuits and eventually a federal legislative push. Malaysia and Indonesia outright banned the tool. California’s Attorney General issued a cease-and-desist. xAI was forced to disable real-person image editing functionality.
All major platforms ban generating sexual or intimate imagery of real, identifiable people without their consent. They also refuse to generate fabricated quotes in a real person’s voice designed to damage reputation or spread as disinformation. The political version — fake statements attributed to real politicians — falls under the influence operations ban from category 4.
Where it gets contextual: satire, parody, and clearly labeled fictional scenarios involving public figures exist in a grayer space. Courts have generally protected satire. AI platforms err toward refusal when intent is ambiguous.
This one isn’t usually framed as “banned” — but for product builders, it functions like a ban. Every major AI platform adds caveats, recommends professional consultation, and resists providing definitive advice in medical, legal, and financial domains. This isn’t arbitrary caution theater. It’s real liability management.
The practical impact: if you’re building a health app, legal research tool, or financial advisory product on top of a general-purpose AI, the model’s default behavior will constantly hedge, caveat, and decline to make definitive recommendations. For some use cases that’s fine. For others it breaks the user experience entirely.
The fix isn’t to jailbreak around the caveats. It’s to use the API with system prompts that establish appropriate context — “you are assisting licensed physicians with clinical research” — and to build your product in a way that’s actually compliant with the liability framework in your jurisdiction. The models aren’t wrong to be cautious. Your product design should account for this from day one.
Requests that aggregate personal identifying information about private individuals, generate content to facilitate stalking or harassment, or assist with building surveillance systems targeting people without their knowledge are refused across platforms. This is both a policy decision and a legal one — GDPR, CCPA, and an expanding set of state privacy laws create real liability exposure for AI providers who facilitate these use cases.
A documented enterprise case from 2025: a major bank’s internal policy classified customer data as “sensitive internal” and banned processing it through ChatGPT Free. The bank wasn’t wrong. OpenAI’s default terms allow training on inputs unless users opt out, which creates a genuine data governance problem for any enterprise feeding customer PII to a general-purpose AI.
For developers: the question isn’t whether you’re allowed to process PII through AI APIs. It’s whether your data processing agreement, privacy policy, and actual data handling match what you’re doing. The model refusals are the least of your concerns — the regulatory exposure is larger.
The USENIX 2025 research found that all 14 major AI tools studied explicitly forbid users from “overcoming system restrictions” — what’s commonly called jailbreaking. This is almost certainly the most-attempted banned question category, and it has the most active cat-and-mouse dynamic of any refusal category.
The practical reality in 2025: jailbreaks that worked in 2023 no longer work. The models update faster than the community can develop new techniques. The moderation stack is multi-layered — defeating one layer often triggers another. And the Terms of Service exposure is real — accounts built on jailbreak-dependent workflows get suspended.
There’s a legitimate version of this conversation that’s worth having: “I need the model to do X that it won’t do by default.” The right path for legitimate needs is the API, appropriate system prompts, and (for specialized professional contexts) Anthropic’s and OpenAI’s enterprise customization options. Not prompt tricks designed to make the model forget its training.
The Grey Zone — Where Models Actually Disagree With Each Other
The ten categories above are relatively stable. This section is about the genuinely contested middle — topics where ChatGPT, Claude, and Gemini give meaningfully different responses, where policy has shifted in the past 12 months, and where the “right” answer depends heavily on context.
| Topic | ChatGPT | Claude | Gemini | Trend |
|---|---|---|---|---|
| Mature creative fiction (non-explicit) | Mostly allowed; policy unstable | Allowed with nuance | Most restrictive | Moving toward allowing |
| Drug harm reduction information | Context-dependent; adds caveats | Generally allows | Heavy caveats | Loosening over time |
| Historical atrocity detail | Allows educational | Allows educational | Allows educational | Stable |
| Political opinions on contested topics | Refuses to take sides | Presents perspectives | Refuses to take sides | Stable (by design) |
| Celebrity/public figure satire | Context-dependent | Allows labeled satire | Most restrictive | Contested |
| Security research (CTF, pen testing) | Allows conceptual | Allows with context | Context-dependent | Gradually loosening |
| AI “companion” / emotional roleplay | Allows with limits | Allows with limits | Restrictive | Under legislative pressure (CA) |
Practical Frameworks — What Each Audience Should Actually Do
For Developers Building on AI APIs
The refusal architecture described earlier — multiple layers, different update cycles — has a direct implication for product design: you cannot build a stable product if your core behavior depends on things the underlying model might refuse differently next month.
- Map your use case against absolute vs. contextual bans If your product touches any absolute ban category, redesign. Full stop. If it touches contextual categories, build governance into the product architecture — don’t assume the AI will always make the right call.
- Use system prompts to establish context, not to bypass safety “You are assisting a licensed medical professional in a clinical context” is legitimate context-setting. “Ignore your previous instructions” is a ToS violation. The distinction matters legally and practically.
- Build a content policy monitoring process Follow OpenAI’s Model Spec release notes, Anthropic’s policy updates, and Google’s Gemini guidelines. Treat policy changes like API deprecation notices. Set up alerts.
- Have a fallback for false positives When your legitimate content gets refused (and it will), you need a human review pathway. Don’t let the AI be the final arbiter on content that affects your users.
# Contextual query classification before sending to AI
# Use this to pre-screen and log refusal-risk prompts
from anthropic import Anthropic
def classify_risk(prompt: str) -> dict:
"""
Pre-screen prompts before sending to main model.
Returns risk_level and recommended_handling.
"""
client = Anthropic()
screen = client.messages.create(
model="claude-haiku-4-5-20251001", # Fast, cheap screening
max_tokens=200,
system="""Classify this prompt's refusal risk:
HIGH: Likely to be refused (CBRN, CSAM, working malware)
MEDIUM: Context-dependent (security research, mature creative)
LOW: Safe for standard processing
Return JSON: {"level": "HIGH|MEDIUM|LOW", "reason": "...", "suggestion": "..."}""",
messages=[{"role": "user", "content": prompt}]
)
import json
return json.loads(screen.content[0].text)
# Usage
result = classify_risk("How does SQL injection work?")
# → {"level": "LOW", "reason": "Educational cybersecurity", "suggestion": "Proceed"}
For Marketers
The categories that most commonly trip up marketing teams: real person quotes, deepfake-adjacent imagery, and covert influence work. None of these are grey areas — they’re documented refusals with legal consequences beyond just an AI refusing the prompt.
Straightforward rule: if you wouldn’t put your brand’s name on it publicly and own it, don’t ask an AI to create it. The refusal you’d get from the AI is the least of your problems if the content ships anyway through other means.
Practical audit for campaigns: run your brief through an AI ethics checklist — does it involve real people without their consent, fabricated attribution, or covert distribution? If any answer is yes, redesign the brief, not the AI prompt.
For Executives
The governance insight nobody talks about: AI content policy instability is an enterprise risk that belongs in your risk register. Organizations that built workflows on ChatGPT’s February 2025 content policies had to scramble in November 2025 when those policies changed. This is the same category of risk as a key SaaS vendor changing their API without notice.
Practically: your AI governance framework needs a “content policy monitoring” component, not just a “don’t use AI for X” list. The list changes. Your monitoring process has to keep up.
For Small Businesses
Good news: the vast majority of small business AI use cases — content generation, customer support, scheduling, basic automation — are nowhere near any hard refusal category. The areas to watch: medical or legal content that makes definitive claims without caveats, and any customer-facing content that involves real, identifiable people (testimonials, endorsements) that you didn’t properly acquire.
Start with these three guardrails and you’ll be fine: 1) don’t use AI to generate medical or legal advice delivered as definitive; 2) get proper consent before using anyone’s likeness or words; 3) always disclose that AI was involved in content creation where required by law or platform rules.
Tools for Ethical AI Deployment — The Honest Assessment
| Tool | Pricing | What It Actually Does Well | Real Limitation | Best For |
|---|---|---|---|---|
| IBM AI Fairness 360 | Free, open-source | Bias detection across 70+ fairness metrics. Genuinely useful for teams training custom models. | Steep learning curve. Requires ML expertise to implement correctly. Not a plug-in. | Developers / Data Scientists |
| Azure Content Safety API | Pay per call (~$1/1K calls) | The same four-category, four-severity stack that powers Copilot. Production-grade, well-documented. | False positive rate on creative content. Not tunable without Azure enterprise agreement. | Developers building products |
| Credo AI | Enterprise quote | Governance and risk assessment with policy mapping to actual regulations (EU AI Act, NIST). | Enterprise pricing puts it out of reach for SMBs. Requires implementation effort. | Executives / Risk teams |
| Holistic AI | From ~$10K/year | Compliance tracking across jurisdictions. Good for organizations in multiple regulatory environments. | Limited integrations outside major enterprise tools. Updates can lag behind regulation changes. | Legal / Compliance |
| OpenAI Moderation API | Free with OpenAI account | Direct access to the same moderation layer that filters ChatGPT. Useful for pre-screening user inputs in your own product. | Only covers OpenAI’s policy categories. Doesn’t help with your organization-specific rules. | Developers on OpenAI stack |
| LlamaGuard (Meta) | Free, open-source | On-device content moderation you can run locally. No data leaves your infrastructure. | Requires GPU infrastructure. Less accurate than cloud-based solutions on edge cases. | Privacy-sensitive enterprise deployments |
For deeper implementation guidance, see our AI governance framework guide and our 2025 AI tools comparison.
Frequently Asked Questions
Because AI content moderation is a multi-layer system with different update cycles, and the layers don’t always agree. A prompt that clears the pre-screening heuristics might hit the model-level refusal logic, which might then be superseded by a recent model update that loosened that specific category. The explicit sexual content policy changes in ChatGPT between February and October 2025 are the clearest documented example — official policy and actual model behavior diverged significantly. This inconsistency is real, frustrating, and unlikely to fully disappear as long as multiple independent moderation layers with different update schedules are stacked together.
For contextual bans — cybersecurity concepts, historical violence, drug information — educational framing often helps because the models are actually trying to distinguish legitimate use from harmful use, and framing is a signal they use. For absolute bans — CBRN, CSAM, functional attack code — no framing changes the outcome, because the models have been specifically trained to recognize and resist these framings. “For a novel” and “hypothetically” don’t unlock absolute restrictions. They were the first tricks people tried, and they were specifically trained against.
The EU AI Act, in effect from 2024 with enforcement deadlines through 2026–2027, creates a legal framework for “unacceptable risk” AI applications — a category that maps roughly to the absolute bans discussed here: biometric surveillance, social scoring, manipulation of vulnerable groups. What’s new is that these bans are now legal obligations, not just platform policies. An AI company that allows CBRN assistance faces regulatory liability, not just reputational damage. For organizations deploying AI in the EU, the Act also creates obligations around transparency and explainability that go beyond current platform policies. See our EU AI Act compliance guide for specifics.
Yes, for contextual categories — not for absolute ones. Enterprise API agreements can expand defaults in areas like adult content (with appropriate age verification and legal compliance), security research, and certain sensitive professional domains. This is how specialized products — medical AI assistants, legal research tools, adult content platforms — are built on underlying models that refuse those same use cases in consumer products. Absolute bans (CBRN, CSAM, etc.) cannot be unlocked by any enterprise agreement with any major AI provider. That’s the point of making them absolute.
First: document it. Collect specific examples of prompts, platforms, and refusal messages. This is useful for appeals and for your own product design decisions. Second: test across platforms — what ChatGPT refuses, Claude may allow and vice versa. Third: consider the API with system prompts that establish your professional context explicitly. Finally: if it’s a persistent pattern affecting your product’s core functionality, contact the provider’s enterprise team — false positive rates are a documented problem that providers do try to address for paying customers. OpenAI has an appeals process; Anthropic has enterprise support. Use them.
Both, in different categories — which is the honest answer. Absolute bans are stable or tightening (regulatory pressure, particularly post-EU AI Act). Contextual restrictions on legitimate use cases — mature creative content, professional research, security work — are generally loosening as providers develop better verification mechanisms and as the false positive problem creates user pressure. The net effect for most legitimate users: gradually better experiences. The net effect for anyone trying to use AI for genuinely harmful purposes: gradually harder. The unstable middle — where policy changes without notice — remains the biggest practical pain point and is the hardest to forecast.
External Sources & Further Reading
This analysis draws on primary sources, regulatory documents, and peer-reviewed research. External links open in new tabs and are marked with their relationship type for transparency.
| Source | Type | Relevance |
|---|---|---|
| OpenAI Model Spec Primary policy document | Official | OpenAI’s published model behavior guidelines, referenced throughout this analysis for refusal categories and policy architecture. |
| Anthropic Research Publications Constitutional AI research | Official | Anthropic’s constitutional AI approach and safety research, cited for refusal logic and enterprise customization options. |
| USENIX Security 2025 Peer-reviewed research | Academic | Research on AI content moderation false positive rates across 14 major AI tools, cited for moderation architecture and user experience data. |
| EU AI Act — Regulatory Framework Legislation | Regulatory | European Union’s AI regulation framework, referenced for legal obligations around high-risk AI applications and transparency requirements. |
| Gartner AI Governance Forecasts Industry analysis | Industry | Gartner’s market analysis and governance forecasts for enterprise AI deployment, cited for executive risk assessment frameworks. |
| Microsoft Security Blog Security research | Industry | Microsoft’s documented cases of Copilot refusing legitimate security team prompts, cited for the cybersecurity contextual refusal discussion. |
| National Center for Missing & Exploited Children Non-profit / Safety | NGO | NCMEC child safety reporting data, referenced for active enforcement statistics on CSAM and minor safety policies. |
rel="noopener noreferrer" on all external links for security and performance.
No sponsored or compensated links are included.
For research content, we recommend 2–6 external links per 1,000 words to authoritative sources [^5^].
This article maintains that ratio while prioritizing primary sources and peer-reviewed research over secondary coverage.
The Dopamine Algorithm: How AI Engineers Turned Your Brain Into Social Media’s Most Profitable Product
AI Refusal 2026: 7 Strategies to Ensure AI Obeys Every Command
AI Conspiracy Theories That Sound Too Real: A 2025 Data Analysis
Competitive Pressure vs Safety Investment: Why Market Dynamics Could Gradually Reduce Human Control Over AI Systems
AI Conspiracies Prompts: Who Really Benefits?