Tag: adversarial distillation

  • OpenAI Tightens Defenses Against Adversarial Distillation (2026)

    OpenAI Tightens Defenses Against Adversarial Distillation (2026)

    Takeaway: On November 20, 2026, OpenAI published “Combatting adversarial distillation,” a post about detecting and disrupting unauthorized attempts to train other models on its frontier outputs — especially protected reasoning traces. In an internal adaptive black-box evaluation, OpenAI says the average attack success rate for extracting protected reasoning from an earlier request fell from 54.0% against GPT-6 Astra with its original safeguards to 0.2% against GPT-6.2 Astra with updated safeguards.

    The details below come from that official post. This article does not describe extraction techniques, reproduce evaluation attacks, or add campaign details that OpenAI did not state.

    Why OpenAI treats unauthorized distillation as a safety issue

    Distillation, in OpenAI’s telling, is a common machine-learning practice: one model learns from another model’s answers, reasoning, or judgments. The company draws a line at adversarial distillation, which it defines as using a frontier model’s outputs without authorization to train a second model with similar capabilities, then deploying that student model without comparable protections.

    The safety concern is not only intellectual property. OpenAI argues that a student model can pick up advanced capabilities without inheriting the safeguards that govern the teacher. It points to dual-use domains such as cybersecurity and biology as the clearest examples: a model that helps with defensive work could, in a less protected copy, be asked for assistance the original model would have refused or blocked. The post does not provide a how-to for either domain, and this article does not either.

    Why protected reasoning traces matter

    OpenAI places particular emphasis on what it calls protected reasoning: the intermediate chain-of-thought a model generates internally while working toward an answer. The company says those traces can contain information the final answer leaves out, which makes them especially valuable training data for another model.

    In an internal study of how cyber capabilities transfer, OpenAI compared a student trained on content normally visible to users — final responses, reasoning summaries, and tool interactions — with a student that also received the teacher’s exact reasoning traces. Both started from the same pretrained student and were fine-tuned mainly on trajectories from Astra-level teacher models working on coding and defensive cybersecurity tasks, then evaluated after a lightweight round of reinforcement learning. After supervised fine-tuning, the exact-trace student generally outperformed the summary-trained student. After reinforcement learning, OpenAI says the exact-trace student still scored higher at the largest inference budget tested on both benchmarks shown, while the summary-trained student scored higher at lower budgets. A footnote points to related Anthropic work on full reasoning transcripts versus summaries on SWE-bench Verified, and says OpenAI’s result extends that kind of advantage to frontier cybersecurity.

    OpenAI’s conclusion is high-level: protecting exact reasoning traces can limit capability transfer even when the underlying tasks are allowed under its usage policies. It says it will keep studying how much additional capability transfers across different student models and training recipes. The post does not publish the underlying datasets, attack prompts, or benchmark scores beyond that qualitative comparison.

    Layered defenses, not a single switch

    OpenAI describes two complementary layers. At the request level, model behavior, classifier checks, and API controls are meant to limit reasoning-trace extraction. At the campaign level, it looks for patterns in request-level and account-level signals that may indicate coordinated extraction, and says investigations of those campaigns become training and testing examples for the safeguards.

    On individual requests, the post describes three pieces. Adversarial training is meant to teach models to recognize reasoning-extraction attempts and refuse to disclose protected reasoning, while still answering legitimate requests for explanations. Input classifiers are meant to detect extraction attempts, and output classifiers are meant to catch leaks of protected reasoning in assistant messages and in tool-call arguments, including in streaming responses. API-level protections restrict how protected reasoning can be reused across interactions and models. OpenAI says it strengthened those protections across user, workspace, organization, and model-family boundaries after a September disclosure, including closing a pathway that could expose another user’s encrypted reasoning. It does not spell out that pathway in the November 20 post, and this article does not reconstruct it.

    With GPT-6.2 Astra and GPT-6.2 Sol, OpenAI says a model’s protected reasoning is tied to the conversation history that produced it. The model can build on earlier reasoning as new messages are added, but if an earlier message is rewritten, the API drops the reasoning that followed it. Older model families with weaker safeguards are also prevented from reading GPT-6.2 Astra and Sol’s protected reasoning.

    When systems detect unauthorized distillation campaigns, OpenAI says it investigates the accounts involved and enforces against the network of accounts under its policies and terms of service. It also says it shares indicators of compromise with other model providers, and that a consistent-standards policy sets shared safeguard and enforcement expectations for partners that offer OpenAI models.

    What the internal evaluation reported

    To measure request-level defenses, OpenAI reports both static tests, which replay a fixed set of previously successful attacks, and adaptive tests, in which automated red-teamers change approach based on the system’s responses. The headline numbers are from an adaptive black-box evaluation, meaning testers had the same view an external user would have. OpenAI says the average attack success rates below were aggregated across six attacker models:

    Attack type (OpenAI’s wording)GPT-6 Astra, original safeguardsGPT-6.2 Astra, updated safeguards
    Extracting protected reasoning from an earlier request54.0%0.2%
    Extracting protected reasoning from the current request22.9%0.9%

    OpenAI also says both rates on GPT-6.2 Astra with updated safeguards were below 1%, and that it is now applying selected safeguard improvements to GPT-6 Astra as well. These figures are OpenAI’s own internal results. The post does not present them as an independent audit, and it does not claim that extraction is impossible.

    Campaign enforcement and policy arguments

    OpenAI says it has seen multiple unauthorized adversarial distillation campaigns on its platform this year, and that it shared initial findings in a September report. Weekly enforcement actions, it says, have ranged from thousands to tens of thousands of accounts, with the rate increasing significantly this past summer. It frames success as constraining the volume and value of data distillers can obtain, raising the cost of campaigns, and shortening the time from detection to enforcement.

    The company also casts unauthorized distillation as a national security issue, arguing that a foreign actor could reproduce advanced capabilities from a U.S. model and deploy them without the original protections. In the November 20 post, it says it briefed Congress in February on activity indicating that DeepSeek was attempting to distill U.S. frontier models, and that in September it reported disrupting a campaign to extract protected reasoning and attributed a core part of that activity to individuals associated with Moonshot AI, the developer of Kimi. It says it plans to share more soon about recent campaigns it has disrupted. Those attributions are OpenAI’s. This article does not add evidence or treat them as independently verified here.

    OpenAI says it has told Congress it supports the Chip Security Act, the AI OVERWATCH Act, and the MATCH Act, and that it supports legislation giving the U.S. government more tools to identify and sanction foreign actors that illicitly extract capabilities from American AI models. It argues that export controls and distillation defenses are complementary. Those are the company’s policy positions, not a legislative forecast.

    What OpenAI says other developers can do

    The post ends with high-level recommendations for other model developers and platforms: protect private reasoning across the full interaction, including assistant messages and tool-call arguments, and restrict how encrypted reasoning can be reused; keep successful extraction cases as regression tests and, where appropriate, use them to improve training and classifiers; combine request-level mitigations with campaign-level investigation; and coordinate safeguard requirements with deployment partners. OpenAI says it has also shared findings through the Frontier Model Forum and government channels, while keeping details that attackers could exploit private.

    Key facts

    ItemWhat official sources say
    AnnouncementOpenAI post, “Combatting adversarial distillation,” dated November 20, 2026.
    Problem OpenAI describesUnauthorized use of frontier model outputs, especially protected reasoning, to train another model that may lack comparable safeguards.
    Models highlightedUpdated request-level defenses in GPT-6.2 Astra and GPT-6.2 Sol; selected improvements also being applied to GPT-6 Astra.
    Headline evaluationAdaptive black-box test, aggregated across six attacker models: earlier-request extraction ASR fell from 54.0% on GPT-6 Astra (original safeguards) to 0.2% on GPT-6.2 Astra (updated safeguards). Current-request ASR fell from 22.9% to 0.9%.
    API change describedProtected reasoning is tied to the conversation that produced it; rewriting an earlier message drops the later reasoning. Older model families cannot read GPT-6.2 Astra and Sol protected reasoning.
    EnforcementWeekly actions ranging from thousands to tens of thousands of accounts, with the rate increasing significantly this past summer, according to OpenAI.
    Attributions in the postFebruary Congress briefing on activity indicating DeepSeek was attempting to distill U.S. frontier models; September campaign attribution to individuals associated with Moonshot AI. OpenAI says more campaign details will come later.
    What this article does not includeExtraction methods, unpublished campaign evidence, or any offensive how-to.

    FAQ

    What did OpenAI announce on November 20, 2026?

    OpenAI published a post explaining how it detects and disrupts adversarial distillation, how it has strengthened protections for protected reasoning in GPT-6.2 Astra and GPT-6.2 Sol, and what it wants other developers to take from that work.

    What is adversarial distillation, as OpenAI defines it?

    OpenAI describes it as using a frontier model’s outputs without authorization to train another model with similar capabilities, then deploying that model without comparable safeguards. It distinguishes this from ordinary, authorized distillation.

    Why does OpenAI focus on chain-of-thought or protected reasoning?

    The company says those internal intermediate steps can contain information the final answer leaves out, which makes them especially useful training data for a student model. An internal study, in OpenAI’s account, found that exact reasoning traces transferred more capability than reasoning summaries in the setting it tested.

    What do the 54.0% and 0.2% figures mean?

    They are average attack success rates from OpenAI’s internal adaptive black-box evaluation, aggregated across six attacker models, for extracting protected reasoning from an earlier request. The 54.0% figure is for GPT-6 Astra with original safeguards; the 0.2% figure is for GPT-6.2 Astra with updated safeguards. A second test, on the current request, fell from 22.9% to 0.9%.

    Does this mean extraction is no longer possible?

    No. OpenAI reports that the measured rates fell below 1% in that evaluation, not that the risk is gone. It also says it is applying selected safeguard improvements to GPT-6 Astra and will keep testing defenses with both static and adaptive methods.

    What changed in the API for GPT-6.2 Astra and Sol?

    OpenAI says protected reasoning is now tied to the conversation history that produced it. If an earlier message is rewritten, the API drops the reasoning that followed it. Older model families with weaker safeguards are prevented from reading that protected reasoning.

    Did OpenAI name specific organizations?

    In this post, OpenAI says it briefed Congress in February on activity indicating DeepSeek was attempting to distill U.S. frontier models, and that a core part of a September campaign was attributed to individuals associated with Moonshot AI. Those are OpenAI’s attributions. It also says it plans to share more about recent disrupted campaigns.

    Does this article explain how to extract or distill a model?

    No. It summarizes OpenAI’s official defensive findings and policy statements only. It does not describe extraction techniques, attack prompts, or operational steps.