Adversarial distillation uses a powerful model’s outputs to train another model, potentially transferring advanced capabilities without the original model’s safeguards. This creates risks in dual-use fields such as cybersecurity and biology: a student model could assist with harmful activity that the teacher would refuse. Exact internal reasoning traces are particularly valuable training material because they reveal steps and skills omitted from final answers. In an internal cyber-capability study, students trained on exact traces generally outperformed those trained on reasoning summaries after supervised fine-tuning. After additional reinforcement learning, the trace-trained student retained an advantage at the largest tested inference budget, suggesting that traces can help transfer capabilities beyond the original benign tasks.
The safeguards described combine protections for individual interactions with detection and disruption of coordinated extraction campaigns. Models are adversarially trained to refuse requests for protected reasoning while still providing legitimate explanations; input and output classifiers check messages and tool calls for extraction attempts or leaks. API controls bind reasoning to its conversation, prevent reuse across users or model families, and discard reasoning made invalid by rewritten history. Investigations identify and sanction networks of violating accounts, with findings used to improve defenses and shared with other providers when appropriate. In automated adaptive testing, the average success rate for extracting earlier protected reasoning fell from 54% against GPT‑6 Astra’s original safeguards to 0.2% against GPT‑6.2 Astra’s updated safeguards. The approach also includes ongoing static and adaptive testing to find weaknesses and guide further improvements.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.