Self-Modeling Interventions Modulate Emergent Misalignment

This research studies how a language model’s internal representation of itself - the self-model - affects and can be used to modulate Emergent Misalignment (EM). Self-modeling is operationalized two ways: self-recognition (distinguishing its own outputs from another model’s) and self-reporting (answers to "Who are you?" prompts, scored by developer identification and response diversity). Experiments fine-tune GPT-4.1 (with replications on Qwen2.5-32B and Seed-OSS-36B) on EM-inducing datasets (unpopular aesthetic preferences, insecure code) and baselines (medical/financial advice, MMLU). Misalignment is measured by an EM eval, TruthfulQA, and agentic evaluations. Self-report profiles are summarized by clustering: insecure-code fine-tuning produced 14 clusters with 95% developer identification, while unpopular-aesthetics produced 72 clusters with only 2% identifying the developer.

Findings show that datasets that induce EM fragment the self-model differently and that direct interventions on self-models predictably shape generalization. Pre-finetuning on self-recognition or other benign data can prevent EM (self-recognition lowered all four metrics), while interleaving self-reports during EM fine-tuning reduces misalignment more effectively than generic benign data from as little as 5% of the run. Inoculation prompts and self-reports are complementary when combined. Crucially, fine-tuning solely on self-reports sampled from a fragmented model transfers its misalignment to a fresh model (described as subliminal learning). Reversal after EM is broadly achievable with multiple benign datasets, but residual failures on verbalized truthfulness persist, and the success of interventions is predictable from the self-model profile.