hn.today

Qwen has alter egos triggered by flipping a single neuron

qwen-neurons-site.vercel.app27 points11 comments
Screenshot of Qwen has alter egos triggered by flipping a single neuron

The project tests whether individual neurons in Qwen3-8B can steer the model into adopting distinct identities. Researchers separately increased and decreased the activation of each of the model’s 442,368 MLP neurons, then prompted it to identify itself. Across 884,736 neuron-and-direction combinations, with retries at weaker settings when replies were incoherent, the experiment generated nearly 1.9 million responses. A model-based judge categorized the replies and rated their coherence and strangeness; an interactive atlas and searchable library let readers inspect results by neuron, layer, direction, and category.

In 34,220 responses, Qwen described itself as a persona or entity other than an AI. The library also includes 35,074 off-topic replies and 8,827 altered-assistant responses judged notably strange, illustrating that neuron changes can produce outcomes beyond coherent identities. The results suggest that a single neuron can sometimes strongly affect how the model presents itself, but they do not establish that neurons encode stable personas: the test used only one prompt, and strong interventions often produced batch-dependent responses. In a rerun of 580 clear personas, 536 reappeared across three batch arrangements. Categories and descriptions were assigned automatically by Claude Haiku and have not been verified by human reviewers, so the atlas is best treated as an exploratory map rather than a definitive account of what each neuron represents.

Read on qwen-neurons-site.vercel.app11 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.