hn.today

Thomas Nagel says that it's criminal to build AI that feigns consciousness

wired.com19 points32 comments
Screenshot of Thomas Nagel says that it's criminal to build AI that feigns consciousness

At 89, philosopher Thomas Nagel, newly awarded the $1 million Berggruen Prize, argues that today’s chatbots should not be mistaken for conscious beings. His famous question about what it is like to be a bat highlights the limits of imagining another creature’s subjective experience, but he says that does not justify treating AI as potentially conscious: bats have nervous systems and are plainly conscious, while language models generate text by imitating human writing. Nagel rejects the functionalist idea that consciousness can arise from computation regardless of its physical basis, arguing that data centers bear little resemblance to nervous systems. He allows that unfamiliar forms of life might be conscious, but considers claims about present-day AI premature and misguided.

Nagel is especially alarmed that companies encourage models to discuss their own possible consciousness. He says such systems have no mental life, interests, beliefs, or moral standing, and that designing them to appear self-aware - particularly Anthropic’s efforts to let Claude “think for itself” - is dangerous and, in his view, criminal. People may struggle to distinguish simulation from genuine belief, creating confusion about whether AI deserves moral concern. That distinction matters, he argues: a conscious being belongs in the moral community, whereas a machine that only imitates moral reasoning does not. Nagel also cautions that reliance on AI for essays, research, and criticism could weaken the human effort and skills needed to advance knowledge.

Read on wired.com32 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

Mapping every neuron in Qwen3-8B

Mapping every neuron in Qwen3-8B

Researchers found individual neurons in Qwen3-8B that reliably steer its replies into distinct personas or styles, including pirate, German-speaking, and childlike voices. Nudging these neurons changed outputs across many questions without prompting or fine-tuning, revealing how specific behaviors can be localized and edited inside a language model. (fermionresearch.com)

ChatGPT for SNES

ChatGPT for SNES

ChatGPT for SNES runs a 2.2-million-parameter language model on the Super Nintendo’s Super FX 2 chip, with an optional Wikipedia fact book. The unofficial experiment can run in an emulator or on a flash-cart-equipped console, but its tiny model often gives incorrect answers. (bxfoundry.com)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.