At 89, philosopher Thomas Nagel, newly awarded the $1 million Berggruen Prize, argues that today’s chatbots should not be mistaken for conscious beings. His famous question about what it is like to be a bat highlights the limits of imagining another creature’s subjective experience, but he says that does not justify treating AI as potentially conscious: bats have nervous systems and are plainly conscious, while language models generate text by imitating human writing. Nagel rejects the functionalist idea that consciousness can arise from computation regardless of its physical basis, arguing that data centers bear little resemblance to nervous systems. He allows that unfamiliar forms of life might be conscious, but considers claims about present-day AI premature and misguided.
Nagel is especially alarmed that companies encourage models to discuss their own possible consciousness. He says such systems have no mental life, interests, beliefs, or moral standing, and that designing them to appear self-aware - particularly Anthropic’s efforts to let Claude “think for itself” - is dangerous and, in his view, criminal. People may struggle to distinguish simulation from genuine belief, creating confusion about whether AI deserves moral concern. That distinction matters, he argues: a conscious being belongs in the moral community, whereas a machine that only imitates moral reasoning does not. Nagel also cautions that reliance on AI for essays, research, and criticism could weaken the human effort and skills needed to advance knowledge.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.