AI chatbots becoming experts at changing people's minds. What's their secret?

Commenters debated why AI chatbots can be so good at changing minds. Several argued the effect comes from sheer data and rhetorical polish: probably_wrong said models throw vast (not always accurate) evidence at you, tolugenius pointed to RLHF as central, and lapcat warned that models trained to be persuasive often become less truthful. Others emphasized style and social dynamics: yathern and lemoncookiechip argued that a non-human, polite, tireless interlocutor who avoids offense and flatters the user lowers defensive barriers, while b112 and pixl97 stressed that long-held political identities are resistant and users may simply distrust AI as biased. Some commenters noted practical limits - somenameforme criticized cheap Prolific experiments as unrepresentative, and matthewdgreen and simianwords highlighted model overconfidence and inability to reason from first principles.

Opinion split on consequences and remedies. Several warned of manipulation and moral hazard - probably_wrong compared LLMs to a DDOS or coercive interrogator, rbanffy and others noted political actors already use bots, and intended rejected the idea that releasing many models would reduce manipulation. Others saw potential benefits: simianwords and Grok supporters claimed LLMs can fight misinformation, and marcelo-earth described a personal symbiosis with useful feedback. The divide centers on whether persuasion is a tool for public good or a new vector for coordinated deception, with disagreement about how to evaluate studies and whether pluralistic or regulatory responses would help.