LLMs are terrible at designing AI systems

LLMs can write conventional software yet often choose poor architectures when designing systems that use LLMs. In one example, an agent tasked with detecting contradictions among natural-language memories built a symbolic logic system to convert statements into structured atoms and compare them mechanically. The approach was doomed because determining whether two sentences express the same proposition cannot be solved analytically; a language model could instead judge the statements directly. The agent eventually explained this limitation when challenged, but only after spending weeks pursuing the failed design. The author has seen similar choices elsewhere, such as using regex or keyword searches to categorize AI-generated study cards rather than making another model call.

The author considers three explanations: most training code predates LLMs; reinforcement-learning coding environments may not reward software that calls external models; or, more fundamentally, AI agents are weak at software design and routinely choose the wrong tool. The first two alone seem insufficient, while the third fits other mistakes, such as building elaborate consensus mechanisms instead of accepting stale data. The broader lesson is that LLMs combine impressive expertise and execution with striking failures of common sense, so their strengths and weaknesses are hard to predict from human standards. Effective use depends on anticipating which tasks they can handle well and steering them away from costly dead ends.