We probably cannot automate the software engineer away

A teaching assistant for Carnegie Mellon’s undergraduate software engineering course built an autograder that reads 1,200 paper participation sheets from 89 students using AI agents without writing any code. The grader handled roster matching, handwriting recognition and scoring, but real-world usage exposed many hidden assumptions: students drew ducks answering prompts, circled and spelled out multiple choices, submitted faint pencil marks, or left snarky notes that confused the model. Those quirks forced repeated specification changes, architectural refactoring to extract reusable components, and judgment calls that no prompt could have anticipated. Iteration on real inputs revealed requirements and priorities the designer only discovered by deploying the system.

The core argument is that AI removes accidental complexity - writing code - but cannot replace the essential work of deciding what to build, testing assumptions against the world, and responding to unknown unknowns. Drawing on Brooks, Jackson, and examples from aviation, distributed storage, and model safety failures, the case shows that unforeseen environmental behaviors are the bottleneck. AI speeds iteration and helps implement specifications, but it cannot automate noticing or choosing how to handle phenomena it was never told about; human engineers still must interpret, update specifications, and make value judgments.