A software engineer with deep LLM-driven development experience bought a preassembled LeRobot SO-101 arm, a webcam and repurposed an old Raspberry Pi 3 to create a physical sandbox for LLM-guided robotics. The build included a simple web dashboard, servo protection code to cut torque on overloads, and AprilTag-based calibration. Early work used Codex/Astra to provision the Pi and scaffold control code, but the project ran into practical issues: a weak power supply caused serial drops when multiple servos moved, and an open-loop inverse-kinematics calibration produced positions that didn’t match camera observations.
Shifting strategy, the engineer handed control to Claude/Fable and removed the human-in-the-loop, adding resistance-based zeroing for the gripper and camera feedback to create a closed-loop pick-and-place routine. The LLM iteratively corrected mistakes (including an overaggressive grip), improved vision by swapping high-contrast tape on blocks, and completed an overnight task that placed three colored blocks into a target area. Workflows produced interval screenshots that were compiled into a timelapse; next steps are converting LLM training into VLA (vision-language-action) data and attempting more complex manipulation like Jenga towers.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.