Evals for 3D CAD

MUSE is a benchmark and evaluation framework that reframes Text-to-CAD from single-part shape matching to engineering-ready B-Rep assemblies. It provides 106 expert-curated design instances paired with structured Design Specifications that include a design description, a physical assembly graph, a valid parameter space, and a manufacturing plan grounded in process and material constraints. Evaluation follows a three-stage funnel: sandbox execution of generated CadQuery scripts, four binary geometric checks on exported B-Reps (watertightness, manifoldness, self-intersection freedom, overlap freedom), and a rubric-based design-intent assessment using a visual LLM judge. Each instance-specific rubric decomposes into six binary sub-criteria - Assembly-ready, Connectable, Well-toleranced, Functional, Robust, and Manufacturable - so outputs are judged on functionality, manufacturability, and assemblability rather than mere visual similarity.

Experimental results across 15 LLMs reveal a pronounced failure cascade: closed-source models average 42.9% sandbox execution, 30.7% geometric validity, and 20.3% final design-intent score (open-source averages fall from 22.3% execution to 3.7% final). Overlap-free geometry fails most sharply, pointing to multi-component spatial reasoning as the primary bottleneck. GPT-5.5 performs best (77.4% exec, 68.9% geometric validity), and the VLM judge shows strong instance-level agreement with humans (Pearson r ≈ 0.835), supporting scalable automatic evaluation. Limitations include no physical manufacture of designs, lack of assembly-sequence modeling and quantitative physics checks, sensitivity of scores to judge identity, and a relatively small (106-instance) corpus.