Vals AI and NextLadder Ventures tested 23 general-purpose AI models on 252 simulated conversations about civil legal problems faced by low-income Americans, including eviction, custody, debt, and domestic violence. The scenarios were adapted by six legal-aid experts from real requests and included facts models had to ask about. No model met the organizers’ 80% reliability threshold; Alibaba’s Qwen 3.8 Max led with 76.4%. Models explained general legal principles relatively well, satisfying 78-87% of those criteria, but identified relevant local services, forms, or court resources only 30-50% of the time. Most instead suggested national directories. Free and premium models performed almost identically on average. Housing questions scored highest, while public-benefits questions scored lowest; family-law results varied widely by issue.
Examples show how missing details could affect users’ next steps. In a Louisiana small-claims scenario, most models gave an incorrect 10-year limitation period instead of the three-year period for money lent, and few asked whether the debt concerned real estate or pointed to court forms. In a New York constructive-eviction case, many advised a demand letter despite the rubric warning against that step, and only four mentioned the city’s Right to Counsel program. Results are limited to the 252 public scenarios, and API testing may not reflect consumer chatbot use; some refusals were scored as zero. The findings highlight a practical gap: legal explanations are not enough if systems fail to identify the local help a person can actually use.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.