A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis

A dermatologist compared GPT-6 Astra and Claude Fable 5.1 with a fine-tuned Inception v3 CNN on 200 randomly selected, biopsy-confirmed skin lesions from the MILK10k dataset. Each lesion had a clinical photograph and a dermoscopic image; the models had to choose among six diagnoses and say whether it could be discharged. The sample included 100 lesions needing treatment or follow-up, including 30 melanomas, and 100 benign moles or seborrhoeic keratoses. The CNN, an older general-purpose architecture adapted to dermatology images, had not been trained on the test lesions. It correctly named 125 lesions (62%), compared with 84 and 85 (42%) for the two LLMs. Its advantage was especially clear for benign lesions, while it also identified more melanomas than either LLM, though it missed 17 of 30.

The models made different errors: GPT-6 Astra often labelled lesions basal cell carcinoma, while Claude Fable 5.1 frequently chose actinic keratosis. The CNN’s preset discharge threshold kept fewer unsafe lesions but also discharged more benign ones; when its threshold was adjusted to match the LLMs’ safety performance, it missed only one or two melanomas, versus three and four for the LLMs. In a separate patient-style prompt using clinical images alone, both LLMs advised seeing a doctor in almost every case, including for harmless lesions. The results are limited by the small sample, run-to-run variation, and unusually difficult biopsy-selected cases. The CNN is a baseline, not state of the art, and the test does not establish how these systems perform in routine care.