COMPUTER VISION / RESEARCH

DermAI — Cascaded Skin Triage with Retrieval-Grounded Guidance

An OOD gate, a calibrated 21-class skin-condition classifier, and a multimodal advisor whose every clinical sentence traces to a MedlinePlus passage — plus the data forensics that had to happen first. Not a medical device: an educational research demo on public datasets.

Research prototype · code private

36.4pp — provenance shortcut removed

The problem

Public notebooks on this dataset routinely claim 95%+ accuracy. Those numbers are real arithmetic and false claims. The vendor had inflated every class to exactly 2,000 images with rotations of the same photos, so a random split scatters copies across train and test — and because each class came from a different source dataset, the source itself predicted the label. The hard problem was not fitting a model; it was proving the benchmark was fiction before trusting a single number.

The approach

Three defects found by measurement before any training. (1) Augmentation leakage — only 28,848 of 54,000 images are real, so splits group on base photo identity and augmented copies are excluded from val/test entirely. (2) A provenance shortcut — a random forest on image dimensions and file size alone, never loading a pixel, scored 99.0% on skin-vs-non-skin; fixed by canonicalising every image (384², uniform JPEG q90, EXIF stripped) and merging 27 folders into 20 classes + Healthy so no class is provenance-pure. (3) Label fiction — a candidate 'Normal' folder contained a leopard, a banana and a shoe rack, so Healthy was curated with zero-shot CLIP and then inspected by eye. Inference is a cascade with two exits a single softmax cannot express: a MobileNetV3 OOD gate rejects non-skin, an EfficientNet-B0 classifier abstains below 30% confidence, and gpt-5 reasons over the photo, the symptoms, the calibrated distribution and BM25 ⊕ dense passages fused by RRF.

The results

The metadata probe collapses from 99.0% to 62.5% on canonicalised data — below the 62.6% majority floor — while the trained gate still hits 99.6% balanced accuracy, so that accuracy is earned from skin rather than file size. The deployed EfficientNet-B0 reports 0.724 balanced accuracy [0.706, 0.741] and 92.2% top-3 across 21 classes with ECE 0.023 after temperature scaling, measured with grouped splits and bootstrap CIs. A ConvNeXt-Small ablation with 5× the parameters ties it and calibrates markedly worse — the bottleneck is label noise, not capacity. A second, harder shortcut is reported rather than buried: a CNN predicts the source dataset from pixels at 93.4%, which compromises the malignant-vs-inflammatory boundary and is documented as such.

Stack: React 19, Vite 8, FastAPI, PyTorch, timm, CLIP, sentence-transformers, gpt-5