COMPLETED RESEARCH PROJECT

FoodGenome AI

A 97% food classifier that also says, per image, when not to trust it

Research question
When a food classifier is right 97% of the time on a benchmark, can it also say — for each image — when its answer should not be trusted?
Problem
A Food-101 leaderboard number says nothing about an individual prediction. Raw softmax confidence is overconfident, and a single top-1 label hides the cases where the model is genuinely torn between near-identical dishes.
Dataset
Food-101 (Bossard et al., 2014): 101 classes, 101,000 images, official 75,750 train / 25,250 test. A fixed 4% class-stratified validation slice is carved from train (seed 1337; 3,030 images) and used for every selection, calibration and conformal threshold. The test split was untouched until final evaluation.
Setup
Embeddings from frozen SigLIP-SO400M, EVA-02-L and DINOv2-L were computed once over all 101,000 images (about 20 hours) and cached, so each downstream experiment runs in seconds. MLP probe heads are trained on the cached embeddings (72,720 training images). The shipped classifier averages the SigLIP and EVA-02 probabilities; temperature scaling is fitted on validation (T = 0.7621); split-conformal sets (LAC and APS) are calibrated on the 3,030 validation images.

Results

  • 97.16% — top-1, Food-101 test (n = 25,250) (source: web/data/reports/ensemble.json)
  • 0.051 → 0.006 — expected calibration error after temperature scaling (source: web/data/reports/calibration_ensemble_siglip_eva02.json)
  • 99.56% — conformal coverage at α = 0.01, 1.54 candidates on average (source: web/data/reports/conformal.json)

Limitations

  • Probe heads on frozen backbones — no end-to-end fine-tuning was compared.
  • One benchmark. Food-101's training images contain label noise by design, and photos of regional dishes are out of distribution.
  • The conformal guarantee is marginal and assumes calibration and test images are exchangeable; it does not hold under distribution shift.
  • Not peer reviewed; no manuscript.

Contribution

Sole author: data pipeline, backbone selection, probes, calibration, conformal prediction, the retrieval-grounded nutrition layer and the deployed application.

Code and demo are public.