COMPUTER VISION / RESEARCH
FoodGenome AI — Calibrated Food-101 with a Conformal Guarantee
97.16% on Food-101, a confidence figure that means what it says, the full set of candidates the model cannot rule out, and nutrition you can follow back to the USDA record it came from. It also declines to answer — twice, for two different reasons — which turned out to be harder to build than answering.
Public demo
99.56% — conformal coverage, 1.54 candidates
The problem
A Food-101 leaderboard number tells you nothing about whether a given prediction can be trusted. Raw softmax is overconfident, a single top-1 label hides the cases where the model is genuinely torn, and a nutrition figure with no provenance is a number someone made up. The interesting problem is not the accuracy — it is knowing, per image, when the accuracy does not apply.
The approach
Three frozen backbones were run once over all 101,000 images and their embeddings cached — roughly twenty hours, after which every downstream experiment finished in seconds and could therefore actually be ablated. MLP probe heads are trained on the cached embeddings, and the shipped classifier averages the probabilities of the SigLIP-SO400M and EVA-02-L probes; temperature scaling fitted on held-out data cuts expected calibration error eightfold, and split-conformal prediction adds a coverage guarantee that assumes nothing about the model. Abstention needs both signals, because calibrated confidence and conformal set size fail differently. Question answering runs a second pipeline over 693 documents — BM25 ⊕ bi-encoder fused by RRF, cross-encoder reranked, conditioned on the dish the vision model already identified so 'how much sodium is in this' can resolve against 101 near-identical sodium documents.
The results
97.16% test top-1 on a split untouched until final evaluation; 99.56% measured conformal coverage averaging 1.54 candidates; 97.94% on accepted images after abstaining on 2.28%. RAG answers are 98.6% correct on a 76-case gold set and 100% grounded — every quantity is matched against the retrieved sources per unit, so milligrams can never be supported by the same digits in grams, and an answer that fails is withheld rather than annotated. Three results ran against expectation and are reported as such: a parameter-free probability average beat a 3.97M-parameter gated fusion head trained for 21.9 minutes (97.16% vs 96.97%), and its gain over SigLIP alone is significant (exact McNemar, 198 vs 115 discordant, p = 3 × 10⁻⁶); DINOv2-L, expected to decorrelate as the only self-supervised member, is the most correlated of the three and made the ensemble worse; and the pooling head's own attention weights are worse than chance at locating the food, so Grad-CAM does the attribution instead.
Stack: Next.js, FastAPI, PyTorch, SigLIP-SO400M, EVA-02-L, ONNX Runtime, Firebase, Firestore