MULTIMODAL RETRIEVAL / RESEARCH
RoboDesign Atlas — Cross-Modal Retrieval over RobotDesign1M
Text→figure search over 95,000 scientific figures streamed out of a 1.08 TB corpus, a domain-adapted CLIP that lifts R@1 by 39%, and an audit showing the dataset is far less robot-specific than its name. The demo runs the fine-tuned text tower in your browser — no backend, nothing sent to a server.
Public demo
83 → 26 — median rank, 5,000 candidates
The problem
Zero-shot CLIP is trained on natural web imagery and lands the correct scientific figure first for roughly one query in eight — median rank 83 out of 5,000. And the corpus underneath is itself unexamined: RobotDesign1M was assembled by hand-labelling 32k images, training a classifier, then filtering figures out of PDFs at scale. Classifier-filtered corpora drift, and nobody had measured how far.
The approach
Images are embedded inline in the parquet shards, so the release is 1.08 TB; a streaming builder reads row groups over HTTP and keeps 0.14% of the source with the official split boundaries intact — 8×8 perceptual hashing rejects 1.7% of train candidates and the seen-set is shared across splits so a figure can never leak from train into test. Adaptation unfreezes only the top four transformer blocks of each tower plus projections and temperature (27.5% of parameters) and re-runs symmetric InfoNCE for four epochs — 58 min on an M3 Pro. The audit scores every caption against a seven-domain keyword lexicon that ships in the repo so it can be inspected and argued with.
The results
R@1 12.40 → 17.24 text→image on the official 5,000-candidate test split, R@10 27.18 → 38.34, median rank 83 → 26 — a 3.2× improvement in where the correct figure lands, significant under McNemar's exact paired test at p < 1e-12. The gain is the visual domain, not caption boilerplate: re-running the whole pipeline on the harder conversations config, whose text side has 19,468 distinct instructions rather than 228, reproduces the relative R@1 gain to within 0.06 points (+38.97% vs +39.03%). The audit finds only 5.58% of captions match robotics terms exclusively and 50.0% match none of the seven lexicons — stable to 1.3 points across all six config/split combinations, so corpus-wide rather than split skew.
Stack: Next.js 16, FastAPI, PyTorch, CLIP ViT-B/32, FAISS, ONNX Runtime Web, Hugging Face, uv