COMPLETED RESEARCH PROJECT
RoboDesign Atlas
Domain-adapting CLIP to retrieve engineering figures, and auditing what the corpus contains
- Research question
- How much does lightweight domain adaptation help CLIP retrieve scientific and engineering figures from text — and what is the RobotDesign1M corpus actually made of?
- Problem
- Zero-shot CLIP is trained on natural web imagery. On scientific figures it ranks the correct figure first for about one query in eight, with a median rank of 83 out of 5,000 candidates. The corpus itself was assembled by a classifier filter and had not been audited.
- Dataset
- RobotDesign1M (Le et al., IROS 2026; 1.08 TB, images inline in parquet). A streaming builder kept 0.14% of the source with the official split boundaries intact: image_caption config, 60,000 train / 5,000 val / 5,000 test. 8×8 perceptual-hash deduplication with a seen-set shared across splits, so no figure can leak from train into test.
- Setup
- CLIP ViT-B/32. The top four transformer blocks of each tower, the projections and the temperature are unfrozen (27.5% of parameters) and trained with symmetric InfoNCE for four epochs (58 minutes on an Apple M3 Pro). Retrieval uses FAISS exact cosine over the 5,000 test pairs, 1:1 pairing.
Results
- 83 → 26 — median rank of the correct figure, text→image, 5,000 candidates (source: README.md results table)
- 12.40 → 17.24 — Recall@1, text→image (exact McNemar p ≈ 7 × 10⁻³¹) (source: artifacts/reports/significance.json)
- 5.58% — captions that match robotics terms and nothing else (source: README.md corpus audit)
Limitations
- One backbone (ViT-B/32) and one adaptation recipe; no comparison with larger CLIP variants or other domain-specific encoders.
- 0.14% of the corpus; the image_caption text side has only 228 distinct templates.
- The audit uses keyword lexicons that ship in the repository — coarse, and not a human annotation.
- Not peer reviewed; no manuscript.
Contribution
Sole author: dataset streaming and deduplication, fine-tuning, evaluation, significance testing, corpus audit and the in-browser demo. The underlying dataset is Le et al.'s.
Code and demo are public.