LANGUAGE MODELLING / SYSTEMS
NWP-Core — A Transformer Trained From Scratch, Running In Your Tab
A 19.5M-parameter next-word transformer trained from scratch on 262M tokens, quantised to int8 and executed in a Web Worker via WebAssembly. It shows you the distribution rather than one word — candidates with real probabilities, entropy in bits, a two-level branch lattice, and a per-token surprisal profile of your own prose. Nothing you type leaves the machine.
Public demo
60.7% — top-5 accuracy, 19.5M params, 0 servers
The problem
Autocomplete shows you one word and hides the fact that the model produced a probability distribution over 16,384 of them. A 12%-confident suggestion is rendered identically to a 90%-confident one, so the interface systematically misrepresents what the model knows. Underneath that is a harder constraint: to show the distribution honestly the model has to be small enough to run on the reader's own hardware, because round-tripping every keystroke to a server is neither private nor fast enough to keep a caret responsive.
The approach
An 8-layer decoder — RMSNorm, RoPE, SwiGLU, grouped-query attention at 6 query heads to 2 KV, and tied embeddings that save 6.3M parameters on a 16k vocabulary. Model shape was chosen by measuring MPS throughput rather than guessing: L8/E384 lands at 26 tokens per parameter, just past Chinchilla's compute-optimal ratio, which is the right side to err on when the weights also have to be downloaded. None of the usual micro-optimisations (enable_gqa, native-dtype RMSNorm, torch.compile) moved throughput at all — shape was the only lever, which is why a benchmark drove the config. The browser tokenizer is a reimplementation of the Python one, so a parity harness checks them across 24,241 tokens spanning emoji, CJK, Cyrillic and whitespace edge cases: if a single merge disagreed the model would be fed ids it never saw in training and degrade silently, with no error anywhere.
The results
Perplexity 28.3 held-out, top-5 accuracy 60.7% — the word you wanted is in the five shown about three times in five. Loss fell 9.76 → 3.34 over 16,000 steps and 7.8 hours on an M3 Pro. Shipping int8 was decided by measurement, not default: quantisation costs 3.13% perplexity but only 0.27 points of top-5, and since top-5 is what a person actually experiences, that is the number the ship gate reads — cutting the download 4× from 103.5MB to 26.6MB. Measured in-browser latency is 15–20 ms per prediction with zero inference servers, so prediction cost scales with users at exactly zero marginal compute. Row-level security covers every table, admin mutations route through audited SECURITY DEFINER RPCs, and the profile update policy re-reads role from the existing row so self-promotion to admin is impossible. The known limitations are documented rather than hidden: 262M tokens is under one epoch and the model is still underfitting.
Stack: PyTorch, ONNX Runtime Web, WebAssembly, Next.js 16, TypeScript, Supabase, Postgres RLS, Vercel