777M parameters · 161M active per tokenA 777M-parameter Mixture-of-Experts language model, built and trained from scratch: tokenizer, attention, expert routing, training loop and chat server, all written from scratch and trained on rented GPU time. No fine-tuned base, no borrowed weights, no framework doing the hard part.
The architecture follows DeepSeek-V2: attention that compresses what it caches, and feed-forward layers where only a few experts fire for any given token. The third card is not a modelling idea at all, and without it the first two would still be training.
Instead of caching a key and a value for each of fourteen heads, the input is projected to one shared latent of 320 numbers and keys are rebuilt from it. Position rides separately on a 32-number rotary key, because a position-rotated key cannot be reconstructed from an unrotated latent. The cache holds 352 numbers per token instead of 1,792.
Every layer above the first holds 32 routed experts and one shared expert. A router picks the top 3, the shared expert always runs, so a token passes through 4 of 33. Seventy per cent of the parameters are idle for any given token, which is why 777M costs about as much to run as 161M.
Experts are three stacked (32, hidden, ffn) tensors with capacity-based routing rather than 32 modules in a Python loop, which forced a GPU synchronisation thirty-two times per layer per step. That single change took training from 6,056 to 179,000 tokens per second.
The two pretraining phases ran on separate machines. The learning rate is parameterised by elapsed fraction of training rather than by step, so the second resumed at 0.5227 and the cosine curve continued rather than restarting. Only the weights crossed the boundary.
| Stage | Minutes | Tokens | Result |
|---|---|---|---|
| Pretrain phase 1 | 227 | 2.8B | val loss 3.1486 |
| Pretrain phase 2 | 340 | 3.1B | val loss 2.9763 |
| Chat fine-tune | 80 | 2.7B | val loss 1.2484 |
9,542 steps, 227 min. Steps 1,730-9,160 were never logged: the run that finished phase 1 lost its log with the machine, and what is drawn either side of the gap is the attempt that dropped at step 1,730 and the tail read back from retained console output.
340 min in total, of which the first 105 are logged. Loss is nearly flat here because phase 1 had already taken it to 3.15 and the schedule was annealing.
9,201 steps, 80 min, complete. Validation bottoms out at 1.2484 on step 7,800 and climbs afterwards, so the released weights are that checkpoint and not the last one.
Five-shot, 250 examples per benchmark, every model run through the same harness on the same items. All three Moonfrost checkpoints are shown, so the effect of instruction tuning can be read rather than assumed. One benchmark moves: ARC-Easy falls 54.8, 52.4, 44.4 across the base and the two tunes. Every other gap is inside the noise at this sample size. Figures copied from other model cards would compare harnesses as much as models: Qwen2.5-0.5B publishes 47.5 on MMLU and scores 34.4 here.
| Benchmark | Chance | Moonfrost Base | Moonfrost Instruct v1 | Moonfrost Instruct v2 | SmolLM2-135M | SmolLM2-360M | Qwen2.5-0.5B |
|---|---|---|---|---|---|---|---|
| ARC-Easy | 25.0 | 54.8 | 52.4 | 44.4 | 62.8 | 68.4 | 64.4 |
| ARC-Challenge | 25.0 | 25.2 | 24.4 | 24.4 | 27.6 | 37.2 | 34.8 |
| HellaSwag | 25.0 | 36.0 | 38.4 | 37.2 | 40.0 | 43.6 | 42.4 |
| WinoGrande | 50.0 | 51.2 | 53.2 | 54.0 | 54.0 | 56.0 | 56.8 |
| BoolQ | 50.0 | 62.4 | 61.2 | 58.8 | 62.0 | 63.6 | 65.2 |
| MMLU | 25.0 | 28.8 | 30.0 | 30.8 | 32.4 | 36.8 | 34.4 |
Moonfrost is last or near-last on most rows, and the architecture is not the reason. Six billion tokens for 777 million parameters is about eight tokens per parameter, against a compute-optimal ratio of roughly twenty and against reference models that read two to eighteen trillion at half the size. The budget bought behaviour rather than knowledge: it holds a conversation and writes short working code, but it does not know much.
MMLU is the one score published widely enough to put open and closed models on a single axis, which is the only way GPT-4, Claude and Gemini can appear at all: none of them has published a parameter count or a token budget, so they cannot be placed on the chart above. Bars marked here were measured on this machine; the rest are each developer’s own five-shot figure.
Seventy per cent of the model is experts that stay idle for any given token. That is the whole trade: the capacity of a 777M model at roughly the compute of a 161M one.
| Property | Value |
|---|---|
| Total parameters | 777,148,032 |
| Active per token | 161,036,224 |
| Layers | 14, one dense and thirteen Mixture-of-Experts |
| Hidden size / heads | 896 / 14 |
| Experts | 32 routed with top-3 routing, plus 1 shared |
| Attention | Multi-head Latent Attention, 320 KV latent + 32 rotary key |
| KV cache per token | 352 numbers, against 1,792 for standard attention |
| Context | 1,024 tokens |
| Vocabulary | 32,768, byte-level BPE trained from scratch |
| Pretraining | ~6B tokens of FineWeb-Edu, val loss 2.976 |
| Chat fine-tune | 1.7 epochs, best at step 7,800, val loss 1.2484 |
| Peak / min learning rate | 6e-4 / 6e-5 pretraining, 2e-4 / 2e-5 fine-tune |
| Batch | micro-batch 24, accumulation 12, 294,912 tokens per step |
| Precision | bf16 autocast with fp32 master weights |
| Held-out perplexity | 51.64 on an unseen shard |
| Training throughput | 179,000 tokens/second on one H100 |
| Inference | 16.8 tokens/second, fp32, batch 1, RTX 3050 |
| Compute | 1x H100, ~12 GPU-hours, about $55 |
Weights, code, training data and the raw run logs are all public, including the parts that did not work.
Pretrained weights, 6B tokens of FineWeb-Edu, validation loss 2.976. Continues text rather than answering it. Both tunes start here.
INSTRUCT1.7 epochs of supervised fine-tuning including the persona set. Best validation loss 1.2484 at step 7,800. Answers as Moonfrost and holds a thread.
INSTRUCT0.42 epochs, no identity data, validation loss 1.3251. Claims no persona, which makes it the neutral starting point for a tune of your own.
DATASET140,000 generated identity conversations, 3.3% of the v2 mixture and the whole reason v2 has a stable self-description.
Everything that produced the weights, and this page.
Tokenizer, model, training loop, evaluation harness and chat server. About 6,900 lines of Python, no modelling framework above PyTorch.
REPOSITORYThe source for this page: Next.js, static export, every figure on it read from one module so the page cannot disagree with itself.
The runs as they happened, loss curves included.