Haiku H2
Haiku H2 is the canonical successor to Haiku Mini — a 217M-parameter small language model from Rootcomputer. Modernized architecture, full SFT + DPO post-training pipeline, and a published benchmark profile that wins its weight class.
Overview
H2 is Rootcomputer's second-generation experimental model. It is deliberately small — 217M parameters — so that the full pipeline of pretraining, supervised fine-tuning, and preference alignment can be iterated rapidly on commodity hardware. The goal is quality over scale: H2 trades parameter count for tighter training control, modern architecture choices, and a well-curated corpus.
H2 is available for public testing in open beta via chathaiku.com. It is a research model, not a production system. Its purpose is to make small-model behavior studyable end-to-end, to validate training pipelines that will scale to larger Rootcomputer models, and to publish reproducible benchmarks against comparable open small models.
Specifications
H2 is a decoder-only transformer with grouped-query attention, rotary positional embeddings, and SwiGLU activations. Tied input and output embeddings keep parameter count low without sacrificing capacity.
(4 KV heads)
(Flash Attention 2)
From raw model to reliable conversation
H2 is trained in three stages. Each stage is validated independently against a stable upstream checkpoint, and each can be rerun cheaply. DPO is always retrained from the SFT base — never stacked on a previous DPO checkpoint.
Featherweight v1 benchmark
H2 is evaluated on Featherweight v1, a 100-question multi-choice benchmark designed for the sub-1B parameter class. Ten categories of ten questions each: geography, history, arithmetic, science, language, vocabulary, common sense, logic, categorization, reading comprehension. Answer positions are balanced 25/25/25/25 across A/B/C/D so a position-bias-only model scores at chance.
Each model is tested with deterministic greedy decoding (temperature=0, top_k=1, max_new_tokens=20) and two prompt formats. The plain prompt matches the deployed evaluator format, ending with "Answer with just the letter of the correct choice." The completion prompt is an MMLU-style format ending with a bare "Answer:" — fairer to non-instruction-tuned base models. Both columns are reported for transparency.
| Model | Params | Plain | Completion | Δ |
|---|---|---|---|---|
| distilgpt2 | 82M | 27% | 24% | −3 |
| gpt2 | 124M | 23% | 26% | +3 |
| opt-125m | 125M | 28% | 23% | −5 |
| smollm-135m | 135M | 26% | 19% | −7 |
| smollm-135m-instruct | 135M | 30% | 32% | +2 |
| pythia-160m | 162M | 25% | 28% | +3 |
| Haiku H2 | 217M | 38% | 39% | +1 |
| gpt2-medium | 355M | 24% | 23% | −1 |
| smollm-360m | 362M | 20% | 29% | +9 |
| pythia-410m | 405M | 25% | 32% | +7 |
| qwen2.5-0.5b | 494M | 40% | 70% | +30 |
| tinyllama-1.1b-chat | 1.1B | 45% | 60% | +15 |
Where H2 is strong, where it isn't
On the completion prompt, H2's strongest categories are geography (60%) and arithmetic (50%). Its weakest are categorization (20%) and logic (30%). The profile is moderately uneven but stays above the 25% chance floor across every one of the ten categories — H2 is not random in any single area.
Plotted against larger small models on the same prompt, the picture is honest: Qwen 2.5 (494M) and TinyLlama 1.1B Chat outperform H2 on most knowledge-heavy categories. Qwen reaches 100% on categorization and 90% on common sense. H2's polygon sits nested inside theirs, narrower and more uniform. The gap is real and consistent with the parameter-count and release-date difference (both are 2024-vintage, larger, trained on more recent corpora).
Three claims that hold across both prompts
- H2 wins its weight class. Among the seven models in the comparison with ≤217M parameters — distilgpt2, gpt2, opt-125m, smollm-135m (base and instruct), pythia-160m, and H2 itself — H2 scores highest on both the plain (38% vs next-best 30%) and completion (39% vs next-best 32%) prompts. The 7–8 point gap is comfortably larger than the benchmark's ±9 point confidence interval and survives both formats.
- H2 beats every 218–410M base model in the field. gpt2-medium (355M, 23–24%), smollm-360m (362M, 20–29%), and pythia-410m (405M, 25–32%) all trail H2 by 7–18 points on both prompts despite having more parameters. This is the clearest signal in the data that H2's full SFT + DPO post-training is doing real work — scale alone, on a base-pretrained model, does not get you instruction-following at this size.
- H2 is the most prompt-robust model in the field. H2 shifts by 1 percentage point between the plain and completion prompts. The next-most-robust model shifts by 3 points; Qwen 2.5 shifts by 30 and TinyLlama by 15. Prompt robustness is a property in its own right — a user who phrases a question one way versus another should get answers of similar quality, and H2 delivers that more reliably than any model in the comparison.
H2 trails two models, both larger and from a more recent vintage: Qwen 2.5 0.5B (494M, 2024) by approximately 30 points on the completion prompt, and TinyLlama 1.1B Chat (1.1B, 2024) by approximately 20 points. These gaps are real and reflect genuine differences in scale and training recency. They are not claims to dispute. The claim H2 makes is that within its weight class, and against pre-2024 architectures up through ~410M parameters, it is the strongest small model we have tested.
What H2 is for
- Research baseline. A reproducible, fully-documented small language model that other researchers can compare against, fork, and improve on.
- Studying small-model behavior. Understanding how SFT and DPO interact at the 200M-parameter scale, where capabilities emerge and where they fail.
- Pipeline validation. Testing architecture, corpus, and training decisions on H2 before committing to longer runs on larger models in the Haiku family.
- Casual conversational interaction. Via chathaiku.com, for users who want to interact with a competent, fast, small chat model.
- Featherweight benchmark reference. H2 anchors the Featherweight v1 leaderboard at its weight class, and serves as the reference SFT+DPO-trained baseline for comparison with other small models.
What H2 is not for
H2 is a 217M parameter research model. The same properties that make it fast and iterable make it unsuitable for any safety-critical or precision-critical use. Known weaknesses, drawn directly from the Featherweight v1 results and ongoing development:
- Multi-step logical reasoning is near chance. Logic category accuracy is 20–30%. Syllogisms, contrapositive reasoning, and chained inferences are not reliably solved.
- Arithmetic beyond single-digit multiplication is unreliable. Even on simple problems, H2 can produce answers that are not among the provided choices (e.g. "7 × 8 = 57"). Do not use H2 for any calculation you would not personally verify.
- Knowledge is bounded by the training corpus. H2 will confidently produce incorrect facts on topics outside its corpus or after its cutoff. Treat all factual outputs as starting points for verification, not endpoints.
- Hallucination risk on long-form factual queries. The model's tendency to fill the available token budget means it will generate plausible-sounding continuations even when it doesn't have the underlying knowledge.
- Not suitable for medical, legal, financial, or safety-critical applications. H2 has not been evaluated for any of these use cases and has no safety guarantees beyond standard conversational refusals trained during SFT and DPO.
- Long-context performance untested at scale. H2 is evaluated on short conversational exchanges. Behavior on long-context inputs (multi-thousand-token documents) is not characterized in this card.
Reproducing these numbers
The Featherweight v1 benchmark, the runner that produced these tables, and the model-comparison configuration used in this report are all available. Anyone with a CUDA GPU and pip install transformers torch can reproduce the comparison table for the open-weights models. The H2 column is reproducible against the public chathaiku.com endpoint.