Models / Haiku H2

Haiku H2

Haiku H2 is the canonical successor to Haiku Mini — a 217M-parameter small language model from Rootcomputer. Modernized architecture, full SFT + DPO post-training pipeline, and a published benchmark profile that wins its weight class.

Overview

H2 is Rootcomputer's second-generation experimental model. It is deliberately small — 217M parameters — so that the full pipeline of pretraining, supervised fine-tuning, and preference alignment can be iterated rapidly on commodity hardware. The goal is quality over scale: H2 trades parameter count for tighter training control, modern architecture choices, and a well-curated corpus.

H2 is available for public testing in open beta via chathaiku.com. It is a research model, not a production system. Its purpose is to make small-model behavior studyable end-to-end, to validate training pipelines that will scale to larger Rootcomputer models, and to publish reproducible benchmarks against comparable open small models.

Architecture

Specifications

H2 is a decoder-only transformer with grouped-query attention, rotary positional embeddings, and SwiGLU activations. Tied input and output embeddings keep parameter count low without sacrificing capacity.

217M
Parameters
16
Transformer layers
1,024
Embedding dim
GQA · 4
Grouped-query attention
(4 KV heads)
50k
RoPE positional base
SwiGLU
Activation
Tied
Input/output embeddings
bf16
Precision
(Flash Attention 2)
Training

From raw model to reliable conversation

H2 is trained in three stages. Each stage is validated independently against a stable upstream checkpoint, and each can be rerun cheaply. DPO is always retrained from the SFT base — never stacked on a previous DPO checkpoint.

01
Pretrain
Base language modeling on the Rootcomputer multi-source corpus — web text, books, news, scientific papers, biomedical sources.
02
SFT
Supervised fine-tuning on conversational data. Teaches H2 the shape of instructions, responses, refusals, and the difference between a short greeting and a substantive answer.
03
DPO
Direct Preference Optimization on accumulated chosen / rejected pairs. Sharpens behavior on identity consistency, format-following, false-premise handling, and answer quality.
Evaluation

Featherweight v1 benchmark

H2 is evaluated on Featherweight v1, a 100-question multi-choice benchmark designed for the sub-1B parameter class. Ten categories of ten questions each: geography, history, arithmetic, science, language, vocabulary, common sense, logic, categorization, reading comprehension. Answer positions are balanced 25/25/25/25 across A/B/C/D so a position-bias-only model scores at chance.

Each model is tested with deterministic greedy decoding (temperature=0, top_k=1, max_new_tokens=20) and two prompt formats. The plain prompt matches the deployed evaluator format, ending with "Answer with just the letter of the correct choice." The completion prompt is an MMLU-style format ending with a bare "Answer:" — fairer to non-instruction-tuned base models. Both columns are reported for transparency.

38–39%
H2 accuracy across both prompt formats. Random chance is 25%.
1st
In its weight class (≤217M parameters), on both prompt formats, by 7–8 points.
+1pp
Score swing between plain and completion prompts — the most prompt-robust model tested.
11
Comparison models, from distilgpt2 (82M) to TinyLlama 1.1B Chat.
Bar chart of Featherweight v1 scores for 12 small language models, sorted by parameter count ascending. H2 (217M) highlighted in bold white, scoring 38% on the plain prompt and 39% on the completion prompt. Beats every model up through 410M parameters. Trails only qwen2.5-0.5b at 70% completion and tinyllama-1.1b-chat at 60% completion.
Accuracy by model and prompt format, sorted by parameter count. H2 highlighted.
Model Params Plain Completion Δ
distilgpt2 82M 27% 24% −3
gpt2 124M 23% 26% +3
opt-125m 125M 28% 23% −5
smollm-135m 135M 26% 19% −7
smollm-135m-instruct135M 30% 32% +2
pythia-160m 162M 25% 28% +3
Haiku H2217M38%39%+1
gpt2-medium 355M 24% 23% −1
smollm-360m 362M 20% 29% +9
pythia-410m 405M 25% 32% +7
qwen2.5-0.5b 494M 40% 70% +30
tinyllama-1.1b-chat 1.1B 45% 60% +15
Featherweight v1, all 100 questions. Δ = completion − plain. With n = 100 the 95% confidence interval on individual scores is roughly ±9 percentage points.
Capability profile

Where H2 is strong, where it isn't

On the completion prompt, H2's strongest categories are geography (60%) and arithmetic (50%). Its weakest are categorization (20%) and logic (30%). The profile is moderately uneven but stays above the 25% chance floor across every one of the ten categories — H2 is not random in any single area.

Radar chart of Haiku H2's per-category accuracy on the Featherweight v1 completion prompt. Ten axes for ten categories. Strongest on geography 60% and arithmetic 50%. Weakest on categorization 20% and logic 30%. Mid-range elsewhere. Polygon stays above the 25% chance ring everywhere.
H2 per-category accuracy on the completion prompt. Polygon stays above the 25% chance ring across all ten categories.

Plotted against larger small models on the same prompt, the picture is honest: Qwen 2.5 (494M) and TinyLlama 1.1B Chat outperform H2 on most knowledge-heavy categories. Qwen reaches 100% on categorization and 90% on common sense. H2's polygon sits nested inside theirs, narrower and more uniform. The gap is real and consistent with the parameter-count and release-date difference (both are 2024-vintage, larger, trained on more recent corpora).

Overlaid radar chart of Haiku H2 (solid line) vs TinyLlama 1.1B chat (dashed) vs Qwen 2.5 0.5B (dotted) on the completion prompt. H2 is the smaller polygon nested inside the other two. Qwen reaches 100% on categorization and 90% on common sense; TinyLlama is most balanced.
H2 (solid) vs TinyLlama (dashed) vs Qwen (dotted). H2's polygon is contained inside both larger models — the size and recency gap is consistent across most categories.
Position

Three claims that hold across both prompts

  • H2 wins its weight class. Among the seven models in the comparison with ≤217M parameters — distilgpt2, gpt2, opt-125m, smollm-135m (base and instruct), pythia-160m, and H2 itself — H2 scores highest on both the plain (38% vs next-best 30%) and completion (39% vs next-best 32%) prompts. The 7–8 point gap is comfortably larger than the benchmark's ±9 point confidence interval and survives both formats.
  • H2 beats every 218–410M base model in the field. gpt2-medium (355M, 23–24%), smollm-360m (362M, 20–29%), and pythia-410m (405M, 25–32%) all trail H2 by 7–18 points on both prompts despite having more parameters. This is the clearest signal in the data that H2's full SFT + DPO post-training is doing real work — scale alone, on a base-pretrained model, does not get you instruction-following at this size.
  • H2 is the most prompt-robust model in the field. H2 shifts by 1 percentage point between the plain and completion prompts. The next-most-robust model shifts by 3 points; Qwen 2.5 shifts by 30 and TinyLlama by 15. Prompt robustness is a property in its own right — a user who phrases a question one way versus another should get answers of similar quality, and H2 delivers that more reliably than any model in the comparison.

H2 trails two models, both larger and from a more recent vintage: Qwen 2.5 0.5B (494M, 2024) by approximately 30 points on the completion prompt, and TinyLlama 1.1B Chat (1.1B, 2024) by approximately 20 points. These gaps are real and reflect genuine differences in scale and training recency. They are not claims to dispute. The claim H2 makes is that within its weight class, and against pre-2024 architectures up through ~410M parameters, it is the strongest small model we have tested.

Intended use

What H2 is for

  • Research baseline. A reproducible, fully-documented small language model that other researchers can compare against, fork, and improve on.
  • Studying small-model behavior. Understanding how SFT and DPO interact at the 200M-parameter scale, where capabilities emerge and where they fail.
  • Pipeline validation. Testing architecture, corpus, and training decisions on H2 before committing to longer runs on larger models in the Haiku family.
  • Casual conversational interaction. Via chathaiku.com, for users who want to interact with a competent, fast, small chat model.
  • Featherweight benchmark reference. H2 anchors the Featherweight v1 leaderboard at its weight class, and serves as the reference SFT+DPO-trained baseline for comparison with other small models.
Limitations

What H2 is not for

H2 is a 217M parameter research model. The same properties that make it fast and iterable make it unsuitable for any safety-critical or precision-critical use. Known weaknesses, drawn directly from the Featherweight v1 results and ongoing development:

  • Multi-step logical reasoning is near chance. Logic category accuracy is 20–30%. Syllogisms, contrapositive reasoning, and chained inferences are not reliably solved.
  • Arithmetic beyond single-digit multiplication is unreliable. Even on simple problems, H2 can produce answers that are not among the provided choices (e.g. "7 × 8 = 57"). Do not use H2 for any calculation you would not personally verify.
  • Knowledge is bounded by the training corpus. H2 will confidently produce incorrect facts on topics outside its corpus or after its cutoff. Treat all factual outputs as starting points for verification, not endpoints.
  • Hallucination risk on long-form factual queries. The model's tendency to fill the available token budget means it will generate plausible-sounding continuations even when it doesn't have the underlying knowledge.
  • Not suitable for medical, legal, financial, or safety-critical applications. H2 has not been evaluated for any of these use cases and has no safety guarantees beyond standard conversational refusals trained during SFT and DPO.
  • Long-context performance untested at scale. H2 is evaluated on short conversational exchanges. Behavior on long-context inputs (multi-thousand-token documents) is not characterized in this card.
Reproducibility

Reproducing these numbers

The Featherweight v1 benchmark, the runner that produced these tables, and the model-comparison configuration used in this report are all available. Anyone with a CUDA GPU and pip install transformers torch can reproduce the comparison table for the open-weights models. The H2 column is reproducible against the public chathaiku.com endpoint.

File
100 questions across 10 categories, balanced 25/25/25/25 across A/B/C/D, with optional metadata header.
File
Standalone Python runner. Supports HuggingFace local, HTTP (chathaiku-shape), and OpenAI-compatible backends. Greedy decoding, deterministic, per-model JSON plus a summary CSV.