The next-generation Featherweight benchmark for sub-1B-parameter language models. 300 hand-written multiple-choice questions across 10 categories, a disjoint question set from v1, and the first Featherweight release with a preliminary human-baseline anchor.
Featherweight v1 demonstrated that a small, hand-written multi-choice benchmark can produce informative scores for sub-1B-parameter language models where MMLU and ARC cluster near chance. It also surfaced its own ceiling: with only 100 questions, the 95% confidence interval on an individual model score is roughly ±9 percentage points. That is wide enough that two models scoring 35% and 40% are not reliably distinguishable on v1 alone.
Featherweight v2 keeps the same task shape and the same calibration target — the sub-1B class — while widening the sample to 300 hand-written questions. This narrows the individual-score confidence interval to roughly ±5.7 percentage points, enough to reliably distinguish models that differ by 6–7 points or more. v2 also introduces a preliminary human baseline on paper copies, anchoring the benchmark to what an average informed reader actually scores rather than only to chance.
Featherweight v2 is a four-way multiple-choice benchmark. Each question has a single correct answer among four labeled choices (A, B, C, D). Answer positions are balanced 75/75/75/75 across the 300 questions so that a model that always picks the same letter scores exactly at chance (25%).
The ten categories are unchanged from v1: geography, history, arithmetic, science, language, vocabulary, common sense, logical reasoning, categorization, and reading comprehension. Each contributes 30 questions to v2's 300. Difficulty is calibrated so that a well-trained ~200M-parameter model has a fighting chance of scoring meaningfully above 25%, with headroom for stronger models to approach the human ceiling.
At n = 300, the worst-case binomial 95% confidence interval on an individual model's accuracy is roughly ±5.7 percentage points. Two models that differ by more than ~8 points on v2 are reliably distinguished; two models within 5 points of each other are not.
At the category level (n = 30 per category), the per-category CI widens to roughly ±18pp — about the same as v1's per-category resolution. Category-level differences should be interpreted as suggestive, not conclusive. Aggregate scores remain the headline number.
v2 does not change the difficulty calibration: it gives the same questions the same answers, just more of them. A model that scored 40% on v1 should land in roughly the same neighborhood on v2, with the gap from chance now more reliably measurable.
Every v2 question is hand-written. No translation, no synthetic generation, no LLM-assisted authoring. Questions were drafted category by category, then reviewed in three passes:
The final 300 are checked against the v1 question set to ensure no duplicates or near-duplicates — v2 is genuinely disjoint, not v1-plus-200.
Representative v2 questions, one per category. Correct answers are revealed beneath each.
The full 300 questions are in featherweight_v2.jsonl, linked in §12 below.
Featherweight v2 is the first release in the series calibrated against a human score, not only against chance. The preliminary human baseline is approximately 86.8%, drawn from a small set of paper-administered runs.
Participants received the full 300-question set on paper, in fixed order, with no time limit and no internet access. They were instructed to answer every question and to guess if uncertain. Scoring was identical to model evaluation: 1 point per correct letter, no partial credit, no penalties for wrong answers. The ~86.8% figure is the average across participants.
The human baseline is an anchor, not a ceiling. It tells you what a small sample of informed adult readers scored under untimed, open-book-prohibited conditions. It is preliminary: the sample is too small to give a tight CI around 86.8%, and the participants are not a representative cross-section. As more paper runs accumulate, this number will be refined; the page will be updated when the human-baseline sample is large enough to publish a confidence interval.
It is not the maximum achievable score. A careful, well-read participant willing to think slowly on the logical-reasoning and reading-comprehension items can likely exceed 86.8%. Several questions in the set are genuinely hard at the tail and were designed that way to give headroom above the human-average mark.
An online version of the v2 test is available. Same 300 questions, same scoring as the paper runs that produced the human baseline. Your responses are stored only in your browser; no answers are sent to a server. Take it in one sitting or pause and resume — progress is saved locally.
The online version is for casual self-evaluation and informal comparison against the ~86.8% human baseline. It is not the canonical evaluation path for models: model scores should be produced with the run_featherweight.py runner against the featherweight_v2.jsonl dataset (linked in §12 below) under the deterministic decoding settings described above.
Initial v2 model evaluations are in. The full multi-model leaderboard mirroring v1's twelve-model field is still being assembled and will be published once those runs are complete. Headline numbers so far:
| Subject | Params | FW v2 | Notes |
|---|---|---|---|
| Human (paper, preliminary) | — | ~86.8% | Small sample; CI not yet published. |
| Epos 1 | ~500M | 86.7% | Statistically indistinguishable from the human baseline above. |
| Haiku H2 | 217M | leader of ≤217M class | Wins its weight class on v2; absolute score reported with full leaderboard. |
The Epos 1 score is anchored by the model card: see the Epos 1 model card for the full context, training details, and intended use. Haiku H2's v2 column will be added to the H2 model card when the full v2 leaderboard is published.
Rootcomputer authors Featherweight v2 and also develops the Haiku and Epos model families. The same model lab evaluating itself on its own benchmark is a real conflict of interest.
Mitigations: the question set is published in full as JSONL under CC-BY-4.0; the runner is published as a standalone Python script; published scores are reproducible by anyone with the open-weights or chathaiku endpoint; the v2 questions are disjoint from any Rootcomputer training corpus and from v1. The honest defense is reproducibility — anyone can run any model on v2 and check the numbers.
Featherweight v2 is also the basis for the human-baseline comparison, which is a Rootcomputer-collected number. The paper runs are not yet large enough to publish externally; when they are, the raw scores will be released alongside the methodology.
The v2 dataset and runner follow the same shape as v1. Drop in the new dataset, point the runner at it, and run.
DatasetThe runner supports HuggingFace local models, OpenAI-compatible HTTP endpoints, and the chathaiku.com shape. Greedy decoding (temperature=0, top_k=1) is the default and is the canonical Featherweight setting. Per-model JSON output plus a summary CSV; per-category breakdowns included.
Featherweight v2 is released under CC-BY-4.0. You may use it freely, including for evaluation of competing or commercial models, provided attribution is given.