Featherweight v1

Why Featherweight exists

Established multi-choice benchmarks — MMLU, ARC, HellaSwag, TruthfulQA — were designed for, and calibrated against, much larger models. Run them against a 200M-parameter model and most produce uninformative leaderboards: scores cluster near chance, sub-1B models look indistinguishable, and the small differences that do exist are buried in noise.

Featherweight v1 is calibrated for the sub-1B parameter class. The questions are chosen so that a well-trained 200M-parameter model has a fighting chance of scoring meaningfully above chance, while leaving headroom for larger and better-trained models to score higher still. The goal is to give small models a place to actually compete — not to compete with multi-billion-parameter models, but to be informatively ranked against each other.

What the benchmark asks

Each item is a single multi-choice question with exactly four labelled choices (A through D) and exactly one correct answer. The benchmark contains 100 items, partitioned evenly across 10 categories of 10 items each:

• Geography. Capitals, countries, continents, oceans, mountain ranges, major rivers.
• History. Dates of major world events, well-known historical figures, broad civilizational facts.
• Arithmetic. Single-step integer and simple fraction arithmetic, basic geometry constants.
• Science. Elementary physics, biology, chemistry; well-established, non-controversial facts.
• Language. Parts of speech, simple grammar judgements, plurals, tenses, common spelling.
• Vocabulary. Synonyms, antonyms, definitions of common English words.
• Common sense. Everyday reasoning about objects, weather, time, basic causation.
• Logic. One-step syllogisms, transitive reasoning, contrapositives, basic counting.
• Categorization. Identifying which item belongs / does not belong to a category.
• Reading. Short two-to-three-sentence passages with a factual question.

Scoring is one point per correct answer, no partial credit, no negative marking. Final score is reported as a percentage. Random baseline is 25%; a model that always answers a single letter scores 25% by construction (see §06).

Two formats, both reported

Multiple-choice benchmarks are sensitive to prompt format. Models that have been instruction-tuned on multi-choice data tend to score higher on prompts that look like instructions; non-instruction-tuned base models tend to score higher on prompts that look like text completions. To avoid privileging either family, Featherweight specifies two prompt templates and runs the full evaluation under both. Both columns are reported.

Plain prompt

Closer to a deployed chat interface. Used as the canonical "instruction-style" baseline. Fed to the model as the user turn.

{question}

A) {choice_0}
B) {choice_1}
C) {choice_2}
D) {choice_3}

Answer with just the letter of the correct choice.

Completion prompt

An MMLU-style format ending with a bare "Answer:". Fed as a raw text prefix the model is asked to continue. Fairer to base models that have not seen explicit instruction-following data.

Question: {question}
A) {choice_0}
B) {choice_1}
C) {choice_2}
D) {choice_3}
Answer:

For chat-templated models (TinyLlama-Chat, SmolLM-Instruct), the prompt above is wrapped in the model's tokenizer chat template via apply_chat_template(messages, add_generation_prompt=True). The user message is the same text as for non-chat models — only the surrounding tokens differ.

Deterministic by design

Every model in every run uses identical greedy decoding parameters. There is no temperature sampling, no top-p or top-k truncation, and no repetition penalty. The same model on the same prompt produces the same output every time.

temperature       = 0.0
top_p             = 1.0
top_k             = 1
max_new_tokens    = 20
repetition_penalty = 1.0
no_repeat_ngram   = 0

The max_new_tokens = 20 cap is generous — the expected output is a single letter, and 20 tokens leaves room for a short preamble like "The answer is A". Across all 12 models tested for v1, no model was truncated mid-letter under this cap.

Deterministic decoding means the benchmark has no run-to-run variance from sampling noise. The only sources of variance are (a) the model itself, (b) the prompt format, and (c) potential floating-point non-determinism in some inference backends, which is generally below 0.5pp and below the noise floor set by sample size.

From model output to a letter

The model produces up to 20 tokens of free-form text. The benchmark must reduce this to one of {A, B, C, D, parse-error}. The extraction algorithm is a fixed priority list, applied to the model's reply in this order:

1. Parenthesized letter. If the reply contains (A), (B), (C), or (D) anywhere, that letter is returned. Catches "(B) Paris" outputs.
2. "Answer is X" pattern. If the reply contains the substring answer (case-insensitive) followed by an optional is, optional punctuation, and a standalone letter A–D, that letter is returned. Catches "The answer is B" and "Answer: C".
3. Reply-starts-with-letter pattern. If the reply begins (after optional whitespace) with a letter A–D followed by space, period, comma, or closing bracket, that letter is returned. Catches "A) Berlin..." and "C. The answer is...".
4. Choice-text match. If the reply contains the full text of one of the four choices (case-insensitive, word-boundary aware), the letter corresponding to that choice is returned. If multiple choice texts appear, the earliest one wins. This step recovers the common case where a small model answers with the choice's text instead of its letter — "The capital of France is Paris." returns B.
5. Any standalone letter. If the reply contains any standalone A–D as a single word (surrounded by word boundaries), that letter is returned. Last-resort word match.
6. First-character fallback. If none of the above matched, but the first non-whitespace character of the reply is one of A–D (case-insensitive), that letter is returned.
7. Otherwise: parse error. The item is counted as incorrect, but is also tallied separately in the per-model parse_errors field so format-following failures can be distinguished from knowledge failures.

The ordering is significant. Strong-format patterns (parenthesized, "Answer is", reply-starts-with) win over choice-text matching, because a model that explicitly outputs "A) Berlin" deserves to be scored against A even if "Paris" also appears later in its reply. Choice-text matching is necessary because base and small chat-trained models frequently answer in prose ("The capital of France is Paris.") without ever emitting a letter; without choice-text fallback those models score near chance regardless of underlying knowledge.

The full implementation lives in extract_letter() in run_featherweight.py and is <40 lines of Python with no dependencies beyond re.

What 100 questions can and cannot tell you

Random baseline

A model that answers each item with a letter drawn uniformly at random scores 25% in expectation. A model that always answers a single letter (e.g. always C) also scores 25%, because answer positions are exactly balanced — 25 of the 100 items have answer A, 25 have B, 25 have C, 25 have D. No position-bias-only strategy can exceed chance.

Confidence intervals

With n = 100, the standard error on an individual model's score is roughly sqrt(p(1-p)/100), which is 5pp at p = 50% and 4.5pp at p = 30%. The 95% confidence interval on a single score is therefore approximately ±9 percentage points in the relevant range. A 7pp gap between two models is at the edge of statistical significance; a 13pp gap is well clear of it.

In practical terms: claims of the form "H2 beats every model up to 410M by 13 or more points" are defensible. Claims of the form "Qwen 2.5 0.5B beats TinyLlama 1.1B by 10 points" (70% vs 60%) are at the edge — the gap is plausible but should not be over-stated without replication.

Resolution

Per-category resolution is 10 percentage points (one question = 10pp), so per-category accuracy is informative for spotting clear strengths or weaknesses but should not be over-interpreted. A model that scores 70% on geography and 60% on history is not reliably "better at geography" — that's a 1-question difference.

Position-bias check

The answer-position distribution is verifiable by running a one-line script against the JSONL file. The build script that generates Featherweight v1 contains an assertion that fails if the distribution is ever not 25/25/25/25:

assert answer_counts == {"A": 25, "B": 25, "C": 25, "D": 25}

How the 100 questions were chosen

All 100 questions were hand-written by the benchmark author for the v1 release. No questions were sourced from existing benchmarks (MMLU, ARC, HellaSwag, BIG-bench, etc.), from scraped web data, or from any source that might appear in a model's training corpus in identifiable form. The 10 category quotas were set in advance and questions written to fill them.

For each category, questions were drafted to satisfy four criteria:

• Single unambiguous answer. Each item has exactly one defensible correct answer that an educated human would agree with. Items where the "best" answer depends on convention (e.g. number of continents, longest river by definition) were avoided where possible and noted in the question if unavoidable.
• Plausible but distinguishable distractors. The three incorrect choices are written to be category-appropriate (e.g. for "capital of France", the distractors are other European capitals, not arbitrary words). A guessing model should be able to plausibly select any of the four; an informed model should be able to clearly identify the correct one.
• Roughly similar choice lengths. Choice texts within an item were written to be of comparable length and form. This is not a formally enforced constraint — we did not measure character-length variance per item — but it was a stylistic priority. (See §11 for the limitation.)
• Within reach of a 200M-parameter model. Questions involving multi-step reasoning, specialised expertise, or post-2010 events were deprioritised; the benchmark targets capabilities a well-trained small model should plausibly have.

Answer position balancing

After drafting, the position of the correct answer in each item was assigned to enforce the 25/25/25/25 distribution. Within each category, the 10 questions are split exactly 2–3–2–3 or 3–2–3–2 across A/B/C/D positions. This is verified by an assertion in the build script and is checkable against the published JSONL.

Temporal ordering relative to model evaluation

The Featherweight v1 question set was finalized before any of the 12 v1 comparison models — including Haiku H2 — was scored against it. The benchmark file featherweight.jsonl was committed and frozen prior to the first comparison run. This eliminates the possibility of post-hoc question selection to favour any particular model. (See §12 for the conflict-of-interest disclosure and its mitigations.)

Five items from five categories

Representative items, drawn verbatim from featherweight.jsonl. The full set of 100 is published as a single JSONL file (see §13).

12 models, two prompts

Twelve open small language models were evaluated against Featherweight v1 on 2026-06-13 using the methodology specified above. Models range from distilgpt2 (82M, 2019) to TinyLlama 1.1B Chat (2024). The full bar chart is the canonical Featherweight v1 leaderboard.

Featherweight v1, 12 small language models, two prompt formats. Sorted by parameter count.

For Haiku H2's per-category profile and the discussion of what these numbers mean for individual models, see the H2 model card. The benchmark itself is model-agnostic; future submissions from other model developers are welcome (see §13 for the runner).

Where Featherweight sits

Featherweight is not a replacement for established multi-domain benchmarks. It occupies a specific niche: multi-domain multi-choice evaluation at a difficulty level appropriate to sub-1B parameter models. Where existing benchmarks fit:

Featherweight does not claim to be a replacement for MMLU at scale. It claims to be a more informative benchmark in the parameter range where MMLU is largely uninformative. A small model that scores 60% on Featherweight tells you more than the same model scoring 25% on MMLU.

Honest caveats

Featherweight v1 has known limitations that any user of the benchmark should be aware of before drawing conclusions from a score.

• Sample size. n = 100 gives ~±9pp confidence intervals on individual scores. Differences smaller than ~10 points should not be over-interpreted. We plan a Featherweight v2 with n = 300–500 to narrow these intervals.
• Choice-length parity not formally enforced. Within each item, distractor lengths were chosen to be visually comparable to the correct answer, but no automated length-balancing pass was applied. A length-heuristic-only model might score above 25% on some items. We don't believe this is a major effect at this scale but cannot rule it out.
• English language only. All questions, choices, and instructions are in English. Models trained primarily on non-English corpora will be disadvantaged on this benchmark for reasons unrelated to capability.
• Anglophone / Western knowledge bias. The geography, history, and vocabulary categories lean toward facts an English-language Western-educated reader would consider general knowledge. This is appropriate for the typical training corpus of small open models, but is a real bias.
• No specialized domains. No medicine, no law, no programming, no advanced mathematics, no current events, no domain-specific terminology. Featherweight measures broad common knowledge and basic reasoning, not professional expertise.
• No multi-step reasoning beyond single inference. Logic items are one-step (modus ponens, transitivity). Multi-hop reasoning, chain-of-thought, and arithmetic word problems are out of scope.
• Single-pass scoring. Greedy decoding gives one shot per question. We don't measure self-consistency, majority-voting, or pass@k.
• No preregistered hypotheses. Featherweight is a measurement instrument, not a hypothesis-test framework. Claims about specific models should be replicated against future versions before being treated as settled.
• New and not battle-tested. Featherweight v1 was released in 2026. Established benchmarks have been stress-tested by hundreds of independent runs; this one has not. Treat its rankings as provisional until they're independently replicated.

Disclosure

How to reproduce these numbers

All data and code are open. Reproducing the v1 comparison table for the open-weights models requires a CUDA GPU with at least 8GB of VRAM (the largest model, TinyLlama 1.1B in float16, uses ~2.2GB), Python 3.9 or newer, and a one-line dependency install.

pip install transformers torch
curl -O https://rootcomputer.dev/source/downloads/featherweight.jsonl
curl -O https://rootcomputer.dev/source/downloads/run_featherweight.py
curl -O https://rootcomputer.dev/source/downloads/models.example.json
cp models.example.json models.json

python3 run_featherweight.py --prompt-style plain
python3 run_featherweight.py --prompt-style completion

Each run produces a timestamped directory under results/ containing one JSON file per model (with full per-question detail), a summary.csv for spreadsheet analysis, and a summary.json for programmatic consumption. Total runtime on a single RTX 3060 Ti is approximately 15–20 minutes per prompt style for the full 12-model lineup, dominated by HuggingFace model downloads on the first run.

The runner also supports OpenAI-compatible chat-completions endpoints (vLLM, ollama, LM Studio) and arbitrary chathaiku-shape HTTP endpoints. To benchmark a model not in the default models.json, add an entry with the appropriate backend; documentation is in the runner's README.

JSONL file format

featherweight.jsonl is one JSON object per line. The first line is an optional metadata header. Each subsequent line is a single question record:

{"_meta": true, "name": "Featherweight v1", "version": "1.0", ...}
{"id": "geo01", "category": "geography",
 "question": "What is the capital of France?",
 "choices": ["Paris", "Berlin", "Madrid", "Rome"],
 "answer": "A"}
{"id": "geo02", ...}
...

The format is intentionally minimal so it can be consumed by any evaluation runner with about 20 lines of glue code. The answer field accepts either a letter ("A") or a zero-indexed integer (0); both are normalized identically.

Releases

• v1.0 — 2026-06-13. Initial release. 100 questions across 10 categories. Twelve models scored.

Substantive changes to the question set will be released as new major versions (v2, v3, ...) with their own JSONL files. Typo fixes and metadata corrections that do not change the meaning or correct answer of any question may be released as minor versions (v1.0.1, v1.0.2) with a changelog. Any change that could affect a previously published score will require a major version bump.

Use and attribution

Featherweight v1 is released under the Creative Commons Attribution 4.0 International license (CC-BY-4.0). You may share and adapt the benchmark, including for commercial use, provided you give appropriate credit, provide a link to the license, and indicate if changes were made.

Suggested citation:

Issues, corrections, and proposed v2 additions can be sent via the contact link on rootcomputer.dev/business.