Benchmarks

Open evaluation suites from Rootcomputer for small and mid-size language models. All benchmarks here are published with full data, code, and methodology under permissive licenses — designed to be reproducible by anyone with a commodity GPU and a Python install.

Principles

What a Rootcomputer benchmark commits to

Every benchmark in this directory is published against the same set of commitments. These are not aspirations — they are constraints applied to the design before any model is scored.

Open data, open code
The full question set and the runner that produces the scores are published under a permissive license. Anyone can reproduce a leaderboard from scratch.
Position bias controlled
Correct-answer positions are balanced exactly across choices. A model that always picks one letter scores chance, not higher.
Multiple prompt formats
Every benchmark is run under at least two prompt formats. Both columns are reported so neither instruction-tuned nor base models are privileged.
Deterministic scoring
Greedy decoding only. The same model on the same prompt produces the same output every run. No sampling noise in published scores.
Conflict of interest disclosed
Rootcomputer also develops the Haiku family of models. Every benchmark page that includes a Rootcomputer model has an explicit COI section and its mitigations.
Honest limitations
Sample size, language coverage, cultural bias, and methodological gaps are listed before results, not buried in an appendix.

Released

Available now

Benchmarks that are frozen, documented, and ready to run against any model.

Released
Multi-choice · Sub-1B parameter class · 10 domains

A 100-question multi-choice benchmark calibrated for sub-1B-parameter language models. Ten categories of ten questions each across geography, history, arithmetic, science, language, vocabulary, common sense, logic, categorization, and reading comprehension. Two prompt formats are reported. Designed to give small models a meaningful place to compete — informative scores above the 25% chance floor without requiring multi-billion-parameter knowledge.

100
Questions
10
Categories
2
Prompt formats
12
Models scored
Released 2026-06-13 · CC-BY-4.0 Read documentation →
Released
Multi-choice · Sub-1B parameter class · 10 domains · Human baseline

The next-generation Featherweight benchmark. 300 hand-written multi-choice questions across the same 10 categories as v1, drawn from a disjoint question set. Narrows the 95% individual-score confidence interval from ±9pp to roughly ±5.7pp. First Featherweight release with a preliminary human-baseline anchor (~86.8% on paper). An online version is available for self-testing.

300
Questions
10
Categories
~±5.7pp
95% CI
~86.8%
Human baseline
Released 2026-06-25 · CC-BY-4.0 · Take the online test Read documentation →

In development

Coming next

No benchmarks currently in active development. Featherweight v2 graduated to the Released section above. The next planned benchmark direction is sketched in the roadmap section below.

Roadmap

Beyond Featherweight

The Featherweight series targets sub-1B-parameter models. Future Rootcomputer benchmarks will extend to adjacent weight classes and adjacent task formats as the model lineup grows. Candidate directions, in rough order of likelihood:

  • A lightweight reasoning benchmark. Single-domain, multi-step inference tasks for small models — word problems, syllogism chains, transitive ordering — with the same open-data and deterministic-scoring commitments.
  • A robustness suite. Same questions, multiple paraphrases, multiple prompt formats. Measures how much a model's score depends on phrasing — the metric Featherweight v1 surfaced as Haiku H2's biggest strength.
  • A middleweight class benchmark. If Rootcomputer releases a 1–3B-parameter model in the future, that model needs a benchmark calibrated to its capabilities rather than Featherweight's. The naming convention will continue the boxing-weight metaphor.

None of these are committed timelines. They are the natural extensions of the Featherweight design philosophy, and any of them — or none of them — may exist by the time Rootcomputer's next benchmark is released.

Contributing

Submitting a model, reporting an issue

Each benchmark is shipped with a standalone runner that supports HuggingFace local models, OpenAI-compatible HTTP endpoints, and arbitrary chat-shape HTTP endpoints. To benchmark a model not in a published comparison table, run the appropriate suite locally and the results will be directly comparable to the published numbers (modulo network noise for hosted endpoints).

Corrections to questions, methodology critiques, and proposals for new benchmark domains can be sent via the contact link on rootcomputer.dev/business. Substantive corrections to a released benchmark will be released as a minor version with a public changelog; changes that affect any previously published score require a major version bump.