Benchmarks
Open evaluation suites from Rootcomputer for small and mid-size language models. All benchmarks here are published with full data, code, and methodology under permissive licenses — designed to be reproducible by anyone with a commodity GPU and a Python install.
Principles
What a Rootcomputer benchmark commits to
Every benchmark in this directory is published against the same set of commitments. These are not aspirations — they are constraints applied to the design before any model is scored.
Released
Available now
Benchmarks that are frozen, documented, and ready to run against any model.
A 100-question multi-choice benchmark calibrated for sub-1B-parameter language models. Ten categories of ten questions each across geography, history, arithmetic, science, language, vocabulary, common sense, logic, categorization, and reading comprehension. Two prompt formats are reported. Designed to give small models a meaningful place to compete — informative scores above the 25% chance floor without requiring multi-billion-parameter knowledge.
The next-generation Featherweight benchmark. 300 hand-written multi-choice questions across the same 10 categories as v1, drawn from a disjoint question set. Narrows the 95% individual-score confidence interval from ±9pp to roughly ±5.7pp. First Featherweight release with a preliminary human-baseline anchor (~86.8% on paper). An online version is available for self-testing.
In development
Coming next
No benchmarks currently in active development. Featherweight v2 graduated to the Released section above. The next planned benchmark direction is sketched in the roadmap section below.
Roadmap
Beyond Featherweight
The Featherweight series targets sub-1B-parameter models. Future Rootcomputer benchmarks will extend to adjacent weight classes and adjacent task formats as the model lineup grows. Candidate directions, in rough order of likelihood:
- A lightweight reasoning benchmark. Single-domain, multi-step inference tasks for small models — word problems, syllogism chains, transitive ordering — with the same open-data and deterministic-scoring commitments.
- A robustness suite. Same questions, multiple paraphrases, multiple prompt formats. Measures how much a model's score depends on phrasing — the metric Featherweight v1 surfaced as Haiku H2's biggest strength.
- A middleweight class benchmark. If Rootcomputer releases a 1–3B-parameter model in the future, that model needs a benchmark calibrated to its capabilities rather than Featherweight's. The naming convention will continue the boxing-weight metaphor.
None of these are committed timelines. They are the natural extensions of the Featherweight design philosophy, and any of them — or none of them — may exist by the time Rootcomputer's next benchmark is released.
Contributing
Submitting a model, reporting an issue
Each benchmark is shipped with a standalone runner that supports HuggingFace local models, OpenAI-compatible HTTP endpoints, and arbitrary chat-shape HTTP endpoints. To benchmark a model not in a published comparison table, run the appropriate suite locally and the results will be directly comparable to the published numbers (modulo network noise for hosted endpoints).
Corrections to questions, methodology critiques, and proposals for new benchmark domains can be sent via the contact link on rootcomputer.dev/business. Substantive corrections to a released benchmark will be released as a minor version with a public changelog; changes that affect any previously published score require a major version bump.