Active
llm-bench
Leaderboards were not answering my questions, so I made tests that did.
The problem
Leaderboards don't answer the only question that matters: will this model hold up on your actual work?
What I built
Real workflow tasks: extraction, code, planted bugs, email drafting, prompt injection, each graded by a programmatic verifier. Works with Ollama, Apple Intelligence, Claude CLI, Bedrock, any OpenAI-compatible endpoint.
Proof
21 tests, programmatic verifiers, published model comparisons including Opus 4.8 vs 4.7 vs Sonnet vs Haiku. 148 commits.
What I learned
A benchmark only matters if the measuring stick is explicit enough to argue with. Programmatic verifiers force an honesty that LLM-as-judge lets you skip.
Designing evaluation systems, the exact skill client evals and monitoring work needs.
Stack
Python · Ollama · Bedrock · Claude CLI
- status
- Active · Practical workflow benchmark
- stack
- Python · Ollama · Bedrock · Claude CLI
- scope
- Standard · Hard · Agentic · Adversarial · Messy
- license
- MIT
Links
A benchmark only matters if the measuring stick is explicit enough to argue with.