AI/ML●●Solid
I benchmarked how good LLMs are at proofreading English
Agent loop proofreading evals where HELM and LMSys are too generic.
Solve My ProblemShip It
artursapek
321mo ago
Back to browse
ErrataBench - A Proofreading Benchmark for LLMs
by artursapek·Apr 7, 2026·3 points·0 comments
AI Analysis
●●SolidNiche GemBig Brain
51 models, 1613 runs, $558 spent — finally proofreading benchmarks with real numbers.
Strengths
- •Distinguishes omissions from bad fixes, giving actionable failure mode breakdowns.
- •Five days of runtime with transparent methodology and cost tracking throughout.
- •Rankings include efficiency metrics, not just raw accuracy percentages.
Weaknesses
- •Narrow scope limits utility — proofreading is just one of many LLM use cases.
- •LLM benchmarks are common; LMSys and HELM already dominate the space.
Category
Target Audience
AI researchers, developers selecting LLMs for text tasks
Similar To
LMSys Chatbot Arena · HELM · LiveBench
Similar Projects
AI/ML●●Solid
Kitchen Rush, Overcooked inspired LLM tool calling benchmark
Thinking time becomes game time lost — finally measures latency alongside accuracy.
Big BrainNiche Gem
bombastic311
302d ago
Developer Tools●●●Banger
AgentDX – Open-source linter and LLM benchmark for MCP servers
First linter + benchmark for MCP servers; catches vague schemas before LLMs pick wrong tools.
Solve My ProblemNiche GemBig Brain
yamarldfst
104mo ago
AI/ML●●●Banger
LLM Sycophancy Benchmark: Opposite-Narrator Contradictions
Opposite-narrator test catches models agreeing with both sides of same dispute.
Big BrainDark Horse
zone411
303mo ago
AI/ML●●●Banger
Reducing LLM input tokens by 70%
Cuts token costs 70% with receipts proving no accuracy drop on hard evals.
Zero to OneSolve My Problem
Jbunga
56331mo ago
AI/ML●●Solid
InferBench – Benchmark local LLM engines with one click
One-click LLM benchmarking with real tok/s metrics when llama.cpp requires manual setup.
Ship ItSolve My Problem
JoniMartin
2013d ago