Developer Tools●●●Banger
Cheddar-bench – unsupervised benchmark for coding agents
Unsupervised bug benchmark using agents as both attackers and defenders—novel scoring methodology.
Big BrainWizardryShip It
przadka
904mo ago
Back to browse
New Benchmark from SWE-bench team is 0% solved
by lieret·May 5, 2026·24 points·3 comments
AI Analysis
●●●●GemBig BrainBold BetZero to One
Agents fail completely at rebuilding binaries from scratch without source code.
Strengths
- •248,000 behavioral tests generated via fuzzing ensure rigorous evaluation.
- •Tasks range from simple utilities like jq to massive projects like FFmpeg.
- •Zero percent solve rate exposes the gap between coding and engineering.
Weaknesses
- •Extremely niche audience limited to agent infrastructure researchers.
- •Binary-only constraint may not reflect real-world engineering scenarios.
Category
Target Audience
AI researchers, LLM developers, Benchmark enthusiasts
Similar To
SWE-bench · HumanEval · LiveCodeBench
Similar Projects
AI/ML●●●Banger
RewardHackBench: Using sandboxes to stop agents from cheating
LLM judge on outgoing requests achieves 0% cheat rate while preserving 58% fair-solve ceiling.
Big BrainDark Horse
rotemtam
935d ago
AI/ML●●●Banger
Pencil Puzzle Bench – LLM Benchmark for Multi-Step Verifiable Reasoning
62k puzzle benchmark reveals reasoning depth, cost variance, and stark US vs China model gaps.
Big BrainCrowd PleaserSolve My Problem
bluecoconut
503mo ago
AI/ML●●●Banger
97% on SWE-bench Verified with subscription-token agents
97% on SWE-bench Verified with full artifact transparency, not just a score claim.
Big BrainZero to One
kimjune01
2028d ago
AI/ML●●●Banger
Rogue-Bench – LLMs play the game Rogue
Using 1980s Rogue as an LLM benchmark is genuinely novel and technically clever.
WizardryZero to One
iwhalen
1027d ago
AI/ML●●●Banger
Τ³-Bench is out – can agents handle complex docs and live calls?
Tests agents on 700 policy docs and noisy voice calls where AgentBench stops.
Big BrainNiche Gem
victorbarres
1212mo ago