AI/ML●Mid
CivBench a long-horizon AI benchmark for multi-agent games
Civilization matches expose model divergence that static benchmarks miss—but it's a spectacle, not a measurement.
Rabbit HoleBig Brain
mbh159
12243mo ago
Back to browse
Benchmarking Tangible Interface Understanding in Long-Horizon Tasks
by tellarin·Apr 29, 2026·1 point·1 comment
AI Analysis
●●●BangerBig BrainNiche Gem
First benchmark testing if AI agents can actually flip light switches and read appliance panels.
Strengths
- •Tests causal prediction and delayed outcome verification, not just single-step recognition.
- •351 tasks across 98 real devices expose gaps masked by aggregate scores.
- •Releases held-out splits and code for reproducible community evaluation.
Weaknesses
- •Niche academic benchmark with limited immediate utility for production robotics.
- •Commercial LMMs already fail basic steps, so the bar is currently very low.
Category
Target Audience
Robotics researchers and embodied AI developers
Similar To
BIG-bench · AgentBench · VLA benchmarks
Similar Projects
Developer Tools●●Solid
Lazarus, a coding agent for long-horizon tasks
Persistent Python runtime keeps state alive across tool calls, unlike Claude Code's stateless tools.
Big BrainNiche Gem
Sai_Praneeth
1014d ago
AI/ML●●●Banger
A deterministic ecosystem simulator for long-horizon AI agents
Deterministic multi-agent evolutionary benchmark with SHA-256 reproducible capsules for agent testing.
Big BrainWizardryZero to One
yangkecoy
103mo ago
AI/ML●●Solid
Tiny long-memory benchmark with Harbor running across Islo sandboxes
Compresses long-memory evaluation into three questions testing recall, updates, and abstention.
Niche GemBig Brain
zozo123-IB
201mo ago
AI/ML●●Solid
Agentic Intent Benchmark
First benchmark testing structured requirements on complex greenfield agent tasks.
Niche GemBig Brain
ryan4rtmx
2022d ago
Developer Tools●Mid
OpenCode Benchmark Dashboard
Benchmarks OpenCode models locally, but lacks preloaded datasets and only works with configured OpenAI-compatible APIs.
Niche Gem
grigio
103mo ago