263k config search space benchmarked across robot fleets—nothing like this exists for robotics AI.
Robotics engineers, AI agent developers, edge computing teams
MLPerf · Robotics Open Benchmarks · Hugging Face Open LLM Leaderboard
So I built a distributed evaluator. Robots contribute idle compute to benchmark harness configurations against OHB-1, a small benchmark of 30 real-world robot tasks (grip, navigate, respond, etc.) using local LLM calls via Ollama. The search space is 263,424 configs (8 dimensions: model routing, context budget, retry logic, drift detection, etc.). The demo leaderboard shows results so far, broken down by hardware tier (Pi5+Hailo, Jetson, server, budget boards).
The current champion config is free to download as a YAML and apply to any robot. P66 safety parameters are stripped on apply — no harness config can touch motor limits or ESTOP logic.
Looking for feedback on: (1) whether the benchmark tasks are representative, (2) whether the hardware tier breakdown is useful, and (3) anyone who's run fleet-wide distributed evals of agent configs for robotics or otherwise.
Deterministic agent benchmarking with strict validation—unlike SWE-Bench, measures whether agents actually operate.
Manifest-driven agents with eval feedback loops when most harnesses are prompt-only.
Unsupervised bug benchmark using agents as both attackers and defenders—novel scoring methodology.
Interactive DuckDB-WASM benchmark beats static leaderboards for agentic SQL eval.
90.3 BrowseComp score with verification-centric model architecture.
You can watch an LLM play NetHack step-by-step with the model's reasoning, the exact action code, and a live game canvas — that instrumentation is the product's real selling point. The leaderboard + run/benchmark framing makes it useful for comparing agents rather than just a flashy demo, but it's still squarely for people who care about NetHack or agent evaluation; more detail on reproducible metrics and integrations would push it further.
