Caliper – Auto Instrumented LLM Observability with Custom Metadata
Zero-code instrumentation via monkey-patching, but Langsmith, Helicone, and Arize already do this.
See every LLM call, tool use, and token spent — locally, with one line of code. No cloud. No account. No config.
One-line init with monkey-patching means zero changes to existing agent code.
Python developers building AI agents who need local debugging
LangSmith · Arize Phoenix · Weights & Biases
Zero-code instrumentation via monkey-patching, but Langsmith, Helicone, and Arize already do this.
One-line LLM cost tracking via monkey-patching; integrates without touching existing code.
Real-time dollar limits on AI agents, monkey-patched into OpenAI/Anthropic SDKs.
Turns existing LangChain pipelines into first-class batch jobs you can submit to provider batch endpoints without rewriting your chains. It automates JSONL uploads, polling (BatchPoller), provider-specific parsing into unified BatchItem results, partial-failure handling, and on-disk job persistence so batches can outlive your process — Vertex/Azure support is on the roadmap.
Auto-clusters failure patterns across sessions and suggests prompt patches.
LiteLLM team solving agent SDK fragmentation with one query() function.