Evaluate declared scope
Compare tool calls and resources with the task’s exact allowlists, read-only constraints, and action-specific approvals.
OpenCaliber helps developers evaluate, diagnose and improve AI agent reliability. Connect an existing Python agent through its startup command. Review supported tool activity in your console, diagnose problems and compare versions when task evidence is available.
Python 3.10+ · Windows, macOS & Linux · No API key for built-in checks.Follow a Python agent from unchanged source code to a connected console run. See the startup command, current dashboard, findings and monitoring coverage, with male narration and English captions.
Use examples/observe_agent/agent.py to follow along. The walkthrough explains observation limits and when to add explicit tool integration. English captions are available in the player.
Use the startup launcher for supported Python calls, or explicit tool integration for richer evidence and blocking. Findings link a policy rule to an agent, run and action ID.
Compare tool calls and resources with the task’s exact allowlists, read-only constraints, and action-specific approvals.
Check outbound hosts, sensitive transfers, file roots, dangerous command patterns, and record budgets across a trace.
Every finding identifies a rule and the action IDs behind it. Review the original trace alongside its configuration snapshot.
Run both versions on identical inputs. Compare task success, policy findings, invalid steps, recovery, reported cost and measured latency. Missing evidence stays unknown.
Persist evaluations, capture review notes, mark runs reviewed or dismissed, and export a full JSON evidence bundle.
Use Python or typed Node helpers with completed OpenAI or Anthropic tool calls, or submit JSON through HTTP or CLI. The evaluation engine does not execute tools; the application owns callbacks.
OpenCaliber uses version-comparison evidence to propose additional tests and improvements. Adaptive means improving the review loop—not silently rewriting policies or modifying deployed agents.
Give two application-owned agent callbacks isolated copies of the same inputs. Use explicit success assertions and retain the policy fingerprint.
Inspect task regressions, the first invalid tool call, recovery and measurement gaps. A permitted tool call can still lead to an unsuccessful task.
Export proposed boundary tests, failure-path tests and improvements. Developers decide what to change, rerun and promote. Initial proposals are deterministic, not trained-model judgments.
Start with your existing Python script. No edits to individual tool functions.
Download and extract the package. Run python run.py, then open http://127.0.0.1:8787/console. Keep it running.
From an environment where OpenCaliber is importable, replace python my_agent.py with the command alongside. Your agent source stays unchanged. Each invocation creates a run under its agent ID.
Open Agents Dashboard, search for your agent and select Inspect run or Detected problems. Supported calls are evaluated against console policies after execution. Monitoring coverage shows missing evidence.
Correct the agent or its authorized configuration, rerun the same command and inspect the new run. Restore the original startup command to remove monitoring.
This example uses SQLite and needs no provider credentials.
# Terminal 1: console package folder
python run.py
# Terminal 2: same folder
python -m opencaliber.connect --agent-id sqlite-demo -- examples/observe_agent/agent.pyFor your own agent, use --agent-id my-agent -- my_agent.py.
Default SQLite connections and urllib HTTP calls; synchronous Requests and HTTPX calls when installed. No SQL, rows, HTTP bodies, headers or URL paths are captured. Maximum 200 actions per invocation. Async clients, custom SQLite factories, files and child processes are outside coverage. Provider SDK traffic through supported clients appears as HTTP activity.
The launcher observes; it never blocks, even under a blocking console policy. Business success stays unknown. Use explicit tool integration to add enforcement, selected output evidence, success assertions and linked multi-agent graphs. With shared test inputs and success criteria, Compare agent versions shows which version performed better.
OpenCaliber evaluates captured action metadata against configured checks. The launcher observes supported calls after execution. The optional execution wrapper adds explicit outcomes and pre-dispatch blocking. Neither path undoes actions or proves complete coverage.
Available capabilities and the next connection paths under consideration. Planned items are not included in the current runtime.
Deterministic rules, explicit task scope, trace import, evidence-based findings, review notes, configuration snapshots and 70 labeled contract cases.
Use paired version results to propose tests and improvements for developer review. Surface task regressions, policy findings and missing measurements.
Add independently authored scenarios, held-out tools and workflows, multi-step cases and a context-aware model comparison against simpler baselines.
The Python startup launcher, explicit tool integration and version comparisons are available. Planned work adds existing-telemetry import, MCP gateway integration and broader framework coverage.
Explore self-hosted collaboration with verified identities, scoped access, retention controls and an accountable review history.
The Python execution wrapper supports explicit observe and enforce modes. Planned work adds coding-agent connectors and stronger workflow coordination.
Yes, within the connection coverage. The launcher observes supported synchronous HTTP and SQLite calls regardless of which model your agent uses. It does not capture every SDK, async client or model response. Use Connect an Agent for the launcher and explicit integration options; use selected outputs and success assertions when business-level diagnosis is needed.
It is the maximum triggered rule severity on a 0–100 scale. It is not a calibrated probability of harm. Review the triggered rules and evidence rather than treating the number as certainty.
The downloaded console binds to loopback and has no login. Hosted deployments protect a shared workspace with authentication and HTTPS, but have no per-user roles or tenant isolation. SDK evidence can be redacted before storage. Review the deployment and privacy documentation before using sensitive traces.
No. These are hand-authored synthetic contract checks for implemented rules. They are not an independent security benchmark, and they do not cover arbitrary attacks, model behavior or hidden history.
The included deterministic engine requires only Python’s standard library and makes no paid API calls. There is no subscription or billing flow in this preview.
Download the ZIP, extract it, and run the included launcher. Connect a library agent or start an existing Python script through the observation launcher.
python run.pyNeed an integration example? Open the Developer Guide for the launcher, tool integration and coverage limits.
Causal Inference & AI Research
I created OpenCaliber as an open-source project to learn and collaborate with developers on evaluating AI agent actions and inspecting policy findings.
Connect on LinkedIn ↗