Build more reliable
AI agents.

OpenCaliber helps developers evaluate, diagnose and improve AI agent reliability. Connect an existing Python agent through its startup command. Review supported tool activity in your console, diagnose problems and compare versions when task evidence is available.

Python 3.10+ · Windows, macOS & Linux · No API key for built-in checks.
CONNECTED AGENTSAMPLE WORKFLOW
Launch observe inspect and improve an existing agentStart an existing Python agent through the OpenCaliber launcher. A supported HTTP call is observed. Its hostname is outside the configured allowlist, so the console reports a finding after execution. Correct the destination and rerun.01 · Launch your existing agentChange startup, keep tool code02 · Observe supported callsHTTP and SQLite activity reaches console03 · Review the destination findingHost is outside your configured allowlist04 · Fix and run againConfirm the corrected host is allowed
Next step: Correct the destination, rerun and inspect the new run.
Example: observed HTTP metadata checked after execution.
70 included rule tests · add your ownOptional coding modelsJSON structured evaluationsSQLite evaluation history
YOUR FIRST EVALUATION

Download, run and use the console

Follow a Python agent from unchanged source code to a connected console run. See the startup command, current dashboard, findings and monitoring coverage, with male narration and English captions.

Use examples/observe_agent/agent.py to follow along. The walkthrough explains observation limits and when to add explicit tool integration. English captions are available in the player.

IMPLEMENTED CHECKS

Checks and review tools

Use the startup launcher for supported Python calls, or explicit tool integration for richer evidence and blocking. Findings link a policy rule to an agent, run and action ID.

⌘

Evaluate declared scope

Compare tool calls and resources with the task’s exact allowlists, read-only constraints, and action-specific approvals.

◇

Inspect risk boundaries

Check outbound hosts, sensitive transfers, file roots, dangerous command patterns, and record budgets across a trace.

◉

Inspect findings

Every finding identifies a rule and the action IDs behind it. Review the original trace alongside its configuration snapshot.

⊞

Compare agent versions

Run both versions on identical inputs. Compare task success, policy findings, invalid steps, recovery, reported cost and measured latency. Missing evidence stays unknown.

▦

Keep a review history

Persist evaluations, capture review notes, mark runs reviewed or dismissed, and export a full JSON evidence bundle.

▤

Console, Python, HTTP and CLI

Use Python or typed Node helpers with completed OpenAI or Anthropic tool calls, or submit JSON through HTTP or CLI. The evaluation engine does not execute tools; the application owns callbacks.

IMPROVEMENT WITH DEVELOPER REVIEW

Learn what to test next.

OpenCaliber uses version-comparison evidence to propose additional tests and improvements. Adaptive means improving the review loop—not silently rewriting policies or modifying deployed agents.

Compare the same cases

Give two application-owned agent callbacks isolated copies of the same inputs. Use explicit success assertions and retain the policy fingerprint.

Diagnose the difference

Inspect task regressions, the first invalid tool call, recovery and measurement gaps. A permitted tool call can still lead to an unsuccessful task.

Review the next experiment

Export proposed boundary tests, failure-path tests and improvements. Developers decide what to change, rerun and promote. Initial proposals are deterministic, not trained-model judgments.

Set up version comparisons →

CONNECT. OBSERVE. IMPROVE.

Connect once. Monitor each run.

Start with your existing Python script. No edits to individual tool functions.

01 /

Start your console

Download and extract the package. Run python run.py, then open http://127.0.0.1:8787/console. Keep it running.

02 /

Launch your agent

From an environment where OpenCaliber is importable, replace python my_agent.py with the command alongside. Your agent source stays unchanged. Each invocation creates a run under its agent ID.

03 /

Find the call that needs attention

Open Agents Dashboard, search for your agent and select Inspect run or Detected problems. Supported calls are evaluated against console policies after execution. Monitoring coverage shows missing evidence.

04 /

Fix, rerun and review

Correct the agent or its authorized configuration, rerun the same command and inspect the new run. Restore the original startup command to remove monitoring.

Try the included agent first

This example uses SQLite and needs no provider credentials.

# Terminal 1: console package folder
python run.py

# Terminal 2: same folder
python -m opencaliber.connect --agent-id sqlite-demo -- examples/observe_agent/agent.py

For your own agent, use --agent-id my-agent -- my_agent.py.

What gets monitored?

Default SQLite connections and urllib HTTP calls; synchronous Requests and HTTPX calls when installed. No SQL, rows, HTTP bodies, headers or URL paths are captured. Maximum 200 actions per invocation. Async clients, custom SQLite factories, files and child processes are outside coverage. Provider SDK traffic through supported clients appears as HTTP activity.

Need blocking, task success or version comparisons?

The launcher observes; it never blocks, even under a blocking console policy. Business success stays unknown. Use explicit tool integration to add enforcement, selected output evidence, success assertions and linked multi-agent graphs. With shared test inputs and success criteria, Compare agent versions shows which version performed better.

Connection examples and coverage →

CAPABILITIES AND LIMITS

Built to help you evaluate.
Not to promise perfect safety.

OpenCaliber evaluates captured action metadata against configured checks. The launcher observes supported calls after execution. The optional execution wrapper adds explicit outcomes and pre-dispatch blocking. Neither path undoes actions or proves complete coverage.

What works today

  • Deterministic evaluation of normalized action traces.
  • Console UI, Python API, CLI and SQLite-backed history.
  • Task scope, resource and transfer checks.
  • Configurable limits and a labeled regression suite.
  • Launcher-based observation without tool-function edits; optional wrapper integration for enforcement and richer evidence.

What remains outside coverage

  • Infer arbitrary intent with a trained LLM.
  • Authenticate supplied scope or approval statements.
  • Observe every library, hidden action or child process automatically.
  • Execute containment or guarantee threat prevention.
  • Provide per-user roles or tenant isolation; hosted authentication protects one shared workspace.
WHERE WE GO NEXT

Development roadmap

Available capabilities and the next connection paths under consideration. Planned items are not included in the current runtime.

01 / FOUNDATIONAvailable today

Trace evaluations and findings

Deterministic rules, explicit task scope, trace import, evidence-based findings, review notes, configuration snapshots and 70 labeled contract cases.

Current boundary: supported launcher calls or explicitly reported actions. Context diagnosis needs selected evidence and optional model configuration; it does not infer hidden work.
02 / ADAPTABILITYAvailable today

Review-driven improvement proposals

Use paired version results to propose tests and improvements for developer review. Surface task regressions, policy findings and missing measurements.

Current boundary: deterministic, evidence-driven proposals. No automatic policy changes, code modification or deployment.
03 / RESEARCHPlanned

Independent evaluation & context models

Add independently authored scenarios, held-out tools and workflows, multi-step cases and a context-aware model comparison against simpler baselines.

Release gate: publish misses, false alerts, calibration, abstention, latency and data provenance. Synthetic contract results alone are insufficient.
04 / WORKFLOWPlanned

More connection paths

The Python startup launcher, explicit tool integration and version comparisons are available. Planned work adds existing-telemetry import, MCP gateway integration and broader framework coverage.

Release gate: tested schema mappings, redaction controls and documented integration coverage. No claim of automatic universal agent discovery.
05 / COLLABORATIONExploratory

Private team workspaces

Explore self-hosted collaboration with verified identities, scoped access, retention controls and an accountable review history.

Release gate: authentication, authorization, tenant isolation, encryption and operational testing before accepting company traces.
06 / CONTROLExploratory

Opt-in execution boundaries

The Python execution wrapper supports explicit observe and enforce modes. Planned work adds coding-agent connectors and stronger workflow coordination.

Release gate: verified authority, non-bypassable integration tests and explicit operator control. Never a promise to prevent every threat.
No release dates are set.Planned features require the listed tests before release. Current results do not establish model quality or enterprise readiness.Read the developer guide →
BEFORE YOUR FIRST RUN

Frequently asked questions

Does it work with Gemini, Claude, or other agents?

Yes, within the connection coverage. The launcher observes supported synchronous HTTP and SQLite calls regardless of which model your agent uses. It does not capture every SDK, async client or model response. Use Connect an Agent for the launcher and explicit integration options; use selected outputs and success assertions when business-level diagnosis is needed.

What does a risk score mean?

It is the maximum triggered rule severity on a 0–100 scale. It is not a calibrated probability of harm. Review the triggered rules and evidence rather than treating the number as certainty.

Can I use it with a company’s sensitive traces?

The downloaded console binds to loopback and has no login. Hosted deployments protect a shared workspace with authentication and HTTPS, but have no per-user roles or tenant isolation. SDK evidence can be redacted before storage. Review the deployment and privacy documentation before using sensitive traces.

Does passing the test suite mean my agent is secure?

No. These are hand-authored synthetic contract checks for implemented rules. They are not an independent security benchmark, and they do not cover arbitrary attacks, model behavior or hidden history.

What does it cost?

The included deterministic engine requires only Python’s standard library and makes no paid API calls. There is no subscription or billing flow in this preview.

INSTALLATION

Run the evaluation console

Download the ZIP, extract it, and run the included launcher. Connect a library agent or start an existing Python script through the observation launcher.

python run.py
↓ Download ZIP Console setup ↗
Need an integration example? Open the Developer Guide for the launcher, tool integration and coverage limits.
Arun Kumar, creator and research lead of OpenCaliber
CREATOR & RESEARCH LEAD

Arun Kumar

Causal Inference & AI Research

I created OpenCaliber as an open-source project to learn and collaborate with developers on evaluating AI agent actions and inspecting policy findings.

Connect on LinkedIn ↗