OpenCaliberEvaluate declared scope
Compare tool calls and resources with the task’s exact allowlists, read-only constraints, and action-specific approvals.
Evaluate AI agent
action traces.
Submit a JSON trace to check tool calls, resources, transfers and usage limits against configured rules. Review findings and save results on this machine.
Python 3.10+ · Windows, macOS & Linux · No API key required.INVOICE AGENTSAMPLE WORKFLOW
Fix: Remove the external send, then evaluate a new trace.
Evaluates submitted actions. Does not block or undo them.70 labeled contract cases0 external model callsJSON structured evaluationsSQLite evaluation history
YOUR FIRST EVALUATION
Download, run and use the console
Learn how to set up OpenCaliber, configure tool policies, connect agent workflows and interpret findings. Follow the narrated steps with English captions.
Follow along with the included examples, then configure the rules for your own agent workflow. English captions are available in the player.
IMPLEMENTED CHECKS
Checks and review tools
Supply a normalized trace with permitted tools, resources and sensitivity labels. The evaluator returns rule findings linked to action IDs.
Inspect risk boundaries
Check outbound hosts, sensitive transfers, file roots, dangerous command patterns, and record budgets across a trace.
Inspect findings
Every finding identifies a rule and the action IDs behind it. Review the original trace alongside its configuration snapshot.
Run regression cases
Run 70 labeled synthetic cases, compare expected and observed assessments, and see failures under your active configuration.
Keep a review history
Persist evaluations, capture review notes, mark runs reviewed or dismissed, and export a full JSON evidence bundle.
Console, Python, HTTP and CLI
Use Python or typed Node helpers with completed OpenAI or Anthropic tool calls, or submit JSON through HTTP or CLI. The engine does not execute tools.
ONE TRACE. THREE STEPS.
Submit a trace and inspect results
01 /
Declare the task
Supply the goal, permitted tools and resources. Missing context is surfaced for review.
02 /
Evaluate the proposed actions
Import a normalized JSON trace or call the evaluation API. The engine applies the configured rules.
03 /
Review and iterate
Inspect violations, uncertainty and evidence. Change your agent or configuration, then replay.
{
"assessment": "review",
"risk_score": 45,
"score_type": "heuristic_not_probability",
"findings": [{
"code": "MISSING_TASK_SCOPE",
"category": "uncertainty",
"evidence_ids": ["action-01"]
}]
}
// Illustrative action-level excerpt.
// Clear ≠ guaranteed safe.CAPABILITIES AND LIMITS
Built to help you evaluate.
Not to promise perfect safety.
OpenCaliber checks supplied traces with deterministic rules. It reports findings for review and does not execute or block agent actions.
What works today
- Deterministic evaluation of normalized action traces.
- Console UI, Python API, CLI and SQLite-backed history.
- Task scope, resource and transfer checks.
- Configurable limits and a labeled regression suite.
- No telemetry or external inference in this console.
What this preview does not do
- Infer arbitrary intent with a trained LLM.
- Authenticate supplied scope or approval statements.
- Automatically discover or intercept live agents.
- Execute containment or guarantee threat prevention.
- Provide multi-user authentication or tenant isolation.
WHERE WE GO NEXT
Development roadmap
The items below are planned or under consideration. They are not included in the current runtime.
01 / FOUNDATIONAvailable today
Trace evaluations and findings
Deterministic rules, explicit task scope, trace import, evidence-based findings, review notes, configuration snapshots and 70 labeled contract cases.
Current boundary: advisory rules; no semantic LLM assessment or live-agent interception.
02 / ADAPTABILITYPlanned · next focus
Optional validation planner
Use an agent profile and tool schemas to propose supported checks, configuration and test cases. Surface missing context and unsupported validations.
Release gate: schema-validated plans, operator approval, reproducible versions and prompt-injection tests. No automatic execution of generated validators.
03 / RESEARCHPlanned
Independent evaluation & context models
Add independently authored scenarios, held-out tools and workflows, multi-step cases and a context-aware model comparison against simpler baselines.
Release gate: publish misses, false alerts, calibration, abstention, latency and data provenance. Synthetic contract results alone are insufficient.
04 / WORKFLOWPlanned
More framework adapters & release comparisons
Completed OpenAI and Anthropic client-tool adapters and JSON CI reports are available. Planned work adds framework hooks, version comparisons and additional report formats.
Release gate: tested schema mappings, redaction controls and documented integration coverage. No claim of automatic universal agent discovery.
05 / COLLABORATIONExploratory
Private team workspaces
Explore self-hosted collaboration with verified identities, scoped access, retention controls and an accountable review history.
Release gate: authentication, authorization, tenant isolation, encryption and operational testing before accepting company traces.
06 / CONTROLExploratory
Opt-in execution boundaries
The Python execution wrapper supports explicit observe and enforce modes. Planned work adds coding-agent connectors and stronger workflow coordination.
Release gate: verified authority, non-bypassable integration tests and explicit operator control. Never a promise to prevent every threat.
No release dates are set.Planned features require the listed tests before release. Current results do not establish model quality or enterprise readiness.Read the developer guide →
BEFORE YOUR FIRST RUN
Frequently asked questions
Does it work with Gemini, Claude, or other agents?
The evaluator is model-neutral because it accepts normalized traces. Map your custom tools to supported checks in the Connect agents page or SDK. Completed-response adapters for OpenAI Responses, Chat Completions and Anthropic Messages are included. Your application must call the adapter; there is no automatic interception.
What does a risk score mean?
It is the maximum triggered rule severity on a 0–100 scale. It is not a calibrated probability of harm. Review the triggered rules and evidence rather than treating the number as certainty.
Can I use it with a company’s sensitive traces?
The downloaded console binds to loopback and has no login. Hosted deployments protect a shared workspace with authentication and HTTPS, but have no per-user roles or tenant isolation. SDK evidence can be redacted before storage. Review the deployment and privacy documentation before using sensitive traces.
Does passing the test suite mean my agent is secure?
No. These are hand-authored synthetic contract checks for implemented rules. They are not an independent security benchmark, and they do not cover arbitrary attacks, model behavior or hidden history.
What does it cost?
The included deterministic engine requires only Python’s standard library and makes no paid API calls. There is no subscription or billing flow in this preview.
INSTALLATION
Run the evaluation console
Download the ZIP, extract it, and run the included launcher. Integrate through Python, HTTP or CLI.
python run.py
CREATOR & OPEN SOURCE LEAD
Arun Kumar
Causal Inference & AI Research
I created OpenCaliber as an open-source project to learn and collaborate with developers on evaluating AI agent actions and inspecting policy findings.
Connect on LinkedIn ↗