Halios
Agent evaluation built around your coding workflow.
Halios helps teams build, run, and improve AI agent evaluations using the coding agents they already use every day. Most agent evaluation workflows require stitching together separate SDKs, proprietary instrumentation, static datasets, and disconnected evaluation dashboards. Halios brings that entire workflow directly into your coding environment. Instead of learning a new evaluation framework or writing custom test harnesses, you describe what you want to test. Your coding agent uses Halios to generate realistic multi-turn scenarios, run your agent code, investigate failures, and verify improvements.How it works
1
Install Halios
Add the Halios skill to your coding agent (Cursor, Claude Code, Codex, or Antigravity).
2
Describe what you want to test
Ask your coding agent to evaluate an agent, workflow, or behavior.Example:
3
Your coding agent builds the evaluation workflow
Halios helps your coding agent:
- inspect and understand your agent’s behavior
- generate realistic multi-turn test scenarios
- define deterministic and semantic evaluation checks
- connect OpenTelemetry auto-instrumentation
- run baseline evaluations and calculate reliability scores
4
Improve with feedback
When evaluations fail, Halios provides detailed trace data. Your coding agent investigates the root cause, proposes prompt or code fixes, and verifies improvements against the test suite.
What makes Halios different?
Coding-agent native
Your coding agent is already where your application is designed, built, and refactored. Halios turns your coding agent into an eval engineer by giving it the tools to author, run, and debug evaluation suites directly in your repository.Scenario-based evaluation
Real AI agents are not simple single-turn input/output functions. They call tools, make decisions, and adapt over multiple conversational turns. Halios evaluates agent behavior through dynamic multi-turn scenarios rather than static question-and-answer datasets.OpenTelemetry first
Halios works with standard OpenTelemetry-compatible traces and auto-instrumentors (OpenLLMetry, Traceloop, OpenInference). No proprietary tracing SDK or vendor lock-in is required.Continuous improvement loop
Evaluation is not a one-time benchmark. Halios helps teams catch regressions in CI/CD, diagnose failures from production traces, and systematically optimize system prompts over time.Run your first eval
Connect a repository and run a baseline evaluation in under 2 minutes.
Prompting guide
Learn how to direct your coding agent to build and maintain evals.