Core terms
Scenario
The reusable test situation you want to evaluate. It defines the initial user goal, persona, constraints, and multi-turn conversation arc.Trial
One fresh execution generated from a scenario. Instead of replaying a saved transcript, Halios simulates a live user interaction with your agent for each trial.Trace
The execution evidence emitted by your agent during a trial. Contains the full conversational turn history, model calls, tool arguments, retrieval context, and latency metrics.Check
One specific invariant you want to verify about your agent’s behavior (e.g. answer groundedness, regex format match, policy compliance, or tool argument schema).Eval suite
The collection of scenarios, checks, and configuration defined for an agent or workflow (stored in your repository under.halios/).
Run
One complete execution pass of an eval suite against the current agent implementation, recording trial traces, check evaluations, and pass rates.pass@k
Repeating the same scenario times (e.g. 3 or 5 trials) to measure consistency. A scenario passes only if all trials succeed, ensuring that passing was not just random chance.Run your first eval
Get started with your coding agent in minutes.
CLI reference
Command reference for automation and CI pipelines.