1. Switch to your agent repository root
Ensure your IDE or terminal workspace is open at the root of your agent repository.2. Prompt your coding agent
You can let your coding agent install the skill automatically, or install it manually first:Option A: All-in-one prompt
Ask your coding agent to install the skill and set up evaluations in one step:Option B: Install manually first
Add the skill to your coding agent:3. What happens under the hood
Depending on your coding agent’s permissions and configuration, it may ask for confirmation before making changes or running commands. You remain in control of what gets created and executed.
1
Inspect repository
The coding agent examines your codebase to understand the agent entrypoint, tools, system prompts, and critical user workflows.
2
Connect Halios
On first use, your coding agent helps authenticate your workspace with Halios. The skill and CLI handle communication with the Halios runtime.
3
Create evaluation artifacts
The coding agent creates test scenarios, checks, and configuration files (
.halios/) alongside your application.4
Configure telemetry
Halios uses OpenTelemetry-compatible traces. Existing instrumentation is reused, or your coding agent helps add standard auto-instrumentation (such as OpenLLMetry / Traceloop).
5
Simulate user interactions with agent
Halios generates multi-turn conversations based on the evaluation scenarios to produce realistic traces of agent-user interaction patterns. The coding agent creates a lightweight adapter so Halios can communicate with your agent during simulations.
6
Run evaluations
Halios scores the generated traces against your checks (deterministic rules, LLM judges, and reliability thresholds).
4. Review results
Your coding agent can summarize the run directly in your chat, or you can open the Halios dashboard to inspect:- Scenario results and pass rates across repeated trials
- Full multi-turn traces with user and assistant turns
- Tool calls, arguments, and return values
- Check evaluations and failure explanations
Prompting guide
Learn how to direct your coding agent to build and maintain your eval suite.
Evaluation concepts
Explore the core mental model and key evaluation terms.