Prompting Halios

Tell your coding agent what you want to evaluate or improve. It can inspect your repository, use the Halios skill and CLI, and decide how to carry out the work. Start simple. Add constraints only when they matter. Add details when a minimal prompt does not work or when you need the agent to act on a specific trace, evaluation run, or check.

Build

Create an evaluation suite with realistic scenarios, edge cases, and checks. Start with:

Set up evals for this agent.

Add focus when needed:

Set up evals for the refund flow. Focus on identity verification and policy violations.

Review the current eval suite and add coverage for tool failures and ambiguous user requests.

Add a check that the agent discloses it is an AI before continuing the conversation.

Add an LLM-as-judge check to verify that the agent handled the user's request politely.

Add a deterministic check to verify that the agent disclosed that the conversation may be recorded for quality and training purposes.

Ask your coding agent to explain what it created:

Walk me through the scenarios and checks in this eval suite.

Add a CI release gate:

Add a GitHub Action to run evals on every PR. Block the merge if the eval score drops below 0.8.

Turn a production failure into a regression test:

Add trace <trace_id> as a scenario and check so we catch this failure again.

Run

Simulate realistic user interactions and evaluate how your agent behaves across different paths.

Run the eval suite against the current agent.

Run the refund scenarios with three trials each and show me any failures.

Rerun the scenarios affected by my latest changes.

Compare this run with the previous baseline.

Halios runs the agent again with fresh simulations. Each scenario produces a new interaction rather than re-scoring a saved transcript.

Debug

Trace failures back to conversations, tool calls, and decisions.

Why did this evaluation fail?

Investigate the failed refund scenario and show me what happened.

Which tool call caused this check to fail?

Find the common cause behind the failures in this run.

Trace this failure back to the relevant prompt, tool, or agent code.

You can start from a run, scenario, trace, or failed check. Your coding agent uses Halios trace data together with repository context to investigate the root cause.

Improve

Verify fixes, catch regressions, and continuously improve your agent.

Fix this failure and rerun the affected evals.

Improve the agent on the failing scenarios without regressing the ones that already pass.

Try a better system prompt and compare it with the current version.

Optimize this agent against the eval suite.

Keep the change only if it improves the eval results without introducing regressions.

The loop continues as needed: run → investigate → change → rerun → compare Halios provides the evaluation data; your coding agent uses that information to propose and apply changes.

CLI reference

Command reference for automation and CI/CD pipelines.

Evaluation concepts

Core mental model and key evaluation terms.

Get Started

Reference

Build

Run

Debug

Improve

CLI reference

Evaluation concepts

​Build

​Run

​Debug

​Improve

CLI reference

Evaluation concepts

Build

Run

Debug

Improve