Skip to main content
Critiqor evaluations are only as useful as the sessions that generate them. A session with trivial prompts and no edge cases will produce a limited signal. A session that reflects your agent’s real workload — with varied inputs, realistic tasks, and deliberate stress testing — produces an evaluation you can act on. Follow these practices to consistently get accurate, meaningful results.

Observe realistic workloads

Test with prompts and tasks that reflect how the agent is actually used — not toy examples.

Run multiple sessions

One run is a single data point. Multiple runs across varied scenarios give you a reliable picture.

Compare runs over time

After every meaningful change, run a new session and compare it to your baseline.

Act on recommendations

The Recommendations section is your improvement checklist. Work through it in severity order.

Test edge cases

Deliberately stress your agent with ambiguous inputs and failure scenarios for maximum signal.

Review before promoting

Treat Critiqor as a required review step before releasing agent changes to production.

​
Observe realistic workloads

The quality of your evaluation depends directly on the quality of your session. Use prompts, tasks, and interaction patterns that mirror real production usage. If your agent handles customer support queries, run customer support queries. If it performs multi-step research tasks, run those. Minimal or trivial prompts — simple greetings, single-word queries, or copy-pasted boilerplate — produce limited signal and can make a fragile agent appear more reliable than it is. What good session coverage looks like:
  • A variety of prompt types, not just your most common case
  • Prompts of different lengths and complexity levels
  • Tasks that exercise different tools or capabilities your agent relies on
  • At least a few prompts that are intentionally harder or more ambiguous

​
Run multiple sessions

A single observation run gives you one data point. Trust scores and diagnoses can vary across sessions depending on the specific prompts and conditions you tested. Running multiple sessions — especially across different prompt sets and use cases — gives Critiqor enough evidence to produce a stable, confident reliability picture. Start by establishing a baseline with your current agent. Then re-run sessions whenever you make a meaningful change. Over time, you’ll build a history of runs that makes trends immediately visible.

​
Compare runs over time

After every meaningful change to your agent — a prompt update, a new tool integration, a model version bump, a config tweak — run a new observation session and compare the results to your previous runs. Use critiqor runs to list your full evaluation history, then open specific runs side-by-side:
Look for changes in trust score, shifts in the Primary Diagnosis, and whether previously flagged issues from the Recommendations section have been resolved. Consistent improvement across runs is a strong signal your changes are working.

​
Act on recommendations

Every evaluation includes a Recommendations section in the dashboard — a prioritized list of specific, actionable improvements based on what Critiqor observed. Treat this as your improvement checklist, not background reading. Work through recommendations in order of severity. Higher-severity findings have more impact on your trust score and reliability in production. After addressing a finding, run a new session to confirm the improvement was effective before moving on. Ignoring recommendations and re-running sessions will show little or no improvement in your trust score — Critiqor will continue to surface the same issues until they’re resolved.

​
Test edge cases

Your agent’s weakest points are rarely exposed by your most common, well-formed prompts. Deliberately test the scenarios where it’s most likely to struggle:
  • Ambiguous inputs — questions that are underspecified or have multiple valid interpretations
  • Tool failures — what happens when an external tool returns an error or unexpected output?
  • Long sessions — extended conversations or multi-step tasks that accumulate context
  • Out-of-scope requests — prompts that push the agent beyond its intended use case
  • Conflicting instructions — inputs that create tension between goals or constraints
Edge-case sessions tend to produce the most diagnostic signal. They surface issues that wouldn’t appear under normal conditions — exactly the issues that matter most before a production release.

​
Review before promoting

Treat Critiqor as a required step in your release process, not an optional audit after the fact. Before promoting any agent change — to staging, to a wider audience, or to production — run an observation session and review the evaluation. If your trust level is Low or Moderate, investigate the Primary Diagnosis and Recommendations before releasing. A Low trust level in particular warrants a close review: it indicates reliability issues significant enough to affect real users. Use the critiqor check command to automate this gate in your deployment pipeline. See Policy Checks for details.
Label your sessions mentally before you run them — note what you changed and why. When you’re comparing run_005 to run_003 two weeks later, you’ll know exactly what was different between them and whether the trust score change is explained by your update.