Domain Expert Judge

The domain expert judge evaluates LLM responses and iteratively refines them when they don't meet quality standards. When a response is rejected, the judge's feedback is sent back to the same model for improvement - automatically, before your application sees the result.

How Cynos works

Every request moves through a task-aware inference loop.

1. Understand the intent

Cynos first identifies what kind of cybersecurity work the request represents. It considers the task, its complexity, the capabilities it requires, and the quality threshold the result must meet.

This goes beyond routing on prompt length or choosing a model tier in advance. Cynos reasons about the security intent behind the request so it can apply the right execution and evaluation strategy.

2. Select the best-fit model

Cynos then chooses the model best suited to that task. The selection can span specialized open-source models and closed-source frontier models, balancing demonstrated task performance with cost, latency, context, and availability.

The goal is not to choose the least expensive model. It is to choose the least expensive model that can successfully complete the job.

3. Evaluate the outcome

After the model responds, Cynos evaluates the result using criteria appropriate to the task. A good vulnerability analysis, for example, is judged differently from a good alert classification or detection rule.

This gives Cynos a task-specific quality bar rather than treating fluency as a proxy for correctness.

4. Improve or escalate when necessary

If the result clears the quality bar, Cynos returns it. If it does not, Cynos can continue the loop—refining the work with the same model or escalating the task to a more capable one.

Frontier models remain available for the tasks that need them. They simply stop being the default for every task.

Inside the refinement loop

Within step 3 and step 4 above, the judge runs a tight evaluate-and-retry cycle:

1. Judge evaluation - A lightweight judge model evaluates the response against the original prompt
2. Verdict - The judge returns PASS (serve response) or FAIL (needs refinement) with a reason
3. Refinement loop - On FAIL, the reason is fed back to the original model along with the conversation, and a new response is generated
4. Re-evaluation - The refined response is judged again
5. Iteration - Steps 3-4 repeat until PASS or the iteration budget is exhausted

Override header

Force the domain expert judge on or off per request using the x-cy-eh-domain-expert-judge header:

# Force judge on (even if disabled in config)
curl https://api.cynos.averlon.ai/v1/chat/completions \
  -H "Authorization: Bearer sk-cy-your-api-key" \
  -H "x-cy-eh-domain-expert-judge: on" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "Explain OWASP Top 10"}]
  }'

# Force judge off (skip evaluation entirely)
curl https://api.cynos.averlon.ai/v1/chat/completions \
  -H "Authorization: Bearer sk-cy-your-api-key" \
  -H "x-cy-eh-domain-expert-judge: off" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "Quick question"}]
  }'

When the header is off, the judge_effort body parameter is ignored.

Response metadata

Judged responses include quality information in extra_fields:

{
  "extra_fields": {
    "quality_judge": {
      "verdict": "pass",
      "reason": "Response accurately covers all aspects of the vulnerability",
      "domain": "vul_management",
      "refined": true,
      "iterations": [
        {
          "iteration": 0,
          "verdict": "fail",
          "reason": "Missing CVSS score and affected versions"
        },
        {
          "iteration": 1,
          "verdict": "pass",
          "reason": "Comprehensive coverage including CVSS, affected versions, and mitigations"
        }
      ]
    }
  }
}

Behavior notes

• Fail-open - If the judge itself fails (timeout, error), the response is served unchanged. Quality judging never blocks availability.
• Streaming - Quality judging works with streaming responses.
• Tool calls - By default, only the final response turn is judged, not intermediate tool-call turns in agentic loops.
• Exhaustion - If all refinement iterations are exhausted and the response still fails, the last evaluated response is served (not rejected).

Combining with task routing

Task routing and quality judge work together:

curl https://api.cynos.averlon.ai/v1/chat/completions \
  -H "Authorization: Bearer sk-cy-your-api-key" \
  -H "Content-Type: application/json" \
  -H "x-cy-eh-domain-expert-judge: on" \
  -H "x-cy-eh-task-route: on" \
  -d '{
    "judge_effort": "high",
    "messages": [
      {"role": "user", "content": "Analyze this CVE for exploitability"}
    ]
  }'

The task router picks the model, and the quality judge ensures the response meets standards. If the routed model produces a poor answer that can't be refined, the system falls back to the next model in the chain, which gets its own independent judge/refine cycle.