Evals-as-a-Service Module

The Evals-as-a-Service Module determines whether agent workflows are improving over time, whether policy changes reduce harmful behavior, and whether new versions should be allowed into production. It combines automated scoring with human review for comprehensive quality control.

Core Features

Risk-Weighted Scoring

Test cases can be assigned priority levels that weight their impact on overall scores. Higher priority failures have greater impact on the final pass rate.

Benchmark & Comparison

Compare agent versions systematically to measure improvement:

Shadow Testing

Test new versions safely by running them alongside production with real traffic:

Auto-Generated Tests

Automatically create test cases from real production traces:

API Endpoints

Test Suites

{
  "name": "Customer Support Evals",
  "description": "Test suite for support agent",
  "suite_type": "workflow",
  "thresholds": {
    "pass_rate": 0.85,
    "latency_ms": 5000
  }
}

Test Cases

{
  "name": "Handle refund request",
  "input": { "query": "I want a refund for order #123" },
  "expected_output": { "action": "process_refund" },
  "priority": "high",
  "validation_rules": {
    "required_fields": ["action", "order_id"]
  }
}

Eval Runs

Human Reviews

Benchmarks

Release Gates

{
  "name": "Production Gate",
  "gate_type": "blocking",
  "thresholds": {
    "pass_rate": 0.90,
    "latency_ms": 3000,
    "no_critical_failures": true
  }
}

Auto-Generation

Shadow Testing

{
  "traffic_percentage": 10,
  "baseline_version": "v1.0.0",
  "candidate_version": "v1.1.0",
  "duration_hours": 24
}