Eval suite
Eval.create and Eval.run — eval-as-code with built-in scorers.
Canonical contract:
packages/sdk/docs.md— "Eval suite (v1.15+)".
The Eval API lets you write evals as code (not as dashboards). Inspired by Braintrust + LangSmith but local-first.
Define an eval
import { Eval, Scorers, Agent } from "@theokit/sdk";
const evaluation = Eval.create({
name: "math-basics",
dataset: [
{ input: "What is 2+2?", expected: "4" },
{ input: "What is 10*5?", expected: "50" },
{ input: "What is sqrt(16)?", expected: "4" },
],
scorers: [
Scorers.contains("expected"),
Scorers.llmJudge({
apiKey: process.env.OPENROUTER_API_KEY,
prompt: "Is the output a correct numeric answer? Score 0-1.",
}),
],
agent: async (input) => {
const agent = await Agent.create({
apiKey: process.env.OPENROUTER_API_KEY,
model: { id: "openai/gpt-4o-mini" },
});
try {
const r = await (await agent.send(input)).wait();
return r.result ?? "";
} finally {
await agent.dispose();
}
},
});
const run = await evaluation.run();
console.log(run.aggregate); // { meanScore, p50, p95, errorRows, tokensIn, tokensOut }Built-in scorers
| Scorer | Returns | Use |
|---|---|---|
Scorers.exactMatch("expected") | 0 or 1 | Strict equality |
Scorers.contains("expected") | 0 or 1 | Substring match |
Scorers.jsonValid() | 0 or 1 | Is the output parseable JSON |
Scorers.lengthBetween(min, max) | 0 or 1 | Length in range |
Scorers.llmJudge({ apiKey, prompt }) | 0..1 | LLM-as-judge with separate apiKey |
Scorers.llmJudge requires its own apiKey (ADR D205), separate from the agent under test — prevents cross-contamination of metrics.
Parallelism
Eval.run internally uses Agent.batch (ADR D204) — default concurrency 4, per-prompt isolation (ADR D208), pool sharing via AsyncLocalStorage.
Error isolation
If one row throws, the run continues — the row gets error: { message } in its result and errorRows increments in the aggregate. Plan accordingly.
CLI
The theokit CLI wraps Eval.run (ADR D212):
theokit eval ./my-eval.tsTelemetry
Eval traces piggyback on the Telemetry namespace (D34) — eval.run and per-row spans. No separate tracer.