eval

Runs Eval.create / .run against a real LLM and prints aggregate + per-row results.

Run (Ollama, no API keys)

ollama serve &
ollama pull llama3.2:3b
pnpm install
pnpm run run

Run (OpenRouter cloud)

export OPENROUTER_API_KEY=sk-or-...
pnpm run run

What it shows

Eval.create({...}).run() returns a populated EvalRun shape (D202, D209)
Scorers.containsExpected() + Scorers.regex() applied to each row
Aggregate includes meanScore, passRatio, errorRows, tokensInTotal, durationMsP50, durationMsP95 (D211)
v1 scale: keep datasets ≤ 10k rows (EC-11 — v1 materializes the dataset in memory; partition manually for larger evals or wait for streaming v2)

LLM-as-judge

For subjective scoring, swap the second scorer:

Scorers.llmJudge({
  model: { id: "openai/gpt-4o-mini" },
  apiKey: process.env.OPENROUTER_JUDGE_KEY ?? process.env.OPENROUTER_API_KEY ?? "",
  criteria: "The answer is concise and accurate.",
  rubric: "continuous",
}),

Cost note (EC-12): llmJudge doubles the per-row LLM cost. For 1000 rows × gpt-4o-mini, expect ~$1.50 (eval) + ~$1.50 (judge) = $3.00. The aggregate.tokensInTotal only reflects the EVAL agent's tokens, not the judge's — forecast accordingly.

Code

run.ts

/**
 * Example: `Eval.create / .run` against a real LLM.
 *
 *   pnpm install
 *   ollama serve & ollama pull llama3.2:3b
 *   pnpm run run
 *
 * Or with cloud:
 *   OPENROUTER_API_KEY=... pnpm run run
 *
 * Prints the EvalRun JSON (aggregate + rows) to stdout.
 */

import { Eval, Scorers, type EvalRun } from "@theokit/sdk";

const useCloud = typeof process.env.OPENROUTER_API_KEY === "string";

const agent = useCloud
  ? {
      apiKey: process.env.OPENROUTER_API_KEY ?? "",
      model: { id: "openai/gpt-4o-mini" },
      local: { cwd: process.cwd(), sandboxOptions: { enabled: false } as const },
      providers: {
        routes: [{ capability: "chat" as const, provider: "openrouter" }],
        fallback: ["openrouter"],
      },
    }
  : {
      apiKey: "ollama-local",
      model: { id: "ollama/llama3.2:3b" },
      local: { cwd: process.cwd(), sandboxOptions: { enabled: false } as const },
    };

const run: EvalRun = await Eval.create({
  name: "smoke-eval",
  dataset: [
    { input: "Reply with the single word: ok.", expected: "ok" },
    { input: "Say jazz in one word.", expected: "jazz" },
    { input: "What is 2 + 3? Reply with just the digit.", expected: "5" },
    { input: "Name a primary color. Reply with one word.", expected: "red" },
    { input: "Say hi in one word.", expected: "hi" },
  ],
  scorers: [
    Scorers.containsExpected({ caseSensitive: false }),
    Scorers.regex(/[a-zA-Z0-9]/),
  ],
  agent,
  concurrency: 2,
  metadata: { example: "eval-smoke", mode: useCloud ? "cloud" : "ollama" },
}).run();

console.log(JSON.stringify(run, null, 2));
console.log("");
console.log(
  `Mean: ${run.aggregate.meanScore.toFixed(3)} | Pass: ${(run.aggregate.passRatio * 100).toFixed(1)}% | Errors: ${run.aggregate.errorRows}/${run.aggregate.totalRows} | Tokens in/out: ${run.aggregate.tokensInTotal}/${run.aggregate.tokensOutTotal}`,
);

Run

cd examples/eval
cp .env.example .env  # fill in keys
pnpm install
pnpm run run

Repository

examples/eval

eval

On this page