Cookbook
cache
Demonstrates `Cache.semantic` + `Cache.consult` (Adoption Roadmap #6; ADRs D249-D266).
cache
Demonstrates Cache.semantic + Cache.consult (Adoption Roadmap #6; ADRs D249-D266).
Run (OpenRouter cloud)
export OPENROUTER_API_KEY=sk-or-...
pnpm install
pnpm run runWhat it shows
Cache.semantic({ embedder, threshold, ttl, namespace })factory.cache.consult(prompt)— direct lookup withhit: booleanoutcome +source: "kv" | "semantic".cache.remember(prompt, response)— explicit store after dispatching the LLM yourself.ttl.excluderegex — time-sensitive prompts (weather, today, now) bypass cache.cache.stats()— kvHits / semanticHits / misses / excluded counters.
v1 limitations (documented)
- Plugin mode provides recall + context inject — the LLM is still called on hit and pre-loaded with the cached answer. For true short-circuit (skip the LLM call entirely), use
cache.consult()directly and dispatch your own LLM call only on miss (the demo shows this pattern). - No streaming cache (D256) — only
agent.sendis cached, notagent.stream. - No adaptive threshold per entry (D254) — single global threshold; tune via
Cache.semantic({ threshold: 0.95 })for high-stakes scenarios. - No tool-use cache (D266 / EC-10) — runs that invoked tools are NEVER cached (replay would lose side-effects).
- Embedder change invalidates (D258) —
embedder.idis part of the cache key.
Pairing with Anthropic prompt_caching (D263)
Cache.semantic resolves paraphrases BEFORE the LLM. Anthropic prompt_caching gives 90% discount on prefix-identical input AFTER hitting the LLM. They're orthogonal — use both for compound savings (~95% in ideal workloads):
[user query] → Cache.semantic hit? → return cached
→ miss → LLM call with cache_control on system/tools (90% discount)Code
/**
* Semantic Cache demo (Adoption Roadmap #6; ADRs D249-D266).
*
* Shows:
* 1. First query: miss → LLM called → cache.remember stores it
* 2. Paraphrase: semantic hit (no LLM call needed via consult())
* 3. Time-sensitive query: bypassed via exclude regex
* 4. Stats summary
*
* Run:
* export OPENROUTER_API_KEY=sk-or-...
* pnpm install
* pnpm run run
*/
import { Agent, Cache } from "@theokit/sdk";
const OPENROUTER = process.env.OPENROUTER_API_KEY;
if (OPENROUTER === undefined || OPENROUTER.length === 0) {
console.error("OPENROUTER_API_KEY missing — see .env.example");
process.exit(1);
}
// A deterministic toy embedder for the demo (avoids spending OpenAI tokens
// on every prompt — production users plug a real EmbeddingRuntime here).
const toyEmbedder = {
id: "toy-letter",
model: "letter-bag-1",
dimension: 26,
async embed(texts: ReadonlyArray<string>): Promise<number[][]> {
return texts.map((t) => {
const v = new Array(26).fill(0);
const norm = t.toLowerCase().replace(/[^a-z]/g, "");
for (const ch of norm) {
const i = ch.charCodeAt(0) - 97;
if (i >= 0 && i < 26) v[i] += 1;
}
const sum = v.reduce((a, b) => a + b, 0) || 1;
return v.map((x) => x / sum);
});
},
};
async function main(): Promise<void> {
const cache = Cache.semantic({
embedder: toyEmbedder,
threshold: 0.4,
ttl: {
default: process.env.CACHE_TTL ?? "1h",
exclude: /\b(weather|today|now|current|stock)\b/i,
},
namespace: "demo",
modelId: "openai/gpt-4o-mini",
});
const agent = await Agent.create({
apiKey: OPENROUTER,
model: { id: "openai/gpt-4o-mini" },
local: { cwd: process.cwd(), sandboxOptions: { enabled: false } as const },
name: "demo-agent",
plugins: [cache.asPlugin()],
});
console.log("\n=== Query 1: 'What is the capital of France?' (miss expected) ===");
const t1 = Date.now();
const m1 = await cache.consult("What is the capital of France?");
if (m1.hit) {
console.log("(unexpected hit)", m1.response.slice(0, 80));
} else {
const r1 = await agent.send("What is the capital of France? Answer in one short sentence.");
const result1 = await r1.wait();
const text = result1.status === "finished" ? (result1.result ?? "") : "";
await cache.remember("What is the capital of France?", text);
console.log("LLM:", text.slice(0, 200));
}
console.log("Elapsed:", Date.now() - t1, "ms");
console.log("\n=== Query 2: 'Tell me the capital city of France' (semantic hit expected) ===");
const t2 = Date.now();
const m2 = await cache.consult("Tell me the capital city of France");
if (m2.hit) {
console.log("CACHE HIT (", m2.source, ", dist=", m2.distance, ")");
console.log("Cached:", m2.response.slice(0, 200));
} else {
console.log("(unexpected miss)");
}
console.log("Elapsed:", Date.now() - t2, "ms");
console.log("\n=== Query 3: 'What is the weather in SF today?' (excluded by regex) ===");
const t3 = Date.now();
const m3 = await cache.consult("What is the weather in SF today?");
console.log("hit:", m3.hit, "(should always be false due to exclude regex)");
console.log("Elapsed:", Date.now() - t3, "ms");
const s = cache.stats();
console.log("\n=== Stats ===");
console.log(
`entries=${s.entries} kvHits=${s.kvHits} semanticHits=${s.semanticHits} misses=${s.misses} excluded=${s.excluded} evicted=${s.evicted} embedderFailures=${s.embedderFailures}`,
);
await agent.dispose();
}
main().catch((err) => {
console.error("cache demo failed:", err);
process.exit(1);
});
Run
cd examples/cache
cp .env.example .env # fill in keys
pnpm install
pnpm run run