Semantic cache

Canonical contract: packages/sdk/docs.md — "Semantic cache (v1.18+)".

A cache that recognizes semantically equivalent prompts, not just byte-equal ones. Cuts cost 30-70% in workloads with repetitive queries.

Plugin mode (recall + inject)

import { Cache, Agent } from "@theokit/sdk";

const cache = Cache.semantic({
  embedder: "openai/text-embedding-3-small",
  threshold: 0.85,
  ttl: 60 * 60 * 1000, // 1 hour
  namespace: "weather-bot",
});

const agent = await Agent.create({
  apiKey, model,
  plugins: [cache.asPlugin()],
});

When a prompt arrives, the cache:

1. Looks up exact KV match (composite key: namespace:embedderId:modelId:hash(prompt)).
2. If miss, embeds the prompt and runs vector similarity against stored entries.
3. If a match scores ≥ threshold, injects the cached response as a hint (the LLM still runs, but with a strong nudge).

Direct mode (true short-circuit)

For zero-LLM-call short-circuit:

const cached = await cache.consult(prompt);
if (cached) return cached;

const result = await /* expensive LLM call */;
await cache.remember(prompt, result);

consult returns null on miss. remember stores after the fact.

Composition

The composite key (ADR D253) ensures changing the embedder or model invalidates the cache automatically (ADR D258).

Skipped automatically

The cache skips storing runs that invoked tools (ADR D266). Tool-use turns aren't semantically equivalent enough — they have side effects.

Persistence

Default: in-memory LRU (1000 entries, ADR D261). JSON-on-disk opt-in:

Cache.semantic({
  // ...
  persistence: { backend: "json", dir: ".theokit/cache" },
});

Telemetry

OTel events cache.lookup / cache.store with hit/miss labels (ADR D262).

Composes with Anthropic prompt caching

Anthropic's prompt_caching (cache_control breakpoints) is orthogonal. The SDK semantic cache runs above that layer (ADR D263).

API reference