Semantic Caching

ELI5 - The Vibe Check

A normal cache only helps if someone asks the exact same question twice, word for word. Semantic caching is smarter: it checks whether a new question means roughly the same thing as one it's already answered, using embeddings instead of exact string matching. Ask 'how do I reset my password' today and 'password reset help' tomorrow, and the second one can skip the LLM call entirely.

Real Talk

Semantic caching stores past query-response pairs indexed by embedding, and on a new request, checks whether an existing cached query falls within a similarity threshold before deciding whether to serve the cached response or call the LLM. It reduces both cost and latency for LLM applications with high query overlap, at the risk of returning a stale or subtly wrong answer if the similarity threshold is set too loosely.

Show Me The Code

cached = semantic_cache.lookup(query_embedding, threshold=0.95)
if cached:
    return cached.response  # skip the LLM call entirely
response = llm.generate(prompt)
semantic_cache.store(query_embedding, response)

When You'll Hear This

"Semantic caching cut our API bill by a third on the FAQ bot." / "Loosen the threshold too far and semantic caching starts returning wrong answers."