[{"data":1,"prerenderedAt":126},["ShallowReactive",2],{"term-s\u002Fsemantic-caching":3,"related-s\u002Fsemantic-caching":109},{"id":4,"title":5,"acronym":6,"body":7,"category":89,"description":90,"difficulty":91,"extension":92,"letter":93,"meta":94,"navigation":95,"path":96,"related":97,"seo":103,"sitemap":104,"stem":107,"subcategory":6,"__hash__":108},"terms\u002Fterms\u002Fs\u002Fsemantic-caching.md","Semantic Caching",null,{"type":8,"value":9,"toc":83},"minimark",[10,15,19,23,26,30,72,76,79],[11,12,14],"h2",{"id":13},"eli5-the-vibe-check","ELI5 - The Vibe Check",[16,17,18],"p",{},"A normal cache only helps if someone asks the exact same question twice, word for word. Semantic caching is smarter: it checks whether a new question means roughly the same thing as one it's already answered, using embeddings instead of exact string matching. Ask 'how do I reset my password' today and 'password reset help' tomorrow, and the second one can skip the LLM call entirely.",[11,20,22],{"id":21},"real-talk","Real Talk",[16,24,25],{},"Semantic caching stores past query-response pairs indexed by embedding, and on a new request, checks whether an existing cached query falls within a similarity threshold before deciding whether to serve the cached response or call the LLM. It reduces both cost and latency for LLM applications with high query overlap, at the risk of returning a stale or subtly wrong answer if the similarity threshold is set too loosely.",[11,27,29],{"id":28},"show-me-the-code","Show Me The Code",[31,32,37],"pre",{"className":33,"code":34,"language":35,"meta":36,"style":36},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","cached = semantic_cache.lookup(query_embedding, threshold=0.95)\nif cached:\n    return cached.response  # skip the LLM call entirely\nresponse = llm.generate(prompt)\nsemantic_cache.store(query_embedding, response)\n","python","",[38,39,40,48,54,60,66],"code",{"__ignoreMap":36},[41,42,45],"span",{"class":43,"line":44},"line",1,[41,46,47],{},"cached = semantic_cache.lookup(query_embedding, threshold=0.95)\n",[41,49,51],{"class":43,"line":50},2,[41,52,53],{},"if cached:\n",[41,55,57],{"class":43,"line":56},3,[41,58,59],{},"    return cached.response  # skip the LLM call entirely\n",[41,61,63],{"class":43,"line":62},4,[41,64,65],{},"response = llm.generate(prompt)\n",[41,67,69],{"class":43,"line":68},5,[41,70,71],{},"semantic_cache.store(query_embedding, response)\n",[11,73,75],{"id":74},"when-youll-hear-this","When You'll Hear This",[16,77,78],{},"\"Semantic caching cut our API bill by a third on the FAQ bot.\" \u002F \"Loosen the threshold too far and semantic caching starts returning wrong answers.\"",[80,81,82],"style",{},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":36,"searchDepth":50,"depth":50,"links":84},[85,86,87,88],{"id":13,"depth":50,"text":14},{"id":21,"depth":50,"text":22},{"id":28,"depth":50,"text":29},{"id":74,"depth":50,"text":75},"ai","A normal cache only helps if someone asks the exact same question twice, word for word.","intermediate","md","s",{},true,"\u002Fterms\u002Fs\u002Fsemantic-caching",[98,99,100,101,102],"Embedding","Cosine Similarity","Cost Per Token","Prompt Caching","Latency Budget",{"title":5,"description":90},{"changefreq":105,"priority":106},"weekly",0.7,"terms\u002Fs\u002Fsemantic-caching","jr4fEwXToEgBX65UM_BdvlFbNrdKRt5_jfKwCELpUEA",[110,113,117,120,123],{"title":99,"path":111,"acronym":6,"category":89,"difficulty":91,"description":112},"\u002Fterms\u002Fc\u002Fcosine-similarity","Cosine similarity measures how similar two things are by comparing the angles of their vectors.",{"title":100,"path":114,"acronym":6,"category":89,"difficulty":115,"description":116},"\u002Fterms\u002Fc\u002Fcost-per-token","beginner","Cost per token is how much each token (input or output) costs with a given AI provider. Flagship models cost more per token than cheap ones.",{"title":98,"path":118,"acronym":6,"category":89,"difficulty":91,"description":119},"\u002Fterms\u002Fe\u002Fembedding","An embedding is turning words, sentences, or entire documents into lists of numbers (vectors) that capture their meaning.",{"title":102,"path":121,"acronym":6,"category":89,"difficulty":91,"description":122},"\u002Fterms\u002Fl\u002Flatency-budget","A latency budget is deciding upfront how many milliseconds each part of your AI pipeline is allowed to spend before the whole thing feels slow to a user.",{"title":101,"path":124,"acronym":6,"category":89,"difficulty":91,"description":125},"\u002Fterms\u002Fp\u002Fprompt-caching","Prompt caching is a speed and cost optimization where the AI remembers the beginning of your prompt so it doesn't have to re-process it every time.",1792528122658]