[{"data":1,"prerenderedAt":159},["ShallowReactive",2],{"term-h\u002Fhedged-request":3,"related-h\u002Fhedged-request":139},{"id":4,"title":5,"acronym":6,"body":7,"category":119,"description":120,"difficulty":121,"extension":122,"letter":123,"meta":124,"navigation":125,"path":126,"related":127,"seo":133,"sitemap":134,"stem":137,"subcategory":6,"__hash__":138},"terms\u002Fterms\u002Fh\u002Fhedged-request.md","Hedged Request",null,{"type":8,"value":9,"toc":113},"minimark",[10,15,19,23,26,30,102,106,109],[11,12,14],"h2",{"id":13},"eli5-the-vibe-check","ELI5 - The Vibe Check",[16,17,18],"p",{},"A hedged request is a bet-hedging trick against tail latency: if the first response hasn't come back within a short window, you fire off a duplicate request to a different replica, and just take whichever one answers first, then quietly cancel the loser. It costs a bit of extra load on the system, but it's a surprisingly cheap way to chop off the slowest, most annoying outlier responses.",[11,20,22],{"id":21},"real-talk","Real Talk",[16,24,25],{},"A hedged request is a latency-hiding technique where a client sends a duplicate request to a different replica if the original hasn't returned within a set threshold (often the observed p95 or p99 latency), accepting whichever response arrives first and cancelling or discarding the other. It trades a modest increase in request volume for a significant reduction in tail latency, since duplicating the small fraction of unusually slow requests is far cheaper than duplicating all traffic, and is most effective when slowness is caused by transient per-request variance rather than systemic overload.",[11,27,29],{"id":28},"show-me-the-code","Show Me The Code",[31,32,37],"pre",{"className":33,"code":34,"language":35,"meta":36,"style":36},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Fire a hedged request if the primary is slow\nresult = await asyncio.wait(\n    [call_replica_a(req)],\n    timeout=p95_latency_ms \u002F 1000\n)\nif not result.done:\n    result = await asyncio.wait(\n        [call_replica_a(req), call_replica_b(req)],\n        return_when=asyncio.FIRST_COMPLETED\n    )\n","python","",[38,39,40,48,54,60,66,72,78,84,90,96],"code",{"__ignoreMap":36},[41,42,45],"span",{"class":43,"line":44},"line",1,[41,46,47],{},"# Fire a hedged request if the primary is slow\n",[41,49,51],{"class":43,"line":50},2,[41,52,53],{},"result = await asyncio.wait(\n",[41,55,57],{"class":43,"line":56},3,[41,58,59],{},"    [call_replica_a(req)],\n",[41,61,63],{"class":43,"line":62},4,[41,64,65],{},"    timeout=p95_latency_ms \u002F 1000\n",[41,67,69],{"class":43,"line":68},5,[41,70,71],{},")\n",[41,73,75],{"class":43,"line":74},6,[41,76,77],{},"if not result.done:\n",[41,79,81],{"class":43,"line":80},7,[41,82,83],{},"    result = await asyncio.wait(\n",[41,85,87],{"class":43,"line":86},8,[41,88,89],{},"        [call_replica_a(req), call_replica_b(req)],\n",[41,91,93],{"class":43,"line":92},9,[41,94,95],{},"        return_when=asyncio.FIRST_COMPLETED\n",[41,97,99],{"class":43,"line":98},10,[41,100,101],{},"    )\n",[11,103,105],{"id":104},"when-youll-hear-this","When You'll Hear This",[16,107,108],{},"\"We hedge any request that's taken longer than the p95, it shaved 40 percent off our tail latency.\" \u002F \"Don't hedge during an overload, that just doubles the load on an already struggling system.\"",[110,111,112],"style",{},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":36,"searchDepth":50,"depth":50,"links":114},[115,116,117,118],{"id":13,"depth":50,"text":14},{"id":21,"depth":50,"text":22},{"id":28,"depth":50,"text":29},{"id":104,"depth":50,"text":105},"architecture","A hedged request is a bet-hedging trick against tail latency: if the first response hasn't come back within a short window, you fire off a duplicate reques...","advanced","md","h",{},true,"\u002Fterms\u002Fh\u002Fhedged-request",[128,129,130,131,132],"Tail Latency","Load Shedding","Circuit Breaker","Load Balancer","Latency Budget",{"title":5,"description":120},{"changefreq":135,"priority":136},"weekly",0.7,"terms\u002Fh\u002Fhedged-request","DhLDDKohpvLdzY5vCqzn9oNP-_tnagi8Je2lZaYrhzM",[140,144,148,152,156],{"title":130,"path":141,"acronym":6,"category":119,"difficulty":142,"description":143},"\u002Fterms\u002Fc\u002Fcircuit-breaker","intermediate","Circuit Breaker is like the electrical circuit breaker in your house.",{"title":132,"path":145,"acronym":6,"category":146,"difficulty":142,"description":147},"\u002Fterms\u002Fl\u002Flatency-budget","ai","A latency budget is deciding upfront how many milliseconds each part of your AI pipeline is allowed to spend before the whole thing feels slow to a user.",{"title":131,"path":149,"acronym":6,"category":150,"difficulty":142,"description":151},"\u002Fterms\u002Fl\u002Fload-balancer","networking","A load balancer is like a traffic cop for servers.",{"title":129,"path":153,"acronym":6,"category":154,"difficulty":121,"description":155},"\u002Fterms\u002Fl\u002Fload-shedding","backend","Load shedding is intentionally dropping some requests when your server is overwhelmed so it can keep serving the rest.",{"title":128,"path":157,"acronym":6,"category":119,"difficulty":142,"description":158},"\u002Fterms\u002Ft\u002Ftail-latency","Your average response time might look great, but tail latency is about that unlucky slice of requests, the slowest 1 percent or so, that take way longer th...",1792528090815]