Hedged Request

ELI5 - The Vibe Check

A hedged request is a bet-hedging trick against tail latency: if the first response hasn't come back within a short window, you fire off a duplicate request to a different replica, and just take whichever one answers first, then quietly cancel the loser. It costs a bit of extra load on the system, but it's a surprisingly cheap way to chop off the slowest, most annoying outlier responses.

Real Talk

A hedged request is a latency-hiding technique where a client sends a duplicate request to a different replica if the original hasn't returned within a set threshold (often the observed p95 or p99 latency), accepting whichever response arrives first and cancelling or discarding the other. It trades a modest increase in request volume for a significant reduction in tail latency, since duplicating the small fraction of unusually slow requests is far cheaper than duplicating all traffic, and is most effective when slowness is caused by transient per-request variance rather than systemic overload.

Show Me The Code

# Fire a hedged request if the primary is slow
result = await asyncio.wait(
    [call_replica_a(req)],
    timeout=p95_latency_ms / 1000
)
if not result.done:
    result = await asyncio.wait(
        [call_replica_a(req), call_replica_b(req)],
        return_when=asyncio.FIRST_COMPLETED
    )

When You'll Hear This

"We hedge any request that's taken longer than the p95, it shaved 40 percent off our tail latency." / "Don't hedge during an overload, that just doubles the load on an already struggling system."