On the Effectiveness of Retries and Timeouts in Distributed Systems

2026-02-15T00:00:00+00:00

One of the adages that has stuck with me from my relatively brief employment at AWS was that every single network call had to have retries and timeouts. This can even be seen in the AWS CLI and SDKs, which all have retries (3 by default) and timeouts baked in.

On the surface this seems like a reasonable policy; retries are handy in case of arbitrary network failures and timeouts are a nice to have. However, I would argue that retries and timeouts by default across the entire system is unreasonably effective at mitigating a number of the challenges when developing a large-scale distributed system.

Retries by themselves have a number of nice properties that are easy to see: temporary network failures are mitigated (just retry), temporary service outages are mitigated (even nodes crashing mid-request; just retry), it is a forcing function for idempotency everywhere (if every client has retries, requests must be idempotent), and, combined with exponential back-offs, mitigates the effects of rate-limiting (trading it off for increased latency).

Timeouts add on a system-wide backpressure mechanism. The majority of services can be approximated as a priority queue of asynchronous requests to be processed. As their load increases, the number of requests to be processed increases, and the latency of a given request creeps up. Timing out latent requests reduces load on overloaded nodes, and with retries, gracefully rebalances to other nodes. Assuming exponential backoffs, this also gives time for a service to scale horizontally when under pressure. This effect propagates outwards from the point of overload: calling services will be stuck with more requests, they may timeout, scale-up, etc.

There are aspects to be aware of with this approach. If a client has timeouts lower than a service's reasonable response time you may see increased load as requests are unnecessarily retried. Ensuring retries and timeouts are properly tuned is important, and having clients developed by the service owners ensures that this is the case. The benefit of trivially getting system-wide backpressure and robustness to network|node failure is well worth it, especially as compared to the complexity of other approaches (looking at you message queues).

Example: Kubernetes Readiness Checks & Keepalives</h3>
Kubernetes (k8s) has a variety of health checks</a> with the one relevant to this example being the readiness probe. The readiness probe determines if a Pod is accessible through the Service interface, and on failure a given Pod will be removed from matching service endpoints. This is in theory useful for temporarily removing a given Pod from serving production traffic if it is suffering from a recoverable error that takes time to restart (eg. restarting a connection pool).
However, k8s does not kill all connections to the pod that fails a readiness check, it only removes it from the service abstraction which means new connections won't be made. The issue becomes apparent: connections made to the pod with a `Keep-Alive</code> set will remain active.`
This caused an outage at my current company. Latencies spiked for requests to keep-alive'd connections to Pods with failing readiness checks which meant the latency spiked all the way through the system despite numerous other Pods being healthy and available to respond. Had we had timeouts and retries in place, this issue would not have caused any problems. The requests would have timed out, retried on healthy pods, and the connection pooling would have cleaned up the connections eventually. A small increase in latency, but not a spike to 10s+.
`I love this example because it shows how complicated systems are today. The confluence of two good features, readiness checks in k8s and keep-alive connection pooling behavior, resulted in an outage. And it shows how unreasonably effective retries and timeouts are at preventing unexpected issues in a distributed system. The problem would have still occurred, but the effect would have been mitigated.`

Atomic Polling Intervals for Highly Concurrent Workloads

2024-05-24T00:00:00+00:00

One of the problems I've run into while developing Balter</a> is figuring out how often you are able to poll atomics without affecting data accuracy. For context, Balter is a distributed load testing framework, and under-the-hood it spawns a bunch of concurrent tasks which run the user's load test scenario, as well as a Scenario Coordinator task which polls the data collected via atomics. For the purposes of this post, we'll just consider Transactions Per Second (TPS).1</a>

If you aren't familiar with atomics, they are primitive types which provide synchronization across threads at a hardware level, and are a core building block for many concurrent applications. For instance, Rust provides an atomic version of the i32</code> integer type, AtomicI32</code></a>, which can be shared across threads and updated in a lock-free manner (avoiding the use of a Mutex</code>).

// Achieve the same goal, with the atomic
</span>// being much faster and lock-free.
</span>let</span> mutex_foo </span>= </span>Arc::new(Mutex::new(</span>10</span>));
</span>let</span> atomic_foo </span>= </span>Arc::new(AtomicI32::new(</span>10</span>));
</span></code></pre>
In Balter, after each transaction we increment an atomic, which is done using the .fetch_add()</code> method call and providing an Ordering</code></a>. The Ordering</code> is for specifying the kind of memory synchronization, and for this use-case we can use the most lenient option.</p>
some_transaction</span>().await;
</span>ATOMIC</span>.</span>fetch_add</span>(</span>1</span>, Ordering::Relaxed);
</span></code></pre>
On the Scenario Coordinator side we have a loop which runs on an interval and polls and resets the atomic:</p>
loop </span>{
</span>    </span>let</span> elapsed </span>=</span> interval.</span>next</span>().await;
</span>    </span>let</span> value </span>= </span>ATOMIC</span>.</span>swap</span>(</span>0</span>, Ordering::Relaxed);
</span>    </span>let</span> tps </span>=</span> value </span>as </span>f64 </span>/</span> elapsed;
</span>}
</span></code></pre>
The question is: how often can the Scenario Coordinator loop run?</p>
The reason we want to poll as often as possible is to speed up how fast Balter operates. Currently, Balter's control loops for error rate matching and latency matching are slow. By polling more often, the control loops would be run more often and Balter would be faster at converging to a specified error rate or latency. However, we don't want to provide bad</em> data to the control loops which could lead to instability.</p>
Atomic Experiments</h1>
In order to understand the constraints involved in the polling intervals for atomics, I set up a repository</a> with code which runs a simple test. It spawns 100 concurrent tasks, each of which runs a simple loop that waits for 10ms (with noise added)2</a></sup> and then increments two atomics: a baseline atomic, and a test atomic. The main function then polls the baseline atomic every 1s (to establish what we should</em> see), and polls the test atomic at varying intervals.</p>
// Task loop
</span>loop </span>{
</span>    tokio::sleep(delay_with_noise).await;
</span>    </span>TEST_ATOMIC</span>.</span>fetch_add</span>(</span>1</span>, Ordering::Relaxed);
</span>    </span>BASELINE_ATOMIC</span>.</span>fetch_add</span>(</span>1</span>, Ordering::Relaxed);
</span>}
</span>
</span>// Sampling loop
</span>tokio::join</span>! </span>{
</span>    async {
</span>        </span>loop </span>{
</span>            </span>// Always 1s
</span>            baseline_interval.</span>next</span>().await;
</span>            </span>let</span> baseline </span>= </span>BASELINE_ATOMIC</span>.</span>swap</span>(</span>0</span>, Ordering::Relaxed);
</span>        }
</span>    },
</span>    async {
</span>        </span>loop </span>{
</span>            </span>// Varies
</span>            test_interval.</span>next</span>().await;
</span>            </span>let</span> test </span>= </span>TEST_ATOMIC</span>.</span>swap</span>(</span>0</span>, Ordering::Relaxed);
</span>        }
</span>    }
</span>}
</span></code></pre>
With this setup we are able to measure the effects of atomic polling intervals by directly comparing the test intervals with a baseline which is being taken in parallel (to minimize external variability).</p>




    
</head>

Byron Wasti - Posts

On the Effectiveness of Retries and Timeouts in Distributed Systems

Atomic Polling Intervals for Highly Concurrent Workloads