Configuring Akka HTTP client with retry and circuit breaker logic

Configuring an Akka HTTP client with retries and a circuit breaker is less about adding two libraries than defining which failures are temporary, how long the caller may keep trying, and when an unhealthy dependency should leave the request path. Retries handle narrow, transient faults. A circuit breaker protects the service and connection pool when the fault persists.

This article uses Scala 2.13, Akka Classic and Akka HTTP 10.x, a combination still found in established Scala services. The policy transfers cleanly to newer Akka or Apache Pekko clients, although package names and circuit-breaker APIs differ. Keep transport policy inside a client wrapper; domain code should receive meaningful domain errors rather than inspect HTTP details.

A common mistake is treating every non-2xx response as retryable. A missing API key will not heal after 200 milliseconds, while a 503 during a rolling deployment might. Another mistake is opening a breaker after one timeout, turning a brief network wobble into a self-inflicted outage. Attempt limits, deadlines and breaker thresholds must share one failure budget.

Australian deployments make that budget harder to assume. A Melbourne office may call an AWS workload in the Sydney region, while a phone on 4G or 5G sees different latency from an NBN connection. Local support contracts often follow Australian Eastern business hours, including daylight-saving changes, so incident logs need UTC timestamps. These conditions matter during the arvo rush, when call volume and operational pressure can rise together.

Draw a line around transient failures

Start by listing failures, not just statuses. Connection loss, DNS lookup failure, connect or read timeouts, and selected 502, 503 or 504 responses are retry candidates. Authentication failures, malformed requests, validation errors and most 4xx responses are terminal. A 429 needs separate handling because a useful retry depends on the Retry-After value and the caller’s remaining deadline.

The HTTP method changes the answer. GET, HEAD, PUT and DELETE are normally safe to repeat when the downstream implementation is genuinely idempotent. POST is not. If a payment or order endpoint supports an idempotency key, keep that key stable across attempts and let the server return the original result. A fresh key on every try turns a useful retry into a duplicate charge.

I keep related implementation notes on Mike Patt’s site, but the useful rule is simple: use one transient classification across configuration, metrics and tests. Keep domain-level command retries separate, because generic HTTP retries solve transport faults while business workflows need their own conflict handling.

Set bounded retries in Akka HTTP

A small Akka HTTP client block can express the transport policy in application.conf:

akka.http.client {
  connect-timeout = 3s
  idle-timeout = 30s

  retry {
    max-retries = 2
    min-backoff = 250ms
    max-backoff = 2s
    retryable-status-codes = [502, 503, 504]
    retryable-exceptions = [java.io.IOException, java.util.concurrent.TimeoutException]
  }
}

Here, two retries permit at most three calls. Akka HTTP still applies its idempotency rules, so this block does not make an unsafe POST safe. The first backoff is 250 milliseconds and later waits grow towards the two-second ceiling. Keep the count low: every retry occupies a request slot and extends latency. If many callers arrive together, two retries can turn one upstream outage into six requests, so a bounded connection pool and caller-side rate limits still matter.

Set one deadline around the whole logical operation, not a fresh timeout for every socket read. A caller with five seconds should not spend 250 milliseconds and then 500 milliseconds in backoff before discovering it has no time for a useful response. Log the remaining deadline when an attempt starts and abort earlier when the target cannot be met. Exponential growth without a total deadline is only delayed failure.

Keep retries and circuit breaking separate

Retries and breakers answer different questions. A retry asks whether this particular fault may clear shortly; a breaker asks whether the dependency has failed too consistently to receive more traffic. The breaker should sit outside the retry operation, so repeated attempts do not inflate its failure count and a logical request contributes one final outcome.

Question	Retry policy	Circuit breaker
Main purpose	Recover one transient fault	Protect a dependency from sustained failure
Trigger	Timeout, selected status or exception	Failure count or rate within a window
Scope	Attempts belonging to one request	All calls to a named upstream
Open state	Not applicable	Fail fast until the cooldown ends
Recovery	A later request may try again	Half-open probes test the dependency

This simplified Akka Classic example uses the older akka.pattern.CircuitBreaker API. Newer stacks can use an equivalent adapter while keeping the same ordering and states:

private val breaker = CircuitBreaker(
  maxFailures = 5,
  duringTimeout = 30.seconds,
  failureCounter = new AtomicInteger(0)
)

breaker.withCallBreaker {
  http.singleRequest(request).flatMap { response =>
    if (response.status.isSuccess()) Future.successful(response)
    else {
      response.entity.discardBytes()
      Future.failed(new UpstreamFailure(response.status))
    }
  }
}

Convert a failed status into a failed future before the breaker evaluates the result; otherwise a run of 503 responses looks healthy. Use a separate breaker per upstream, and expose its closed, open and half-open states as metrics. A rate-based window is often easier to tune across several service instances than a consecutive-failure counter, but neither is useful without the request timeout that bounds each call.

Build a client boundary that can be tested

Put upstream calls behind a small class that owns the base URI, authentication headers, correlation ID, retry configuration and breaker. Accept a HttpRequest => Future[HttpResponse] function as its transport dependency. Production passes http.singleRequest; tests return scripted futures without opening sockets or depending on a live provider.

That seam makes failure paths deterministic. Script a connection timeout, then a success; return a 503 twice and then a 200; return a 401 once and confirm there is no second call. Then force enough failures to open the breaker, advance to half-open and return one successful probe. Also assert the whole-operation deadline, not just the number of requests, because a correct retry count can still exceed the caller’s budget.

The same controlled-environment principle appears in performance work: custom Android emulator setup keeps emulator changes measurable rather than guessing at their effect. For an HTTP client, inject the delay, status sequence and clock so each test changes one variable and records the resulting attempt count, elapsed time and final outcome.

Keep fixtures close to the test that uses them. A single shared happy-path stub often hides the exact problem that resilience code introduces: duplicated writes, an open breaker used by another suite, or a retry that survives after the caller’s deadline. Tests should finish quickly and leave no background timers behind.

Measure behaviour under Australian network conditions

Measure logical requests and physical attempts separately. A dashboard that reports only final successes can look healthy while each request has become three times more expensive. Track retries by reason, final status, breaker state, pool saturation and end-to-end latency. Label the dependency and include the upstream request ID so a support engineer can match an Akka failure to a provider trace.

Exercise the client from both Sydney and Melbourne, and include a deliberately uneven mobile profile rather than one smooth latency line. NBN traffic, 4G and 5G handovers, DNS delay and a short packet-loss window can expose retry storms that an idealised local test misses. Against an endpoint in AWS’s Sydney region, ap-southeast-2, record connection time separately from response time; the former points towards networking, while the latter may point towards the service.

Do not tune thresholds from a total-outage test alone. That encourages tiny breaker windows and aggressive retries. Use a baseline of ordinary latency, then test partial degradation, recovery and normal traffic returning together. Australian on-call handovers should include both UTC and local time because AEDT changes can make otherwise identical incident windows look shifted.

Give the policy an owner and a safe default

Document what each threshold means and who can change it. A useful runbook explains the retry count, maximum backoff, whole-call deadline, breaker cooldown and the action taken when the breaker is open. Put those values under version control, deploy them like code and show the active configuration in a health response that is not publicly accessible.

Canary the wrapper against a small slice of live traffic, comparing final error rate and latency with the previous client. If the breaker opens, retain that state rather than resetting it on every deployment; otherwise a bad release can look recovered simply because the process restarted. Include client policy version in structured logs so old and new behaviour can be separated during the canary.

Make the next change concrete: put the wrapper behind one upstream, run the scripted failures in staging, and publish attempt count, breaker state and final latency on the service dashboard before enabling it for live Australian traffic.