Why Akka Cluster Members Vanish and How to Bring Them Back Cleanly

Akka powers a lot of serious distributed infrastructure across Sydney and Melbourne, from Atlassian's collaboration tooling to the back-office systems at big banks. When one node drops out unexpectedly, the cluster's reaction can range from graceful to chaotic. Untangling the difference between a member that has left voluntarily, one that has been forcibly removed, and one that is just unreachable is the first step towards keeping your service available.

I've been on the receiving end of cluster alarms at 2am AEDT, scrambling to figure out whether the seed node was down, the network dropped out, or whether a JVM had truly crashed. The pattern is familiar to anyone who has run a JVM-based cluster in anger: a member goes quiet, watchers panic, the downing machinery takes over, and then the real work begins - migrating singletons, rebalancing shards, and replaying events that are now scattered across surviving nodes.

This piece walks through what actually happens when an Akka cluster member fails, why the automatic node removal sometimes gets it wrong, and the practical options you have for shaping that behaviour so your service stays online and your operators stay sane.

Membership states beyond simply up or down

An Akka cluster member is never just "online" or "offline". The cluster tracks membership through a gossip-based protocol that propagates state between nodes, and each peer can be in any of several recognised states. A node starts in Joining, transitions to Up once the leader has acknowledged it, may move to Leaving and then Exiting if it shuts down cleanly, and finally lands in Removed. Two additional states exist for a node that the cluster suspects of being in trouble: Unreachable and Down.

The cluster leader is responsible for orchestrating state changes, and it works from the seed-node list to bootstrap the gossip. This is why seed nodes are a critical weak point for many Australian deployments running across regions - if the primary seed lives in an AWS Sydney availability zone and that AZ has a hiccup, the gossip convergence fails before anything else. Member unreachable events get triggered when heartbeats cease, and the leader decides what to do based on the downing provider you have configured.

Understanding this state machine is essential. If you treat every member's disappearance as a sudden outage, you'll over-react. If you treat it as routine, you'll miss genuine JVM crashes. The sweet spot is reading the cluster's own state, listening for the MemberEvent stream, and reacting with awareness of which transition has actually occurred.

How the failure detector pulls the trigger

Akka uses the Phi Accrual Failure Detector by default. Rather than declaring a node dead after a single missed heartbeat, it maintains a sliding window of heartbeat arrival times and computes a phi value that crosses a configurable threshold when something looks genuinely wrong. The lower the configured threshold, the more aggressive the detection - and the more false positives you will see.

Network blips on the NBN, jitter across a corporate VPN, or a temporary Azure blob-storage outage inside another workload can all produce heartbeats that arrive a few hundred milliseconds late. With a tight threshold, those interruptions get flagged as failure. With a relaxed one, real crashes get missed until the JVM has been wedged for a minute or more. There is no universally correct value. I've found a phiThreshold of around 8 to 12 suits most workloads running between Sydney and ap-southeast-2, but the right number depends on the round-trip times your specific network produces.

When phi crosses the threshold, the node is marked Unreachable. From there, the downing decision takes over, and that's where many deployments get into trouble.

The downing decision can break your cluster

The default downing provider in classic Akka is auto-down. As soon as a configurable number of nodes cannot reach a specific peer, the cluster marks it Down and the leader arranges for its removal. In practice, auto-down is dangerous for any production workload bigger than a handful of nodes. A network partition that splits your cluster 3-and-3 in Sydney can lead both halves to down the other simultaneously, and your system ends up running two competing incarnations of itself.

Lightbend's Split Brain Resolver, now part of the commercial offerings, was built to fix this. It provides several strategies for making safer downing decisions. Reachability decisions use cluster reachability information, static quorums rely on a fixed-size denominator, and keep-majority styles preserve the largest contiguous partition. Lease-based downing goes further, using an external lock service - which, in Australia, often means something like a Hazelcast cluster on-prem or an etcd cluster running in another region's VPC.

Pick a strategy before the outage, not during it. The on-call engineer at 3am AEST is not the right person to be negotiating split-brain semantics.

What happens after a node leaves the cluster

Once a downing decision has been made and the leader removes the affected member, the cluster reshuffles. Cluster singletons - those actors that must only ever exist once - get moved to a surviving node, and any subscribers to ClusterEvent.MemberRemoved get notified. If you are using cluster sharding, the shards that lived on the removed node are redistributed, which can take a while and produces a flurry of ShardHomeAllocated events during the migration.

Event-sourced actors that lived on the failed node replay their journal on the new host. This is why persistence matters during recovery: the new host needs access to the journal, usually a Cassandra cluster or PostgreSQL with proper indexes. Any in-flight messages to the old host vanish into the void. Akka does not retransmit, so anything important needs an at-least-once delivery wrapper or a durable queue in front of the actor.

Recovery is where most of the user-facing pain actually lives. From outside, the cluster looks like it came back together quickly, but underneath the read load on the journal spikes, the new host spends several seconds catching up, and downstream consumers may need to reconcile their state against a database that has moved on during the outage.

Common pitfalls in Aussie deployments

Australian infrastructure has its own quirks. The Sydney-Melbourne corridor is well-served by cloud regions, but anything further afield - Perth, Hobart, Cairns - tends to ride over longer fibre paths with measurable latency. For teams running branches or point-of-sale systems in regional Queensland or Western Australia, that latency translates into heartbeat jitter that confuses the failure detector.

Australian banks and fintechs - ANZ, CBA, the various Buy Now Pay Later providers in Melbourne's fintech lane - have been early Scala and Akka adopters for exactly this reason. They tend to use quite conservative downing providers because the cost of a split-brain in a payment context is unforgivable. By contrast, smaller SaaS shops often run auto-down for convenience and only fix the configuration after they've been bitten.

On-call culture is another factor worth naming. The whole country occasionally treats distributed-systems outages with a "no worries, will sort it tomorrow arvo" attitude, but Akka issues don't wait for daylight. Documented procedures, runbooks tuned to your specific downing strategy, and an on-call handover that respects AEDT/AEST transitions all matter. If your only seasoned cluster operator is in Perth and the rest of the team is in Brisbane, you are three hours adrift from a clean handoff every evening.

Observability and the boring work of recovery

You will not debug a cluster member-removal incident without solid logs. Kamon, Lightbend Telemetry, and plain structured logging with Logback all help, but the specific bits you want to capture are the MemberEvent stream, the phi values at the time of suspicion, and the timing of any singleton migrations. When the cluster comes back together, the self address and the membership data centre values are not secrets; persist them somewhere readable.

If a node genuinely crashed rather than being partitioned, the surviving cluster will eventually Down it, and you will need to confirm before bringing a replacement online. Memory dumps, heap snapshots, and the JVM's own error log are the next investigation steps - none of which the cluster itself can give you. Plan for that handoff between the Akka world and the OS world before something blows up.

Downing strategy	Best for	Failure mode	Quorum source
Auto-down	Tiny clusters and dev	Partition doubles the carnage	Count of suspects
SBR with static quorum	Fixed-size production clusters	Hard to scale membership	Hard-coded size
SBR with keep-majority	Even-odd balanced workloads	Hard threshold for three-node setups	Reachability plus size
SBR with keep-oldest	Clear ownership semantics	Stale leaders reign during partial outages	Oldest node priority
Lease-based	Critical singletons and payments	External lease outage blocks removal	External lock service

That table summarises the trade-offs. Most Australian teams running mid-sized clusters will pick keep-majority or fall back to keep-oldest; payment systems usually want lease-based.

Practical recommendations before the next outage

• Configure phiThreshold to match your measured network jitter rather than the default.
• Replace auto-down with Split Brain Resolver or another explicit strategy before you ship to production.
• Size your seed-node list at three, hosted across multiple availability zones in ap-southeast-2.
• Persist the journal on storage that survives a node loss - PostgreSQL or Cassandra, not local disk.
• Subscribe to ClusterEvent.MemberRemoved and react explicitly in your own supervisors.
• Add Kamon or Lightbend Telemetry before you need it; dashboards do not appear during incidents.
• Practice a node-loss scenario quarterly; muscle memory beats clever design at 3am AEDT.

If you want a quieter kind of bug story after all this, I previously wrote about Cocoa event tracking - completely different stack, but the same feeling of untangling someone else's choices.

Subscribe for the next troubleshooting deep-dive, drop your war stories in the comments, and pass this on to the engineer in your team who still thinks auto-down is fine.