Akka remoting over SSL for secure inter-actor communication

Securing actor-based microservices in production is a recurring theme for teams building distributed systems across Sydney, Melbourne, and Brisbane. The Akka toolkit remains a popular choice for concurrency-heavy backends that power trading platforms, payment gateways, and logistics services in Australia. When these systems span multiple nodes, remoting becomes essential, and so does encrypting the traffic between actors.

Plain TCP connections between Akka nodes expose sensitive payloads to anyone with network access. For Australian organisations handling financial or personal data, the Privacy Act 1988 and the Notifiable Data Breaches scheme raise the stakes considerably. A misconfigured remoting layer can lead to disclosure incidents that must be reported to the Office of the Australian Information Commissioner. TLS, the modern successor to SSL, mitigates this risk by encrypting both the transport and the actor messages carried over it.

The configuration path is not always obvious, however. Akka remoting relies on a Netty SSL pipeline that needs keystores, trust managers, and cipher suites tuned to current standards. This walkthrough gathers the pieces I pieced together while hardening a Scala Akka cluster for a client in the healthcare sector, and it should save you a few afternoons of digging through source code.

A quick note before diving in: most of the examples assume a Java 11 or 17 runtime, which is standard in most Australian enterprise environments I have encountered. If you are still running Java 8, the cipher suite names will need to be adjusted accordingly.

Preparing the host environment and certificate store

Before touching any Akka settings, you need a working PKI foundation. I usually start with a local certificate authority generated on a build server, then issue per-node certificates signed by that CA. Australian teams operating under the Essential Eight framework from the Australian Signals Directorate often already have an internal CA, which makes this step much easier.

You will need keytool from the JDK, openssl for inspection, and a directory to hold the keystore and truststore files. Keep these files out of source control and load their passwords from a secret manager. Many Sydney-based teams use HashiCorp Vault or AWS Secrets Manager for this, which also helps when running on Kubernetes in the ap-southeast-2 region.

The keystore must contain a private key and a certificate for each Akka node, while the truststore holds the CA certificate that signed those node certificates. A common mistake is to bundle the CA certificate into the keystore and use it for both, which prevents proper chain validation later on.

Generating certificates with keytool

The quickest path to a working certificate is a few keytool commands. Start by creating the local CA keystore, then export its certificate, and finally issue a node certificate that the CA signs.

For the node certificate, you must include a Subject Alternative Name entry that matches the hostname the Akka remoting system will use to reach the node. This avoids the hostname verification failed error that Netty throws during the SSL handshake. If you are deploying to AWS in the Sydney region, the SAN should match the private DNS name assigned by the cluster orchestrator.

Some details on password handling and key algorithms are covered in the external reference material, which saved me from several rabbit holes. The article linked there walks through a slightly different scenario but the certificate generation commands are directly applicable.

One more thing: stick to RSA-2048 or ECDSA-P256 for the keys. Larger key sizes introduce noticeable handshake latency on the cross-AZ links that most Australian clusters rely on, and smaller sizes will fail compliance checks during a standard security audit.

Configuring application.conf for SSL

With the certificates ready, the next step is wiring them into Akka. The application.conf file accepts Netty SSL parameters under the akka.remote.netty.ssl namespace. You need to set enable-ssl = on, point to the keystore and truststore files, and supply the passwords through configuration or environment variables.

A few settings deserve special attention. random-number-generator should be set to a secure source rather than the default. hostname-verification should be enabled, which forces the TLS stack to compare the connection hostname against the certificate SAN. protocol should be set to TLSv1.3 wherever possible, with TLSv1.2 as a fallback for older nodes that might still be in the cluster.

Configuration key	Recommended value	Reason
enable-ssl	on	Activates the SSL pipeline
protocol	TLSv1.3	Modern cipher suite negotiation
hostname-verification	on	Prevents man-in-the-middle attacks
trust-requiring	on	Forces mutual authentication
enabled-cipher-suites	TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256	Strong AEAD algorithms
random-number-generator	AESCounterSecureRNG or platform default	Avoids weak entropy sources

This table reflects the values I settled on after benchmarking a four-node cluster in a Sydney data centre. If you are constrained to Java 8, swap the cipher suite list for TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 and similar TLS 1.2 names.

Enabling mutual TLS and trust managers

One-way TLS only proves the identity of the server to the client. For actor systems where every node talks to every other node, mutual TLS is far more appropriate. Akka remoting supports it through the trust-requiring setting, which makes the client side also present a certificate and verify the server certificate against the truststore.

The trust manager is what decides whether a presented certificate is acceptable. The default manager in Netty accepts any certificate that chains back to a trusted CA, but for production systems I prefer a custom manager that also checks the certificate Extended Key Usage extension. This stops a certificate issued for one purpose from being accepted for remoting.

Australian financial services firms often have additional checks encoded into their trust managers. For example, a payments processor in Melbourne I worked with required the certificate Organization field to match a known value, which prevented certificates from accidentally being reused across environments. If you are building under APRA CPS 234, similar controls are essentially mandatory for any system that processes regulated data.

Testing, troubleshooting, and going live

Once the configuration is in place, testing should be the next priority. The akka.remote actor system logs a lot of useful information at DEBUG level, including the cipher suite negotiated and the peer certificates. I usually start by pointing two nodes at each other on the same machine using 127.0.0.1, then expanding to the real network once the handshake succeeds.

A failure at the handshake stage often surfaces as a javax.net.ssl.SSLHandshakeException deep in the stack trace. The most common causes I have seen are: the truststore path is wrong, the password has a stray whitespace character, the certificate has expired, or the SAN does not match the connection hostname. Each of these produces a slightly different message, and the log lines are worth reading carefully rather than just searching for the exception class.

For ongoing operations, it pays to monitor certificate expiry. A small script that checks every node certificate against the notAfter field, and pages someone when the remaining validity drops below thirty days, prevents the kind of outage that hits the news when a major institution goes offline on a Sunday afternoon. I have seen Australian NOC teams adopt this pattern after a single painful incident, and it quickly becomes part of the standard operational runbook.

With the SSL pipeline working in development and staging, the same configuration can be promoted to production by replacing the test certificates with ones issued by your production CA. The Australian Cyber Security Centre publishes guidance on key management that aligns well with the approach described here, and it is worth a read before the first production deploy. If you are running Akka on Kubernetes, mount the keystore and truststore as secret volumes rather than baking them into the image, which keeps the certificates separate from the application code and makes rotation much simpler.

For more notes on Akka, Scala, and other tools I work with, the https://coding-journal.com/ has a backlog of troubleshooting posts that complement the configuration steps above. Share the corner cases you have hit when securing Akka remoting in a regulated environment so other practitioners can learn from your experience.