Debugging a Capistrano deploy that hangs on git checkout

There is a particular kind of dread that comes with watching a deploy freeze on the git checkout line. The output stops, the spinner keeps turning, and somewhere in another tab your monitoring dashboard starts to blink. A Capistrano run that hangs during version control operations is one of the most common stumbling blocks for teams using the tool, and it rarely fails loudly enough to give you a clear clue.

The most frequent culprits are well known: SSH host verification prompts, credential prompts, slow connections to remote git servers, and repositories that have grown large enough to make every clone a chore. Some of these problems are amplified by the distance data has to travel, and others are amplified by tools that try to be too helpful.

In Australia, the geographic spread of the country plays a noticeable role. A deployment server hosted in a Sydney data centre talking to a git provider in Frankfurt or Virginia will see latency that simply does not exist for teams based in Europe. Throw in a flaky NBN link or a corporate VPN that routes traffic through Singapore, and a fast git operation can suddenly take many minutes.

This post walks through the usual suspects, how to reproduce them locally, and the small set of configuration tweaks that resolve most hangs. Whether you are pushing code from a Melbourne office or a regional town in Queensland, the diagnosis process is the same, even if the underlying cause varies.

Network latency and the great Australian distance

Latency hides inside git operations more often than people realise. Git does not download the whole repository every time, but a git fetch followed by a git checkout on a freshly provisioned server still requires round-trips to enumerate refs and download pack files. When each round-trip adds 200 milliseconds, the time adds up quickly.

In practice, Australian teams often deploy to servers in Sydney or Melbourne but host their primary git repository overseas, on platforms with no local mirror. If your deploy script also performs a git submodule update --init --recursive, the cost multiplies across every submodule. The first thing to check is whether the hang is actually a network problem.

You can confirm this by running time git clone <repo> directly on the target server, bypassing Capistrano entirely. If the clone takes an unusually long time, you are looking at a bandwidth or latency issue rather than a Capistrano bug. In that case, consider mirroring the repository to a closer host, or use a git smart HTTP proxy.

Another Australian-specific issue is the NBN connection used by small studios or remote workers. If the deploy server sits in a home office in Perth or Adelaide and uses a residential NBN plan with contended upload speeds, the bottleneck may be the local link. Running deploys from a data centre VM instead of a local workstation can completely change the timing.

Run the deploy in verbose mode first

Before you change anything, you need to know exactly where the hang happens. Capistrano passes through git output by default, but it is worth forcing the underlying commands to be louder. Set the GIT_SSH_COMMAND environment variable to include verbose SSH logging, and run the deploy with cap deploy -vvv to see every command as it executes.

The verbose flag reveals whether git is waiting for a passphrase, waiting for a host key confirmation, or simply sitting idle on a slow socket. In one project I worked on, the deploy appeared to hang for two minutes, but the verbose output showed git retrying an HTTPS connection because the certificate bundle on the server was out of date. A single update-ca-certificates fixed the hang.

If you suspect a credential prompt, set GIT_ASKPASS to a script that returns a stored token, or use SSH keys with a passphrase managed by ssh-agent. Capistrano does not run an interactive terminal by default, so any prompt will appear to be a hang rather than an error.

For a quick sanity check, you can also try the same git fetch and git checkout commands manually on the deploy target, using the same user account that Capistrano uses. This separates Capistrano-specific issues from general git-on-the-server issues. A note from the author of The Coding Journal covers this exact workflow in more detail.

Handle SSH host keys and multiple identities

One of the classic causes of a Capistrano deploy hanging on git checkout is the SSH host key prompt. When Capistrano first connects to your git host from a new server, SSH asks for confirmation that you trust the host fingerprint. Because Capistrano does not allocate a TTY for the git command, the prompt never appears and the process just waits until it times out.

The fix is to pre-populate ~/.ssh/known_hosts on the deploy user. You can do this by running ssh-keyscan github.com >> ~/.ssh/known_hosts (or your git host) as the deploy user before the first deploy. Many Australian teams keep this in their server bootstrap script, which is especially useful if you spin up new environments in a Sydney region regularly.

A related issue is having multiple SSH keys. If your personal key, a deploy key, and a backup key all live in ~/.ssh/, SSH may offer the wrong one first and trigger authentication failures that look like hangs. Configuring ~/.ssh/config with a Host block that explicitly maps your git host to the right IdentityFile makes the selection deterministic.

You can also try setting ForwardAgent yes in your SSH config for the connection to the deploy server, then adding your local key with ssh-add. This lets the deploy server use your local agent to authenticate, which avoids storing keys on the server at all. Under Australian privacy guidance, keeping keys off production servers aligns well with the Australian Privacy Principles around data minimisation, especially for teams handling personal information under the Privacy Act 1988.

Tame large repositories with shallow clones

If your repository has grown over the years, the git clone step in Capistrano may simply be doing too much work. Capistrano by default does a full clone, which includes the entire history, all branches, and all tags. For a multi-gigabyte repo, this is wasteful when you only need the deploy ref.

The simplest improvement is to set :git_shallow_clone to true in your config/deploy.rb. This passes --depth 1 to the git clone command and dramatically reduces the amount of data transferred. For Australian teams pushing code from interstate or from regional areas, this single change can cut deploy time by more than half.

If you use submodules, the gain is even larger. A recursive submodule update pulls every subproject's history, and a few large submodules can dominate the deploy time. Combine shallow clones with :git_enable_submodules set to true and consider adding --depth 1 to the submodule update in a custom task.

For very large monorepos, you may also want to enable partial clones with --filter=blob:none, which omits blob objects from the initial fetch. Git will then fetch individual files on demand. This is a more advanced option and requires git 2.21 or later on the server, but for repos over several gigabytes it can be transformative.

After a long debugging session, it is worth stepping away from the terminal for a bit. Some folks unwind by browsing everyday lifestyle reads before returning to the problem with fresh eyes.

Quick reference for common fixes

Here is a short comparison of the symptoms and remedies discussed above. Use it as a checklist when a deploy next hangs on git checkout.

Symptom	Likely cause	First thing to try
Long pause, no output	Network latency to git host	Run time git clone directly on the server
Hangs then times out	SSH host key prompt	Pre-populate known_hosts with ssh-keyscan
Hangs then auth fails	Multiple SSH keys, wrong one offered	Add a Host block in ~/.ssh/config
Deploy takes 10+ minutes	Full clone of a large repo	Set :git_shallow_clone to true
Submodules slow it down	Recursive submodule update	Combine shallow clone with --depth 1 for submodules
Certificate errors	Outdated CA bundle	Run update-ca-certificates on the server
Works manually, fails via Capistrano	Missing TTY for credential prompt	Set GIT_ASKPASS to a non-interactive script

If none of these resolves the issue, capture a full trace with strace -f -e trace=network git fetch origin and inspect the syscalls. The trace will show whether the process is waiting on a read, a connect, or a DNS lookup, which usually pinpoints the remaining problem. Most hangs in Australian deployments end up being one of the items above, but a strace is a reliable last resort when nothing else gives a clue.

Subscribe to the RSS feed or share your own debugging war stories in the comments below — the strangest deploy hang you have ever hunted down is always a good read.