Observing Capistrano Deploys Through a Ruby Middleman

Capistrano can report a failed deployment, but its terminal output rarely tells another system how far a release has travelled. I wanted a Ruby middleman—a small wrapper, not the static-site gem—that observes Capistrano deploy progress without changing the deployment: it launches cap, mirrors the stream, exposes lifecycle events, and preserves the exit status.

The useful design is a thin wrapper plus explicit hooks. The wrapper handles terminal mechanics, Capistrano supplies structured milestones, and the caller receives readable logs with machine-readable progress. SSHKit output is presentation rather than a stable API, so a named phase is more honest than a guessed percentage.

Turning terminal output into useful progress

Capistrano's normal stream mixes SSHKit command output, coloured stage labels, warnings, and release paths. Those lines are useful to a person watching a terminal, but a dashboard cannot safely infer that fifty lines mean fifty per cent. I treat progress as a state machine: started, code updated, published, and finished, with failures recorded separately.

Each event should carry a deployment identifier, release identifier, revision where available, and a UTC timestamp. The revision links the observation to a checkable artefact; the deployment identifier prevents overlapping staging releases from sharing one timeline. A heartbeat can prove the process is alive, but it should never advance the release percentage.

For an Australian team, this model keeps regional differences outside the protocol. Rails applications hosted through AWS Sydney (ap-southeast-2) or Melbourne (ap-southeast-3) can emit the same milestones as staging elsewhere while keeping compute in an Australian region. It gives teams a clear input for data-residency checks and a consistent release feed across the Australian market.

Building a transparent Ruby command wrapper

The wrapper must not swallow output that Capistrano expects to see. A pipe can alter buffering and colour detection, while a pseudo-terminal preserves the interactive behaviour of SSHKit and common Capistrano tasks. Save this minimal Ruby script as bin/deploy_observer:

require "json"
require "pty"

buffer = +""
command = ARGV.empty? ? %w[cap production deploy] : ["cap", *ARGV]
deployment_id = ENV["DEPLOY_ID"] || Process.pid.to_s
status = 1
event = /DEPLOY_EVENT\s+(\{.*\})/

PTY.spawn({ "DEPLOY_OBSERVER" => "1", "DEPLOY_ID" => deployment_id }, *command) do |reader, _writer, pid|
  begin
    loop do
      chunk = reader.readpartial(4_096)
      $stdout.write(chunk)
      $stdout.flush
      buffer << chunk

      while (line = buffer.slice!(/.*?\n/m))
        clean = line.gsub(/\e\[[0-9;]*m/, "")
        match = clean.match(event)
        warn "OBSERVED #{JSON.generate(JSON.parse(match[1]))}" if match
      end
    end
  rescue EOFError
    # The child closed the pseudo-terminal.
  ensure
    _, process_status = Process.wait2(pid)
    status = process_status.exitstatus || (128 + process_status.termsig.to_i)
  end
end

exit(status)

readpartial returns as soon as bytes arrive, so a slow upload does not appear frozen. The observer writes each chunk to standard output and keeps a separate buffer for parsing. It strips ANSI colour codes from the parsing copy only. DEPLOY_EVENT gives the parser a precise boundary even when Capistrano prints formatting around a task.

I reproduced this in an isolated Linux environment before touching deployment code; the virtual machine setup notes cover the base image and SSH access I used. That let me test terminal width, forced colour, and an interrupted connection without risking a release. The wrapper exits with Capistrano's status, so an interrupted child can return 130 and a failed rollout stays non-zero.

Emitting deployment events from Capistrano

The wrapper supplies the environment; Capistrano hooks then write a small JSON object to a reserved line on standard error:

require "json"
require "time"

emit = lambda do |name, details = {}|
  next unless ENV["DEPLOY_OBSERVER"] == "1"

  payload = {
    deployment: ENV.fetch("DEPLOY_ID"),
    event: name,
    at: Time.now.utc.iso8601
  }.merge(details)

  warn "DEPLOY_EVENT #{JSON.generate(payload)}"
end

namespace :observer do
  task(:started) { emit.call("started") }

  task(:code_updated) do
    emit.call("code_updated", { release: release_path, revision: fetch(:current_revision) })
  end

  task(:published) do
    emit.call("published", { release: release_path })
  end

  task(:finished) { emit.call("finished") }
end

after :started, "observer:started"
after :updated, "observer:code_updated"
after :published, "observer:published"
after :finished, "observer:finished"

The wrapper searches each completed line for the marker and parses the JSON after it. A read can end halfway through an event, so the parser keeps the remainder for the next chunk. These records can sit beside the transcript or feed a local collector; they contain release metadata, not command output or environment variables.

The semantic rule is strict: finished means the hook ran, not that deployment succeeded. Success requires that event and a zero process status. Before published, the old release remains active; afterwards, a failed restart may require rollback. Keeping code_updated and published separate prevents copied files from masking a broken boot.

Comparing observation strategies

Capistrano can be observed at several layers, and the right choice depends on whether terminal fidelity or structured state matters more:

Approach	Live output	Structured milestones	Changes deployment	Best fit
cap ... | tee deploy.log	Yes	No	None	Local troubleshooting
Ruby PTY middleman	Yes	With hooks	Small wrapper	Terminals and CI
Capistrano hooks only	Partial	Yes	Adds DSL	Internal dashboards
CI log scraper	Yes	Heuristic	None	Retrospective analysis

Keep the first version deliberately small:

• Give every run a unique deployment identifier and attach it to each event.
• Record UTC timestamps, phase names, release identifiers, and the final exit status.
• Preserve raw Capistrano output beside parsed events for incident review.
• Mark completion only when finished is present and the process exits successfully.

Test the failure paths before adding another metric:

• Split a JSON event across two reads and verify the carry buffer.
• Run with NO_COLOR=1 and a narrow terminal to expose parser assumptions.
• Interrupt the wrapper and confirm the child receives the signal.
• Simulate an SSH timeout and check that unknown states are not shown as successful.

For Australian release teams, I would begin with the Ruby middleman and hooks, then add only the integration the team actually uses. This keeps deployment approval separate from monitoring, which helps during a change freeze or a busy arvo when several people need the same live status. It also leaves room for local tooling without requiring a hosted service in the Australian market.

Handling signals, disconnects, and noisy output

A pseudo-terminal introduces signal responsibilities. In the production wrapper, pressing Ctrl-C should reach the child, not merely break the read loop. The wrapper catches INT and TERM, forwards them to the Capistrano process group, waits for SSHKit to close its sessions, and restores the terminal before exiting. If the child ignores the signal, a short grace period followed by KILL prevents an orphaned deployment from continuing invisibly.

Network failures need similar care. Capistrano may retry a command or abort the flow, and repeated stage lines can appear in the transcript. Hooks should be safe to run more than once, while the observer uses the deployment identifier to deduplicate repeated events and records retries as separate attempts. A disconnected SSH session is an unknown state, not proof that the remote upload stopped.

Timestamp everything in UTC and render local time only at the edge. Brisbane remains on AEST throughout the year while Sydney and Melbourne shift to AEDT, so a log that stores an ambiguous local clock value becomes difficult to interpret twice a year. UTC also makes comparison with cloud audit trails straightforward.

Benchmarking the observer is another form of troubleshooting. The Android emulator custom ROM walkthrough shows how a controlled image makes performance comparisons repeatable; the same discipline applies here. Test with slow output, colour disabled, a split JSON line, and an interrupted connection, then measure the wrapper's overhead. A few milliseconds of local parsing is a fair price for a timeline that does not invent progress.

Putting the observer into daily releases

The result should feel boring in use. An operator still sees the familiar Capistrano stream, a script receives named phases, and CI can fail on the same non-zero status that would fail a shell command. The observer adds visibility rather than a second deployment engine, so there is no hidden queue, background worker, or second source of truth.

A practical rollout starts in staging with the same command used in production. Run the wrapper beside an ordinary deploy, compare the event sequence with the transcript, and confirm that retries, colour, and terminal width do not change the result. Keep the event records for a short period, scrub them if release metadata is sensitive, and add remote shipping only after local verification.

Wire the observer into one staging deployment this week, validate the exit status and release identifier against the normal transcript, then run DEPLOY_OBSERVER=1 bundle exec ruby bin/deploy_observer production deploy and retain the raw transcript beside the parsed events.