Fixing the Invisible Monitor: Curses vs. SSH

My build swarm has a beautiful monitor. It’s written in Python using curses. It has scrolling logs, progress bars, color-coded status indicators, and realtime updates. It looks like the Matrix. I spent way too long on it.

Tuesday night. A friend is on Discord. He’s interested in the build swarm — the distributed Gentoo compilation cluster I’ve been rambling about for weeks. Five drones across two sites, 66 cores total when sweeper-Capella (the 5th drone, a repurposed workstation) is online. I want to show him the monitor in action.

“Hold on,” I say. “Let me pull it up.”

ssh commander@gateway "build-swarm watch"

The terminal clears. Hangs for a second. Exits.

Nothing. No error. No traceback. Just silence and a blinking cursor.

My friend, waiting on the screen share: “So… is it doing something?”

No. No it is not.

The Investigation

First instinct: SSH is broken. Checked the obvious stuff. The gateway is up, the command works locally, I can run other commands over SSH just fine. ssh commander@gateway "hostname" returns gateway-Altair immediately. The connection isn’t the problem.

Second thought: maybe the build-swarm binary isn’t in PATH over SSH. Non-interactive shells have minimal PATH. But no — ssh commander@gateway "which build-swarm" finds it. The binary is there.

Third attempt: ssh -t. The -t flag forces SSH to allocate a pseudo-TTY even when running a remote command. This is the standard fix for programs that need a terminal.

ssh -t commander@gateway "build-swarm watch"

And… it works. The full TUI comes up. Scrolling logs, progress bars, the whole thing. Beautiful.

But this means anyone who runs the command the “normal” way — without -t — gets nothing. No output, no error, no hint about what went wrong. That’s terrible.

The Swallowed Exception

I dug into the monitor code. Found it.

try:
    screen = curses.initscr()
    # ... TUI setup ...
except Exception:
    pass  # Past Me, you absolute genius

A bare except: pass. The most dangerous two words in Python. The curses library was throwing an error — specifically curses.error: setupterm: could not find terminal — and my code was catching it, saying “that’s fine,” and exiting cleanly. No logging. No fallback. No message to the user.

The Python curses library wraps the C ncurses library, and ncurses is picky about its environment. It needs to know what kind of terminal it’s talking to. Screen dimensions, color support, cursor capability — all of that comes from the TERM environment variable and the actual file descriptor connected to stdout.

When you run ssh user@host "command" (with the command in quotes), SSH doesn’t allocate a PTY. It just pipes stdin/stdout/stderr over the connection. Curses calls setupterm(), checks stdout, finds a pipe instead of a terminal, and throws an exception. My code catches that exception and does absolutely nothing.

The fix for the immediate problem was obvious: stop swallowing exceptions. But the real fix was bigger. I shouldn’t need -t for basic monitoring. The whole point of a CLI tool is that it works everywhere.

Building SimpleMonitor

I added a TTY detection check that runs before the monitor even tries to initialize curses:

import sys

def is_tty_compatible():
    # 1. Check if stdout is actually a terminal
    if not sys.stdout.isatty():
        return False

    # 2. Check for the --tui flag (user forced it)
    if '--tui' in sys.argv:
        return True

    # 3. Try a dry-run initialization
    try:
        import curses
        curses.setupterm()
        return True
    except:
        return False

Three layers of checking. First: is stdout a TTY at all? If not, don’t even try. Second: did the user explicitly ask for TUI mode with --tui? If so, trust them. Third: can curses actually initialize? Try it and see.

If any of this fails, the CLI launches SimpleMonitor instead of the full TUI.

The design process for SimpleMonitor was an exercise in figuring out what actually matters. The full TUI had: scrolling build logs, per-package compile progress, a live dependency graph, color-coded status for each drone, historical throughput charts, and animated spinners. It looked incredible. It also contained about 400 lines of curses rendering code.

For SimpleMonitor, I asked myself: what do I actually look at? When I check the monitor, what information am I after?

Two things. Which drones are building what. How many packages are in the queue.

That’s it. All the scrolling logs and dependency graphs and throughput charts? Nice to have. Never actually critical.

SimpleMonitor prints plain text, waits 5 seconds, sends an ANSI clear code (\033[2J\033[H), and prints again. No curses. No terminal capability detection. No PTY required. It works on literally anything that can display text.

═══ BUILD SWARM MONITOR ═══
2026-01-14 23:17:42

DRONES & ACTIVE BUILDS:
  drone-Izar     [16 cores]  BUILDING: app-shells/bash-5.2
  drone-Meridian [24 cores]  IDLE
  drone-Tau-Beta [ 8 cores]  BUILDING: net-misc/curl-8.5
  drone-Tarn     [14 cores]  BUILDING: dev-libs/openssl-3.2
  sweeper-Capella [ 4 cores]  BUILDING: app-misc/screen-4.9

QUEUE: 10 waiting | 4 building | 66 cores available
THROUGHPUT: 8.3 pkg/hr (last hour)

It’s ugly. It flashes when it refreshes. The alignment is manual and breaks if a package name is too long. But it works everywhere:

  • Inside a CI pipeline? Works.
  • Piped to a file with > monitor.log? Works.
  • Over a shaky SSH connection without -t? Works.
  • Through three layers of tmux and screen? Works.
  • On a terminal from 1985? Probably works.

The Before and After

The full curses TUI is still there. I didn’t delete 400 lines of carefully crafted rendering code. You can still get it with build-swarm watch --tui, and it’s still what I use when I’m sitting at my desk watching a big build roll through.

But the default is now SimpleMonitor. Safe. Boring. Functional.

The curses version needs: a real TTY, a terminal with color support, accurate TERM variable, properly allocated PTY if over SSH. Four requirements, any of which can silently fail.

The simple version needs: stdout. That’s it.

I spent two days building a beautiful TUI and one hour building the thing people actually use. The TUI was for me. The simple output is for everyone else. And honestly? I check the simple output more often than the TUI now. It loads faster, and I don’t have to think about whether my terminal session supports 256 colors.

Don’t let your vanity (cool UI) get in the way of utility (seeing the data). The best monitoring tool is the one that works when you need it, not the one that looks good in a screenshot.

Status: TUI is flashy, stdout is forever.