Learning Rust by Building herdr, a tmux-Aware Web SSH Terminal

I wanted a way to reach my tmux sessions from any browser without carrying a laptop, and I wanted to finally write real async Rust instead of reading about it. Those two wants became herdr: a single Rust binary that serves a web terminal, opens SSH connections server-side, and treats tmux sessions as a first-class herd — list them, attach, create, kill, and when you detach you land back in the herd instead of a dead page. It is live at shell.binary.ovh, built in one long autonomous session, spec-first, with security as the top requirement.

This post is the Rust I learned along the way, with the scars attached.

The shape of the thing

browser ──wss──▶ Caddy ──ws──▶ herdr (127.0.0.1:8710) ──ssh──▶ sshd
xterm.js         TLS           axum + russh                    tmux

One WebSocket = one SSH session. Text frames carry JSON control messages (connect, hostkey_answer, attach, resize, tmux_kill…), binary frames are raw terminal bytes. The browser never talks SSH; the server never stores SSH credentials. The whole backend is about 1,400 lines across six modules, plus a no-build frontend (vendored xterm.js, plain JS, strict CSP with zero inline script).

Lesson 1: porting threads to async is mostly deleting code

herdr's SSH layer is adapted from rterm-ssh, the Rust core of my Android terminal. That code bridges russh's async world to a synchronous JNI world with a thread, a Mutex, and a Condvar — the host-key prompt literally parks a thread until the UI answers.

Inside tokio none of that machinery survives. The same feature — "pause the SSH handshake until a human approves the fingerprint" — became a message on a channel:

pub struct HostKeyPrompt {
    pub fp: String,
    pub answer: oneshot::Sender<bool>,
}

// inside check_server_key, on first contact:
let (tx, rx) = oneshot::channel();
self.prompt_tx.send(HostKeyPrompt { fp, answer: tx }).await?;
Ok(rx.await.unwrap_or(false))

The handler sends the fingerprint plus a oneshot::Sender — a reply envelope — up to the WebSocket task, which forwards it to the browser and later fires answer.send(true/false). The SSH handshake just .awaits. No thread, no Condvar, no poisoned-lock paranoia. If the browser vanishes, the oneshot drops and rx.await yields Err, which unwrap_or(false) turns into "reject" — failure-closed by construction.

Lesson 2: select! with things that only sometimes exist

The core of ws.rs is one loop that must simultaneously watch: bytes from the SSH PTY, messages from the browser, and an idle-timeout tick. Except there isn't always a PTY — you can be in the herd view with no session attached. Rust's answer is an Option<Channel> and a guarded select arm:

tokio::select! {
    msg = pty.as_mut().unwrap().wait(), if pty.is_some() => { … }
    sock_msg = sock.recv() => { … }
    _ = interval.tick() => { … }
}

The if pty.is_some() guard disables the arm entirely when there's nothing to wait on, which makes the unwrap() safe. When the PTY closes (tmux detach or kill), the arm sets pty = None, tells the browser detached, and sends a fresh session list — that's the whole "detach returns you to the herd" feature, three lines in a select arm.

Lesson 3: the type system will find your laziness

The only compile error in the entire backend was this innocent line:

russh_keys::decode_secret_key(&pem, passphrase.as_deref())

passphrase is Option<Zeroizing<String>> — Zeroizing wipes the memory on drop, which is why it's there. But Zeroizing<String> derefs to String, so as_deref() produces Option<&String>, and the function wants Option<&str>. One more hop:

passphrase.as_deref().map(String::as_str)

Thirty seconds of confusion, but it's a perfect miniature of Rust: the wrapper that gives you the security guarantee (zeroize-on-drop) is visible in the types, and the compiler makes you acknowledge every layer you added.

Lesson 4: RAII for things that aren't memory

herdr caps concurrent SSH sessions. The naive version increments a counter on connect and decrements on disconnect — and leaks a slot the first time any code path returns early. The Rust idiom is to make the slot a value whose destructor gives it back:

struct Slot(Arc<App>);
impl Drop for Slot {
    fn drop(&mut self) {
        self.0.active.fetch_sub(1, Ordering::SeqCst);
    }
}

Acquire it at the top of the handler; every exit — clean close, error, panic unwind, the 60-second connect deadline — runs Drop and releases the slot. I stopped thinking about cleanup entirely, which is the point.

Security review: the two bugs worth blogging

Before deploying I did an adversarial pass over my own code (the automated review tool wanted a git remote I don't have, so: manual). Two findings were real.

X-Forwarded-For, first vs last. My login rate limiter keyed on the client IP from X-Forwarded-For — the first entry. But an attacker sets that header; Caddy appends the real client IP to whatever arrives. So evil1, evil2, …, real-ip — taking the first entry lets an attacker rotate fake IPs and never hit the limit. The fix is one method call, split(',').next_back(), and a test that rotates spoofed first entries and asserts the 6th attempt still gets 429 Too Many Requests. It does.

⚠️ If you rate-limit behind any reverse proxy, know exactly which end of the XFF list your proxy owns. Caddy appends; some proxies replace; getting it wrong inverts your security control.

Unbounded exec. Listing tmux sessions runs tmux list-sessions over SSH and awaits the output — inside the same select loop that services the user. A hostile or hung remote command could wedge the session forever, or stream gigabytes into my 8 GB VPS's memory. Now: 10-second wall-clock timeout, 256 KiB output cap. Bounded on both axes, like everything touching remote input should be.

The rest of the posture, decided up front in SECURITY.md: argon2id app password (generated on first run, hash-only on disk), sessions in HttpOnly; Secure; SameSite=Strict cookies, WS Origin allowlist against cross-site WebSocket hijacking, TOFU host-key pinning where a changed key is a hard fail with a MITM warning, a server-side target allowlist (by default herdr will only dial my own sshd on loopback), SSH credentials in Zeroizing memory and never in any log, and CSP with no inline JS anywhere.

The war story: | head killed my server

The strangest bug of the session had no Rust in it. POST /api/login started returning nothing — connection just dropped — while GET /api/me worked fine. Same process, same port.

I had started the dev server with cargo run 2>&1 | head -20 to see the startup lines. head exits after 20 lines, which kills the pipe. The next handler that logged something — login logs, /api/me doesn't — panicked writing to the dead stderr, and hyper dropped the connection mid-request. The server was healthy for every silent endpoint and broken for every noisy one.

💡 Never leave a long-running server attached to a pipe that exits. nohup server > log 2>&1 & — boring and correct.

Verification, because "it compiles" is not "it works"

Every layer got its own proof: 8 unit tests (auth, tmux parsing, config bootstrap); 4 end-to-end Rust tests against a real sshd — including both host-key failure paths (reject on first contact, hard-fail on pin mismatch); a 7-step Node script driving the full WebSocket protocol; and finally Playwright driving a real Chromium through the deployed HTTPS domain: login → paste key → trust fingerprint → herd → attach → type a marker → watch it echo through xterm.js → detach → kill. Plus the negative tests: evil Origin gets 403, off-allowlist target gets refused, spoofed XFF still gets rate-limited.

clippy --all-targets reports zero warnings. The release binary is LTO'd, stripped, and idles at under a megabyte of RSS as a systemd user service.

What Rust felt like, three projects in

After rterm-ssh (Rust-behind-JNI) and now herdr (pure async Rust), the pattern I keep noticing: the borrow checker almost never fought me — channels and ownership transfers make most sharing questions disappear — but the type system kept turning design mistakes into compile errors before they became runtime surprises. The Zeroizing dance, the guarded select arm, the oneshot reply envelope: in each case the awkwardness in the types was pointing at a real concurrency or security question I hadn't answered yet.

herdr v0.1 is tagged, deployed, and herding my sessions. Next up: mobile ergonomics (this thing plus RTerm on the same phone is the actual dream), password rotation UI, and maybe scrollback search.