Wednesday morning I opened the dashboard for my home security system and noticed the heartbeat monitor had reported a clean signal all night — even though the RTSP stream had silently died around 3 a.m. Two commits later the system actually does what it was always supposed to do. Small week, real fix.
When the Heartbeat Lies
The system — camera-s3-stream — captures footage from an IP camera over RTSP, segments it with ffmpeg, syncs segments to S3 via rclone, and uses a dead-switch heartbeat to raise an alert if anything in that chain stops moving. The idea is simple: if a fresh segment does not land in S3 within a set window, something is wrong.
The bug was in how liveness was defined. The heartbeat pulse was tied to rclone’s exit code. If rclone finished successfully, the heartbeat fired. That sounds correct — until ffmpeg quietly hangs on a stale RTSP connection. ffmpeg was no longer producing segments, so rclone had nothing to sync, exited cleanly with code zero, and the heartbeat fired anyway. The dead switch never tripped. The camera appeared healthy. It was not.
The fix has two parts. First, I added -timeout and -stimeout flags to the ffmpeg RTSP invocation. If the camera stops responding, ffmpeg now gives up after a defined window rather than sitting there indefinitely. Second — and this is the more important change — I decoupled the heartbeat from rclone’s exit entirely. The pulse now fires only when a fresh segment file actually exists on disk after the sync cycle. No new file, no heartbeat. That is the only signal that matters: did new footage land?
I also updated the flow diagram in the docs to reflect the new semantics. I had been meaning to do that for weeks. The old diagram showed the heartbeat wired directly to the rclone subprocess, which had not been the intended design — just the accidental one.
The Honest Part
I had a false sense of confidence in this system. It had been running without alerts for months, which I read as “working”. It was working most of the time. But silent failures in security systems are the worst kind — the whole point is that the thing should be loudly broken, not quietly broken. A heartbeat that fires on “rclone exited zero” rather than “new footage exists” is not a heartbeat; it is a green light that can stay green while the camera sits dark.
I would not have caught this without re-reading the flow diagram while writing the docs update. That is mildly embarrassing but also a good argument for keeping documentation close to the code and actually reading it occasionally.
For Anyone Building on Similar Infrastructure
If you are running any kind of home-server setup — cameras, sensors, monitoring pipelines — the lesson here is worth keeping: liveness checks must be grounded in output, not process exit codes. A process that exits cleanly without producing anything is not a healthy process. It is a process that found nothing to do, which might be fine, or might be exactly the failure mode you were trying to detect.
If you are thinking about having a custom application built that involves hardware integration, streaming, or monitoring, this is the kind of edge case that tends to surface late and hurt quietly. I cover how I approach reliability in small systems on my services page.
The camera-s3-stream repo is private, but if you are curious about the architecture — RTSP capture, rclone sync, dead-switch pattern — feel free to reach out.