One commit on a public Python repo, one data refresh on a private Astro site. By any measure, a thin week. But the single commit on alertmanager-talk-bridge is the one I want to write about, because the problem it solves is the kind of quiet infrastructure detail that looks trivial until the moment your Kubernetes cluster is on fire and Alertmanager’s webhook has been silently dropping notifications into the void.
Retry Logic for a Webhook Bridge Nobody Notices Until It Fails
alertmanager-talk-bridge is a small Python service I built and maintain as a public project. The concept is simple: Prometheus Alertmanager fires a webhook when something goes wrong in your cluster, and this bridge catches that webhook and forwards the notification into a Nextcloud Talk room. One-way pipe. Monitoring alert in, chat message out.
Simple concept, awkward failure mode.
The original version had no retry behavior at all. If Nextcloud Talk returned an error (a temporary network hiccup, a brief Talk API blip, whatever), the bridge swallowed the failure and moved on. Alertmanager’s own retry logic would kick in after a while, but by then you were depending on two separate retry clocks staying in sync. In practice, under certain timing conditions, notifications would just disappear. You would see the firing alert in Prometheus but no message in the Talk room. The silence was the problem.
The fix I shipped this week adds a retry loop on the bridge’s side. When the Talk delivery fails, the bridge now waits and tries again before returning an error status to Alertmanager. The logic is deliberately conservative: I did not reach for a full exponential backoff library. This is a lightweight webhook receiver. It lives in a sidecar-style deployment, and I wanted the retry behavior to be readable in a single glance at the source. A short loop, a fixed delay, a maximum attempt count. If all retries are exhausted, the bridge returns a proper non-2xx status so Alertmanager knows the delivery failed and can schedule its own retry on its own timeline. Both sides stay honest about what happened.
What I am happy about: the change is small and the surface area stays small. What I am less happy about: I do not have a test that exercises the retry path under a simulated Talk timeout. Right now the retry logic is covered by manual testing against a real Talk instance. That is not good enough for infrastructure code. Adding a proper mock for the Talk HTTP client is on the list for next week.
If you run Prometheus and Alertmanager on Kubernetes and your team uses Nextcloud Talk for internal communication, the repo is public and the setup is documented: github.com/marcelrgberger/alertmanager-talk-bridge.
The Other Commit
The second commit was the data refresh on a small static site I maintain. Same as three weeks ago. Same as two weeks before that. I still do not have a scheduled pipeline in place for this. It is the one maintenance task on this site that I keep doing by hand, and I keep meaning to wire up an automated rebuild trigger, and I keep not doing it because it takes four minutes per month and feels like yak-shaving to automate. At some point the calculus will flip. Not this week.
What This Week Looked Like From the Outside
Two green squares on the graph. Anyone glancing at my GitHub activity would conclude nothing happened. That is the gap between commit count and actual thought. The retry loop took longer to reason through than to write. Getting the failure semantics right was the real work: who owns the retry, which status code communicates what, and where the responsibility boundary between bridge and Alertmanager actually sits. The code is maybe fifteen lines. The thinking behind it was an hour.
If you are considering having custom infrastructure tooling or a monitoring integration built for your business, that ratio is worth knowing going in. A lot of the cost of good software is invisible in the diff. You can see more of how I work on the services page.