Lessons from a Retired Alerting Stack: LibreNMS, ntfy and SMS

Status of this account

This is a historical account of the LibreNMS-based alerting setup and the lessons carried into its replacement. LibreNMS was retired on 20 October 2026 and replaced by Checkmk Community. The publication date is not a claim that LibreNMS is still running.

The current system retained the existing ntfy notification path and an independent SMS gateway. The migration records include a controlled push notification and recovery test. They do not establish a new end-to-end SMS delivery test, and this article does not claim one. Example addresses are documentation placeholders.

What the earlier stack was trying to solve

The aim was to make useful monitoring affordable to pay attention to. Host reachability, services, storage and certificates can all produce legitimate observations. Sending every short-lived change through the most disruptive channel makes the observations harder to act on.

The earlier design used self-hosted ntfy for ordinary push notifications and reserved the mobile gateway for a much narrower notification-path failure. The lesson is the separation of questions: is the application healthy, is the monitoring result stable enough to act on, and can the operator still receive a notification?

Those questions need different evidence. A successful request to a push server does not prove that the phone displayed the message. An HTTP failure at a third-party relay does not prove that the local notification server has failed.

Narrow escalation around the failure you mean

The original escalation work exposed a scope problem: a broad match could include an upstream dependency as well as the local delivery path. That made a brief external failure eligible for the same disruptive treatment as a failure of the operator's own channel.

The important fix was an explicit rule scope and a deliberate delay. A check's name or severity should not silently promote an unrelated dependency into a page-worthy local incident. Delays reduce noise, but a chosen delay also changes how quickly a sustained problem will be reported.

This is not a promise that a five-minute polling interval detects every five-minute outage. Short failures can fall between observations. Detection interval, retries and notification delay describe different parts of the timeline.

A separate mobile path still has dependencies

An LTE gateway can provide a delivery route when the normal internet uplink is unavailable. It does not remove the need for local power, a working gateway, valid mobile service and an observer able to reach the gateway.

That makes “out of band” a useful design goal rather than an absolute claim of independence. Document the remaining shared failure modes. A test that breaks the normal push endpoint exercises one scenario; it does not prove delivery during every possible network or power failure.

The ntfy deployment also has platform-specific behavior. In particular, iOS delivery can involve an upstream wake-up path. Review the deployed relay settings and the documented data flow before making privacy or availability claims about that path.

Keep examples separate from the live configuration

A minimal documentation fragment might look like this:

# Illustrative addresses only; this is not a production configuration.
base-url: "https://ntfy.example.com"
listen-http: "192.0.2.20:8093"
auth-default-access: deny-all
enable-signup: false

This does not include credentials or demonstrate a functioning proxy, firewall or subscription. Separate permissions for publishing and reading are part of the actual deployment, not a reason to paste a live bearer token into a blog post.

For a notification test, choose a controlled event and record both the problem and recovery. If the problem uses SMS and recovery uses push, label those paths explicitly rather than calling it a same-channel recovery test.

What changed with Checkmk

The replacement moved observation and notification rules into the current Checkmk configuration. Existing subscriptions were retained so that the phone continued to receive the intended topic. The independent SMS gateway remained a separate service.

The migration also distinguished hardware observations from the customer operating system. A read-only hardware collector can report what its management interface exposes; it cannot promise application-level health or complete disk diagnostics when the interface does not provide them.

The verification evidence belongs to that specific migration and its controlled tests. The old LibreNMS rule syntax is not a recipe for the current Checkmk system, and a historically successful SMS test must not be relabelled as a new test of the replacement.

The rules worth keeping

Start with an observation that corresponds to a real operational question. Give disruptive escalation a precise scope. Test recovery as well as failure, and keep the remaining dependencies visible. Review quiet channels deliberately: silence may mean stability, a disabled rule or broken delivery.

That is the useful result of the retired setup. The tools changed; the requirement for a clearly defined, actually tested alert path did not.

References: ntfy configuration, Checkmk notification concepts.