How to stop one network failure from sending forty alerts

Why one fault becomes many alerts

A monitor sees each device on its own. If an uplink or a switch fails, every device behind it becomes unreachable together, and each one triggers its own device-down alert. The first alert is the useful one; the other thirty-nine are symptoms. At 2am the symptoms bury the cause.

The fix: tell the monitor what depends on what

1. Require the failure to last. A device is only down after it has been unreachable for a set time. SoloNOC’s recommended “Device down” alert fires after the device stops answering ping for 2 minutes, with a 15-minute cooldown before it can fire again, so a single dropped packet pages nobody.
2. Declare the upstream of each device. Say that the access points and the file server sit behind the core switch, and the switch sits behind the firewall. This is the one step a tool cannot guess for you.
3. Let the root cause win. When a device-down alert fires and a declared upstream already has a firing device-down alert, SoloNOC keeps the downstream alert but marks it suppressed and points it at the root cause. It notifies nobody. If upstream and downstream go down in the same check cycle, the upstream is settled first, so the cause is never suppressed by its own casualty. If the dependencies form a loop, exactly one device still pages.
4. Silence planned work. A maintenance window with “suppress alerts” set, optionally recurring, stops alerts for its site while you work. Suppression rules can also treat flapping or repeats inside a dedup window as noise.

What this does not do

• It suppresses device-down alerts behind a down upstream. Threshold alerts such as high CPU are separate rules.
• It only knows the dependencies you declared. An undeclared dependency is paged like any other device.
• The suppressed alerts are kept, marked suppressed and pointed at the root cause, not deleted, so the record of what failed stays complete.