Why a monitor says a site is up when it is down

A dashboard full of green is not a measurement. It is the last measurement, redrawn. When the thing that takes the readings dies, most monitors keep painting the last known status forever — and nothing alerts, because an alert needs a reading. This is how that happens, how to test it on the tool you already pay for, and what SoloNOC does instead.

How a monitor ends up lying

Almost every monitor stores two things about a device: a status and the time of the last reading. The screen draws the status. If readings stop arriving, the status is never contradicted, so it stays whatever it was the last time anything was heard — usually “up”, because most things are up most of the time.

The alerting side fails the same way, and for a reason worth saying out loud: an alert rule is a question asked of a reading. “Has this device failed to answer ping for two minutes?” needs two minutes of pings. No readings means no failing readings means no alert. The quieter the failure, the less the engine has to say about it.

So the worst outage a monitoring tool can have is not a device going down. It is the collector going down: the agent, the probe, the poller — whatever it calls the thing that takes readings inside the network. One box dies and an entire client site goes dark and green at the same moment.

  • The collector died. The VM was rebuilt, the Pi’s SD card gave out, someone unplugged it, the site’s internet is down. Every device at that site freezes at its last status.
  • The check passes but the service does not work. A ping answers while the application behind it returns errors. The monitor is right about what it measured and wrong about what you wanted to know.
  • The status page is a copy of a copy. A public status page that reads a stored status rather than a recent reading inherits the same staleness and shows it to your clients.

Test it on the tool you have now

This takes fifteen minutes and needs no new software. Do it on your own office, not a client’s.

  1. Stop the collector, probe or agent for one site — stop the service, or pull its network cable.
  2. Wait longer than the tool’s own check interval. Fifteen minutes is enough for most.
  3. Open the dashboard and read what the devices at that site say.

If they still say “up”, with a green summary at the top, your monitor cannot tell the difference between a healthy network and no network. Check three more places before you decide the tool is fine: the site or group summary, the mobile app, and any status page your clients can see. They often disagree.

Then check what arrived on your phone. If nothing did, the tool has no notion of its own blindness, and the first time you will learn about it is a client phoning you.

What SoloNOC does

A SoloNOC collector takes a reading every minute, but on a quiet network it only sends every 5 minutes, so a few minutes of silence is normal and must never be called an outage. What is not normal is 3 missed sends in a row — so the window is 15 minutes, set by the collector’s own interval rather than by a number picked for a screen.

  • Past the window, a device reads “No data”, with the age of the last reading and the status it last held, instead of its last status drawn as though it were current. A reading with no timestamp, or one that cannot be read, counts as no data — an unknown age never resolves to “up”.
  • No screen may say everything is fine while a collector is quiet. The network summary asks when each collector last reported and says how many are not reporting. A check that failed is said out loud with its reason, never folded into health.
  • A site with no collector says so rather than drawing an empty green ring, and a site card drops its health ring once its readings are outside the window.
  • The quiet collector itself raises a notice. A collector silent between 15 minutes and a day sends one notice to the organisation’s channels and its admins’ email, routed by the same notification rules as everything else — once per silence, not once per cycle. A collector that has never reported is not paged: that is an install in progress. One gone more than a day is not paged either, because the screens have been saying it all day.
  • The public status page counts only readings inside the same window, and says “No data” when none arrived, so a client is never shown stale green.

All of it is held by a contract gate that walks the live product and fails when a reading outside the window is drawn as health, and by thirteen test files beneath it.

What this does not fix

  • It does not tell you a service is broken while its host still answers ping. That is a check you have to add, not a freshness problem.
  • It makes a quiet collector visible; it cannot see a site whose collector and internet both went down from inside that site. For a branch office with no server at it, a collector at another site — your own office will do — can watch its address and page for that site by name.
  • The quiet-collector notice goes to channels and email. It is not a phone push: a push on this product carries an alert the phone can acknowledge, and a notice has no alert for that button to act on.
  • A collector silent for more than a day stops producing notices. The screens keep saying it.

Run the test against SoloNOC

Put one collector on one site on the Free plan, with no card, and then stop it. Watch the devices turn to “No data” with their ages, and the summary stop claiming everything is fine. Create an account.

Related: stop one failure from sending forty alerts, and the uptime and SLA calculator, which says what your check interval can and cannot prove.

See it before you sign up

Open a real SoloNOC dashboard on a sample network — sites, devices, alerts, topology and incidents — without creating anything.

Or start the free plan

No sign-up, no card. A real SoloNOC account with a sample network, read-only — nothing you press changes anything.

More from SoloNOC

SoloNOC is network monitoring for the one-person IT shop and the small MSP: one agent per site, flat plans, a 14-day trial with no card. Start free.