Alarms & alerting

Monitoring is only useful if it reaches a human at the right moment — not too early (noise), not too late (an outage). WatchTower15 alarms watch your checks and telemetry, open incidents with a severity, and deliver notifications through the channels your team already uses.

Rules, alarms and incidents

Three things, easy to conflate:

A rule is a template from the built-in catalogue — “server CPU usage is high”. It defines what is measured and what can be tuned.
An alarm is a rule applied to one of your resources, with your thresholds. You can author more than one alarm from the same rule.
An incident is what opens when an alarm’s condition is breached.

Because alarms are authored rather than fixed, “two alarms on one metric at different sensitivities” is a first-class pattern: duplicate the CPU rule, set one to 70% averaged over an hour as an early warning and leave the other at 95% as the page.

Alarms from profiles

Rather than authoring the same alarm on every host, a server agent profile can carry alarm templates — alarms a resource is given automatically when it joins that profile. New hosts arrive already alarmed, with no per-server setup.

The rule catalogue

Each rule is one of two types:

Metric rules compare a numeric value (percentage, days-to-expiry, milliseconds) against a threshold.
Boolean rules are pass/fail conditions (reachable or not, changed or not) and escalate on how many times in a row they fail.

Servers

These server rules read agent telemetry. Most are metric rules measured in percent, which default to 3 consecutive samples at each tier and support an aggregation and window. Four differ, as described below:

Server disk latency is high is measured in milliseconds, and Server network interface has errors and Server is dropping incoming connections per second. All three score whole minutes and default to longer streaks.
Server killed a process (out of memory) counts kills over a fixed 15-minute window.
Server network link is down is the one boolean rule: an interface has a link or it has not.

Disk space and inodes are watched on each volume of the server, and the rules that watch the agent, its ports and its system services are below.

Rule	Warning	Critical
Server CPU usage is high	≥ 85%	≥ 95%
Server CPU steal is high	≥ 10%	≥ 25%
Server CPU pressure is high	≥ 20%	≥ 50%
Server memory usage is high	≥ 85%	≥ 95%
Server memory pressure is high	≥ 20%	≥ 50%
Server killed a process (out of memory)	≥ 1 kill in 15 min	≥ 5 kills in 15 min
Server swap usage is high	≥ 50%	≥ 80%
Server I/O pressure is high	≥ 20%	≥ 50%
Server disk latency is high	≥ 100 ms for 5 minutes	≥ 500 ms for 5 minutes
Server load average (1m) is high	≥ 300%	≥ 500%
Server load average (5m) is high	≥ 200%	≥ 350%
Server load average (15m) is high	≥ 150%	≥ 250%
Server network interface has errors	≥ 0.01 errors/s for 5 minutes	≥ 10 errors/s for 3 minutes
Server network link is down	no link for 2 consecutive minutes, armed once the interface has been up in the last 7 days	5 consecutive minutes
A process is running out of file descriptors	≥ 80%	≥ 95%
Server is running out of file handles	≥ 80%	≥ 95%
Server is running out of process IDs	≥ 80%	≥ 95%
Server connection tracking table is filling	≥ 80%	≥ 95%
Server is dropping incoming connections	≥ 0.1 overflows/s for 5 minutes	≥ 10 overflows/s for 3 minutes

Memory and swap are measured as used ÷ total of the host’s memory and swap.

Steal and the three pressure rules score whole minutes. For Server CPU steal is high, Server CPU pressure is high, Server memory pressure is high and Server I/O pressure is high, each sample is the average over a whole minute, so the default 3 consecutive means three whole minutes in a row, and a brief spike inside a minute does not raise them on its own. An aggregation and window you set works on those minute averages. On the server’s Alarms tab, their tiles show the last minute’s average.

Server CPU steal is high watches CPU steal: time the hypervisor gave to other virtual machines while this server was ready to run. Sustained steal means a busy neighbour on the same physical host or, on a burstable instance in standard credit mode, CPU credits running out. A burstable instance with no credits left shows close to 100% steal inside the server, while the cloud provider’s console shows only the instance’s baseline CPU. Server CPU usage is high counts steal as busy, so on such an instance both can fire: steal says why.

The three pressure rules read the kernel’s pressure stall information: the share of each minute in which tasks were waiting for a resource instead of running.

Server CPU pressure is high: at least one task was waiting for a CPU (some).
Server memory pressure is high: at least one task was waiting for memory — reclaim, swap-in or refaults (some). It rises before the out-of-memory killer runs.
Server I/O pressure is high: every non-idle task was stalled on I/O at once (full). A busy database stalls some tasks on I/O routinely; full means nothing on the server could make progress.

A server whose kernel does not report pressure stall information is never alarmed by them: see which systems report it.

An alarm or template whose name was saved in its drawer keeps that name, even when its rule is renamed. To take the rule’s current name, rename it in the drawer.

Server killed a process (out of memory) opens a warning when the kernel’s out-of-memory killer has killed at least one process in the last 15 minutes, and escalates to critical at five. Every kill counts, including kills inside a container or a service that reached its own memory limit while the server had memory to spare, so it can fire while the server’s memory chart looks calm. It resolves once 15 minutes pass with no kill.

Its value is a number of kills, and its tile on the Alarms tab shows the kills in the last 15 minutes. Its thresholds can be tuned; its 15-minute window and its consecutive counts (one, at each tier) cannot, and it takes no aggregation.
It counts from the kernel’s own tally, so a kill while the agent was restarting — the agent itself included — is counted when the agent next reports. After a gap in reporting longer than 15 minutes, kills during the gap show on the server’s Memory Usage (%) chart but do not raise the alarm.
Its notifications name the processes the kernel killed, with their process IDs, up to five: java (pid 8812) and node (pid 9120) on app-02. The email adds a Killed row. This needs agent 1.1.40 or later with system services watched, on Linux 4.19 or later, with the agent outside a container. Elsewhere the notifications name the server. Rarely, a kill’s name reaches WatchTower15 just after the incident opened; that incident then names the server only.
Kills from before the agent started reporting never raise it.

These five rules need agent 1.1.27 or later, which agents install on their own. A server whose agent does not report their metric is never alarmed by them. On the Alarms tab, steal and CPU pressure sit under Compute, memory pressure and out-of-memory kills under Memory, and I/O pressure under Storage.

Server disk latency is high watches how long the server’s disks take per read or write, waiting in the queue included: what iostat reports as await. Each one-minute sample is the slowest disk’s average over that minute, so one slow write is averaged with the rest of its minute, and the default 5 consecutive means five whole minutes. A minute in which a disk did no reads or writes is not scored for it: an idle disk has no latency. A healthy cloud volume or NVMe drive answers in about a millisecond and a spinning disk in about ten, so 100 ms for five minutes is trouble on any of them. How busy a disk is has no alarm: on a device that serves many requests at once, 100% busy is not saturation.

Server network interface has errors watches the receive and transmit errors on the server’s interfaces, usually from a failing network card, a bad cable or a misconfigured link. Each one-minute sample is the worst sample in that minute, so a single error counts: one error in a 15-second interval reads about 0.07 errors/s, above the 0.01 warning. It warns after errors in 5 consecutive minutes, and goes critical at 10 errors/s for 3. Dropped packets have a chart but no alarm: some drivers count frames they ignore, such as those of an unknown protocol, as drops, so a steady trickle can be normal.

Server network link is down is a boolean rule. It warns when an interface has had no link — no carrier: an unplugged cable, a failed switch port or a detached cloud network card — for 2 consecutive minutes, and goes critical after 5.

It watches physical network cards, bonds and VLAN interfaces, not bridges, tunnels, WireGuard or docker0, which go down by design: see link state.
An interface switched off on purpose never raises it. An administratively down interface (ip link set eth1 down) reports no link state, and neither does a VLAN on it.
It is armed once the interface has been up in the last 7 days. A spare network card that has never had a cable never alarms. A link that stays down for more than 7 days drops out of that window, so its incident resolves itself while the link is still down.
An interface that disappears is not down. Removed, or renamed at boot, it stops reporting and nothing fires. A server that stops reporting altogether is caught by Agent stopped reporting.
An interface whose state the kernel reports as unknown, as some virtual network cards do, has no link state and is not watched.
Its consecutive counts can be tuned. It takes no aggregation or window.

One incident per server. These three rules watch the server as a whole: one incident covers every disk, or every interface, and names the server, not the disk or interface. The server’s Storage and Network tabs show which one. They need agent 1.1.28 or later, which agents install on their own; a server whose agent does not report their metrics is never alarmed by them. On the Alarms tab, disk latency sits under Storage, and network errors and link down under Network. The latency tile shows the last minute’s average, the errors tile the last minute’s peak, and the link tile Up or Down.

Kernel limits and dropped connections. Five rules watch how close a server is to the limits behind “Too many open files” and “Cannot fork”, and whether it drops connections. They need agent 1.1.31 or later; what each limit means and how to raise it is on Monitoring.

A process is running out of file descriptors watches the process nearest its own open-file limit (its soft ulimit -n), the limit a process actually reaches on a current server. With agent 1.1.40 or later its notifications name that process and its process ID, such as postgres (pid 4120) on db-01, and the email adds a Process row. With an older agent they name the server. The Open files column of the server’s Processes tab shows every process near its limit.
Server is running out of file handles watches the host’s fs.file-max. Most current Linux servers have no such limit, and on them this rule is never raised.
Server is running out of process IDs watches every thread against the smaller of kernel.pid_max and kernel.threads-max.
Server connection tracking table is filling watches the kernel’s connection-tracking table, which drops new connections when it is full. A server without connection tracking is never alarmed by it.
Server is dropping incoming connections watches listen overflows: connections the kernel dropped because an application’s accept queue was full. Each one-minute sample is the worst sample in that minute, so one burst counts; it warns after 5 consecutive minutes at 0.1 per second or more, and goes critical at 10 per second for 3.
The four percentage rules score the average over each whole minute, three minutes in a row by default. On the Alarms tab, the three limit rules sit under Compute, and connection tracking and dropped connections under Network.
Retransmits and zombie processes have charts but no alarm.

Volumes

Every monitored volume of a server — each filesystem mounted on it that holds real storage, such as / or /data — is watched on its own, and gets these alarms automatically:

Rule	Type	Warning	Critical
Volume is full	metric	≥ 85%	≥ 95%
Volume is running out of inodes	metric	≥ 85%	≥ 95%
Volume is filling up	metric	full within 7 days	full within 2 days
Volume became read-only	boolean	on the first check	on the next check

The first three default to 3 consecutive checks, about a minute apart, at each tier.

Volume is full is the space the volume has used, as a percentage of its size — the figure df reports. Its thresholds, consecutive counts, aggregation and window can be tuned like any metric alarm’s.

Volume is running out of inodes is the share of the volume’s inodes in use — the figure df -i reports — and is tuned the same way. A volume that has run out of inodes cannot create files: writes fail with “No space left on device”, even while it has free space. btrfs and FAT (vfat, exFAT) volumes keep no inode count, so they have no inode alarm.

Volume is filling up is a forecast. At the rate the volume has filled over the last six hours, it warns when the volume will be full within 7 days and goes critical within 2 days. It catches a disk that is filling fast while it still has room, well before Volume is full would. A volume that is flat or shrinking never raises it, and a volume with less than about six hours of history is not scored yet. Its thresholds, in days, and its consecutive counts can be tuned; the six-hour rate it works from cannot.

Volume became read-only tells you that a volume that was writable is now mounted read-only. The usual cause is a disk error: ext4 remounts itself read-only to protect its data, and from then on nothing can be written, although df still shows free space. It warns on the first check that sees it and goes critical on the next. A volume that is read-only by design — one that has been read-only since WatchTower15 first saw it, such as a disk mounted read-only on purpose — never raises it. It needs agent 1.1.23 or later, which agents install on their own.

Volume incidents are named after the server and the mount point — for example web-01 · /data — and reach your channels like any other incident. They appear on the server’s page — its Incidents tab, and the volume’s own drawer on the Storage tab — in Alarming → Incidents, where the incident’s page names the volume and links to it, in the dashboard’s alarm widgets and on the Audit page. The dashboard’s Recent incidents filter has no option for them. They count towards the server’s health: a critical volume makes its server critical.

A few things follow from watching each volume on its own:

One full disk raises one alarm. Of the mounts that share one disk’s space — a bind mount, a second mount of the same device, the datasets of one ZFS pool — only one is a monitored volume, so a full disk is not reported once per mount.
An ignored volume has no volume alarms at all, inodes included. The volumes WatchTower15 ignores — an EFI partition, a volume under 256 MiB, a duplicate, and others — are not watched for inodes either.
A new volume is alarmed as soon as it is reported. A disk attached later is monitored, with all its alarms, once the agent reports it in its next system-information snapshot, within about 5 minutes.
Volume alarms do not come from profiles. A profile’s alarm templates cover the server rules. Every monitored volume gets its alarms, whatever its server’s profile.
You tune a volume’s alarms in its drawer on the server’s Storage tab, with alarms.edit. A volume that is not monitored lists its alarms read-only: they stay inactive while it is ignored.

Where their settings come from. Volume alarms replace two server rules: Server disk is filling up, which — whatever its name — alarmed on the root volume being full, not on how fast it was filling, and Server is running out of inodes. On each server, those two alarms’ settings carried over to its volumes:

The disk alarm moved to the root volume. Its thresholds, consecutive counts, aggregation and window, whether it was switched on, its notification settings and its alarm policies are now those of the root volume’s (/) Volume is full.
Its switch and routing apply to every volume. Whether it was switched on, its notification settings and its policies also apply to Volume is full on every other volume (with the standard 85% and 95% thresholds), and to every volume’s Volume is filling up and Volume became read-only.
The inode alarm’s settings apply to every volume’s inode alarm: thresholds, consecutive counts, aggregation and window, whether it was switched on, notification settings and policies.
A volume that appears later takes its alarms’ settings the same way.
A deleted alarm stays deleted. If you had deleted a server’s disk alarm, its root volume has no Volume is full; if you had deleted its inode alarm, none of the volumes it had then has an inode alarm. A volume alarm that would have taken its settings from a deleted alarm starts with the standard settings and your organisation’s default alarm policy, like any new alarm.
An open incident carries on. An incident that was open on the disk or inode alarm continues on the volume alarm that took over — the root volume’s, or the volume short of inodes — as the same incident, with the same severity and acknowledgement. No new notification is sent. The server’s Audit tab records the move as Disk alarms moved to volumes.

Volumes that were already there. On a server that WatchTower15 was already monitoring when volume alarms were introduced, each of a volume’s alarms is added the first time that volume reads below its threshold, so a volume that was already over it raises nothing for a condition you already knew about. An alarm whose volume reports no data for it — for example Volume became read-only under an agent older than 1.1.23 — is added after 24 hours anyway. The alarms that continue the old ones — the root volume’s Volume is full, and an inode alarm that took over an open incident — are the exception: they watch from the start.

Servers whose root is not a monitored volume. A server whose / is not a monitored volume — for example an agent running inside a container, where / is an overlay — keeps the Server disk is filling up alarm it had, until its root volume is monitored. Its other volumes’ alarms take that alarm’s switch, notification settings and policies. Likewise, while a Server is running out of inodes incident is open on a filesystem that is not a monitored volume, that server alarm stays until the incident resolves; the volumes’ inode alarms take over after that.

Agent reporting

Three server rules watch the agent itself rather than a metric. All are measured in minutes, and every server gets them by default — or, if its profile carries alarm templates, through that profile’s templates for them. A profile whose templates do not include them gives them to none of its servers; add them to the profile to cover its servers.

Rule	Warning	Critical
Agent stopped reporting	no check-in for ≥ 5 min	≥ 15 min
Port reporting stopped	no list of listening ports for ≥ 30 min	— (warning only)
System services reporting stopped	no list of system services for ≥ 30 min	— (warning only)

Agent stopped reporting opens when WatchTower15 has heard nothing from a server’s agent for 5 minutes — the agent checks in every 30 seconds — and escalates to critical at 15 minutes. It clears as soon as the agent checks in again. It catches a host that has crashed or lost its network, an agent that was stopped or keeps failing, and a reboot that never comes back. From the same 5 minutes the server reads Unreachable on its page, in the Servers list, on heat maps and in the notice bar. A server whose agent WatchTower15 is refusing for its clock reads Rejected instead, and the alarm counts from its last accepted heartbeat. A planned shutdown is told apart from a failure:

Powering off is not an outage. When the host shuts down cleanly — poweroff, shutdown -h, or a cloud instance being stopped or terminated — the agent’s last message says so, and the alarm stands down until the agent checks in again.
A reboot keeps the alarm armed. A normal reboot takes a minute or two and never reaches the 5-minute warning. One that hangs — a filesystem check, a kernel panic, a failed start-up — raises it. If the agent cannot tell a reboot from a power-off, it reports a reboot, so a host that never comes back is still noticed.
A reboot from the AWS console or API counts as a power-off. Reboot instance (or aws ec2 reboot-instances) reaches the server as a press of its power button, so the alarm stands down until the agent checks in again, and a server that never comes back from such a reboot is not reported. Rebooting from the server itself — sudo reboot — keeps the alarm armed.
Stopping the agent on a running host raises it (systemctl stop wt15-server-agent): the host is up, but nothing is monitoring it.
In a container, stopping the container counts as a power-off when the agent is the container’s main process. Started under an init process such as tini (docker run --init), it is not, and stopping the container raises the alarm.
A server waiting in Pending, revoked, archived, deleted or deactivated is never alarmed on. Recognising a power-off needs agent 1.1.21 or later.

What its messages say. When Agent stopped reporting opens, and again when it escalates, its email adds Last check-in: when the agent last checked in, then The server may be down, or unable to reach WatchTower15. When it resolves, its email adds what happened:

Reconnected: when the agent checked in again, and after how long, such as 6 Oct 2026, 11:25 UTC, after 2 h 13 min;
Cause: why the server went quiet, at most three causes in time order, each with its time range in UTC, such as The agent could not look up WatchTower15’s address (DNS), 09:12–11:25 UTC. These are the causes the server’s charts mark, in words. The agent’s own error messages, which can hold host names and addresses, are never quoted;
Data: what became of the data from that time, such as All data was recovered., The data kept on the host is being delivered. or 40 min was dropped on the host (09:40–10:20 UTC); the rest was recovered. Time the host was down or the agent was not running reads 1 h was not collected: the host was down. Time the server was powered off, held or paused is not a loss, and is named in brackets: (20 min powered off).

Slack, Teams, webhook, SMS and WhatsApp messages add the same as sentences after the time: Last check-in 09:12 UTC; the server may be down, or unable to reach WatchTower15., or Reconnected after 2 h 13 min. followed by the causes and the data. SMS and WhatsApp carry the resolve in a short form: Reconnected after 2 h 13 min., the main cause (the one that took most of the gap while the agent was running) without its time range, and the data without time ranges. A resolve sent in the moment the agent reconnects, before WatchTower15 has recorded the gap, carries none of these.

Once the gap is complete, the same recipients may also get a gap report: what the server’s alarms would have shown while its data could not reach WatchTower15.

Port reporting stopped opens when the agent is still checking in but no list of listening ports has arrived for 30 minutes — six snapshots in a row. The agent is running, so its port collection is what is failing. It stays a warning, clears with the next list, and is never raised while the whole agent is silent (Agent stopped reporting covers that), for an agent older than 1.1.21, or for a server that is powered off, waiting in Pending or revoked. Look in the agent’s log, /var/log/wt15-server-agent.log, for listening ports not collected. A hardened service unit can hide the socket tables from the agent: do not add ProcSubset=pid or ProtectProc=invisible to wt15-server-agent.service.

System services reporting stopped is the same kind of rule for the system services list: the agent is checking in, but no list has arrived for 30 minutes, so reading systemd is what is failing. It stays a warning, clears with the next list, and is not raised while the whole agent is silent, for an agent older than 1.1.22, for a server that is powered off, waiting in Pending or revoked, for a host without systemd (such as a container), or for a profile that does not collect system services. Look in the agent’s log for system services inventory not collected or its bus cannot be reached. The agent reads systemd over D-Bus, a Unix socket: do not add ProtectControlGroups, or a RestrictAddressFamilies= that leaves out AF_UNIX, to wt15-server-agent.service.

Restarts

One server rule tells you that a server has restarted. It is measured in minutes since the server’s last boot, and, like the rules above, every server gets it by default, or through its profile’s templates when the profile carries alarm templates that include it.

Rule	Warning	Critical
Server restarted	booted ≤ 10 min ago	— (warning only)

Server restarted opens a warning when the agent reports a boot time within the last 10 minutes. A restart is an event, not a level, so it never escalates, and one report is enough. The agent reports the boot time with the server’s system information, every 5 minutes, so a restart is seen within that time; the warning clears on its own once the boot is more than 10 minutes old. The boot a server enrols with is not reported, nor a restart within 2 minutes of enrolling, so a new server is quiet when it arrives. It is never raised for a server whose boot time has never been reported. The server’s Alarms tab shows the time since the last boot, and its History records each restart as Server rebooted.

Listening ports

Every listening port a server has is watched on its own, and gets its alarms automatically:

Rule	Warning	Critical	Applies to
New port opened	on a new port	— (warning only)	ports reachable from outside the server
Port stopped listening	≥ 10 min	≥ 30 min	every port, loopback included

Ports in the host’s ephemeral range get neither.

New port opened tells you that something new is listening on an address other machines can reach: every address, or a particular one (bind scope any or specific). It opens when a port the server was not listening on is still listening at the next snapshot, about 5 minutes later, so a port that appears for a moment is not reported. It is a warning, and it resolves itself after 15 minutes without a resolve notification: the port did not recover, the news simply stopped being new. It is not raised for:

ports that listen on the server itself only (loopback);
ephemeral ports;
the first snapshot after the agent starts collecting ports, which records what is already there;
a known port that listens again after being closed. A port that comes back after 30 days closed counts as new again.

If you did not expect the port, find out what opened it — sudo ss -ltnup on the host lists the listening sockets and their processes. An unexpected listener can be an unintended exposure, or a sign that the host has been compromised.

Port stopped listening tells you that an established port has gone: one that had been listening continuously for at least 24 hours is missing from a snapshot. It warns at 10 minutes, goes critical at 30, and clears when the port listens again. It is not raised for:

a port that was open for less than 24 hours, so a test service you shut down again is not reported;
the first 10 minutes after the host boots, while services are still starting;
a snapshot that was incomplete: cut at 256 entries, or holding an entry WatchTower15 could not read;
ephemeral ports.

If the service was removed on purpose, resolve the incident. Resolving accepts the closure: the alarm does not re-open unless the port comes back and later stops again. After 30 days closed, a port is retired and any incident still open on it resolves. Deactivating the server resolves its port incidents too; if a port is still closed when you reactivate the server, its alarm starts again from that moment.

Port incidents are named after the server and the port — for example web-01 · tcp/8080 (any) — and reach your channels like any other incident. They appear on the server’s page — its Incidents tab, and the port’s own drawer on the Network tab (see listening ports) — in Alarming → Incidents, where the incident’s page names the port and links to it, in the dashboard’s alarm widgets and on the Audit page. The dashboard’s Recent incidents filter has no option for them.

Port incidents count towards the server’s health, as volume incidents do. A newly opened port turns the server Warning for the 15 minutes its New port opened incident lasts. A port that stopped listening turns it Warning after 10 minutes and Critical after 30, until it listens again or you resolve the incident.

System services

Every system service the agent watches — a systemd unit on the server, not a Service your servers check — gets its alarms automatically, once it has been seen up:

Rule	Warning	Critical	Applies to
System service down	not up for 2 consecutive minutes	5 consecutive minutes	daemons, and one-shot services that stay active (firewall units)
System service down	not up for 2 consecutive minutes	— (warning only)	services started by a timer, socket, path or D-Bus, and other one-shot services
System service flapping	≥ 3 restarts or falls in 15 min	≥ 10	daemons only
System service killed (out of memory)	≥ 1 kill in 15 min	≥ 5	every service
System service CPU is throttled	≥ 50 % of periods for 5 consecutive minutes	≥ 80 %	daemons only, with a `CPUQuota=`
System service is near its memory limit	≥ 85 % of `MemoryMax=` for 3 consecutive minutes	≥ 95 %	daemons only, with a `MemoryMax=`
System service is near its task limit	≥ 80 % of `TasksMax=` for 3 consecutive minutes	≥ 95 %	daemons only, with a `TasksMax=`

System service down tells you that a service that should be running is not. What up means depends on the service: a daemon such as nginx or sshd must be running, and so must a one-shot service that stays active, such as nftables or ufw. A service that runs on a schedule or on demand — started by a timer, a socket, a path unit or D-Bus — stops between runs by design, so it is down only when its last run failed. That kind stays a warning: it stays failed until its next run, and one failed run is worth a heads-up, not a page.

Only services that have been seen up are alarmed. A service that was never running when WatchTower15 looked — one that failed at boot, one enabled for a mode the host does not start, a D-Bus service between activations — gets no alarm until it has been up once. That also holds when WatchTower15 first starts watching a server: services already down at that moment are not reported until they have recovered.
It reports the unit’s state, not what the unit does. For a firewall unit it tells you that the unit was stopped — systemctl stop nftables — not whether any firewall rules are loaded. ufw disable bypasses systemd and leaves ufw.service active, so it does not raise this alarm.
Disabling a service on purpose is not reported. systemctl disable --now stops the agent watching it, and any open incident on it resolves.
A service that was removed, for example by uninstalling its package, is reported as down. If you removed it on purpose, resolve the incident: resolving accepts the removal, and the alarm does not re-open unless the service comes back and stops again. After 30 days gone, a service is forgotten and any incident still open on it resolves.
A service that is still installed but stopped re-opens its incident on the next check if you resolve it by hand, because it is still down. Acknowledge it instead, or start or disable the service.
Deactivating the server, or turning off collection of system services on its profile, resolves its system-service incidents.

System service flapping tells you that a daemon keeps restarting. It counts, over the last 15 minutes, the automatic restarts systemd made (Restart=) or the times the service fell from running, whichever is higher; a manual systemctl restart starts the restart count again, and restarts from before WatchTower15 started watching the service are never counted. It warns at 3 and goes critical at 10, and clears when the count drops below 3. It applies to daemons only. On systemd older than 235, which keeps no restart count, it counts the falls alone, so a service that systemd restarts faster than the agent’s 15-second sample may not be seen.

The limit rules (agent 1.1.32 and newer) tell you that a service is running into a limit it was given in its unit file, which the server’s own CPU and memory alarms cannot see: a throttled service runs slowly while the server looks idle, and a service at its memory limit has a process killed while the server has memory to spare.

System service killed (out of memory) opens at the first kill the kernel’s out-of-memory killer makes in the service, counted from the kernel’s log, and resolves once 15 minutes pass with no kill. Kills from before WatchTower15 started counting are never reported.
CPU is throttled, near its memory limit and near its task limit score each minute’s average, so one busy second does not raise them. Memory is the service’s working set: page cache the kernel can drop is left out.
A service without the limit is never alarmed. Most services set no CPUQuota=, MemoryMax= or TasksMax= of their own, so their tile reads No data with the reason, for example reported only for a service with a MemoryMax=. That is expected, not a fault. The kill count is not kept on a kernel older than 4.19, inside a container, or for a service without memory accounting (what is reported).
Services watched before these rules existed were given them too, including services whose down or flapping alarm you had deleted or switched off. Delete or switch off any of the new ones the same way: a deleted alarm is not recreated.

All these rules work from the service’s state every 15 seconds. They are evaluated every minute, and a server that stops reporting altogether does not raise one alarm per service: Agent stopped reporting and System services reporting stopped cover that. You tune a service’s alarms in its drawer on the server’s System services tab, with alarms.edit.

System-service incidents are named after the server and the unit — for example web-01 · nginx.service — and reach your channels like any other incident. Their notifications name the service and then its server, such as Warning Alarm: System service CPU is throttled — nginx.service on web-01. They appear on the server’s page — its Incidents tab, and the service’s own drawer on the System services tab — in Alarming → Incidents, where the incident’s page names the service and links to it, in the dashboard’s alarm widgets and on the Audit page. The dashboard’s Recent incidents filter has no option for them.

System-service incidents count towards the server’s health, as volume and port incidents do. A daemon or a firewall unit that is down turns the server Warning after 2 minutes and Critical after 5; a service started by a timer, socket, path or D-Bus keeps it at Warning.

Domains

Rule	Type	Warning	Critical
SSL certificate expiring soon	metric	≤ 60 days	≤ 30 days
Domain registration expiring soon	metric	≤ 60 days	≤ 30 days
Domain became unreachable	metric	> 30 s	> 60 s
DNS records changed	boolean	on change	— (warning only)
DNS authority records changed	boolean	on change	— (warning only)
WHOIS information changed	boolean	on change	— (warning only)

The two expiry rules fire on the first scan that crosses the threshold — a certificate does not un-expire, so counting consecutive scans would only delay the alarm. Domain became unreachable defaults to 3 consecutive scans.

Services

Rule	Type	Warning	Critical
Service response time is slow	metric	> 10 000 ms	> 30 000 ms
Service became unreachable	boolean	1 consecutive	5 consecutive
Service response check failed	boolean	2 consecutive	6 consecutive
No collector reporting	metric	—	5 min after a check was due
Collectors disagree	boolean	5 consecutive intervals	—

A service is checked by your own servers, its collectors, and these rules read their check results. The two response rules apply to HTTP and HTTPS services only.

No collector reporting opens when no result has arrived from the service’s collectors 5 minutes after a check was due, whatever its check interval: every server in its pool is down, stopped or cannot reach WatchTower15, or none can check it (a name it cannot resolve, a destination it refuses, a parameter it cannot read). The 5 minutes count from the last result, or from the last change to the service’s collectors or check interval, whichever is later. It stays quiet during a WatchTower15 outage, and a service that has never reported is never alarmed on.
Collectors disagree applies in Every-collector mode only: some collectors reach the service and others don’t, for 5 consecutive check intervals. It stays quiet while the service’s own unreachable or response-failed incident is open.

Log sources

Rule	Type	Warning	Critical
No logs received	metric	≥ 15 min	≥ 60 min

Silence is the alarm here: this rule opens when nothing has arrived from a source for the configured time. A source that has never sent a line is never alarmed on, so a newly created source doesn’t page you before it is wired up. The clock runs from when a batch was accepted, not from when its lines became searchable.

Severity & escalation

Severity is how WatchTower15 distinguishes “worth knowing” from “wake someone up”. It’s decided per evaluation:

Metric rules compare the value against the thresholds — past the critical threshold is critical, past the warning threshold is warning, otherwise clear.
Boolean rules escalate on a consecutive-failure streak. A rule opens at warning once it has failed its warning count in a row, and escalates to critical at the higher critical count.

Recovering takes a clear move back. Once a built-in alarm is firing, it steps down or clears only when its value has moved clearly back past the threshold, not the moment it dips under it. A disk hovering around its 95 % critical threshold stays one critical incident instead of flipping between critical and warning with every sample. Opening and escalating are never delayed, and a real recovery still steps down or clears straight away. Alarms on your own custom metrics step down and clear exactly at their thresholds.

Where a rule’s two tiers are indistinguishable — the same threshold and the same count, as with the two DNS … changed rules and WHOIS information changed — it stays warning-only. A single change is worth a heads-up, not a page.

This is why Service became unreachable pages you only after five consecutive failures by default, while a one-off blip merely raises a warning: sustained failure is what escalates.

Tuning an alarm

Defaults are sensible, but an alarm is yours to tune per resource. A payment endpoint might escalate after two slow responses while a batch job tolerates ten; a flagship domain might warn 90 days before certificate expiry.

Tunable	Applies to
Warning / critical threshold	metric rules
Warning / critical consecutive count	rules whose streak is meaningful
Aggregation and window	Volume is full, Volume is running out of inodes, and server rules except Server killed a process (out of memory) and Server network link is down

Some rules deliberately fix their consecutive counts, because counting would work against the rule: both expiry rules, the three change detectors, No logs received, the agent-reporting and listening-port rules, System service flapping and Server killed a process (out of memory) all evaluate on a single observation. The interface only offers the counts where they mean something.

Aggregation & windows

Server rules, Volume is full and Volume is running out of inodes can be evaluated on an aggregation over a window — last, avg, min or max — instead of the latest sample alone. Averaging CPU over 30 minutes catches a sustained problem while ignoring the spiky minute that a last check would page on. For steal, the three pressure rules and disk latency, the window aggregates their minute averages, and for network errors each minute’s peak. Server killed a process (out of memory) takes no aggregation: its 15-minute window is part of the rule. Nor does Server network link is down: an interface has a link or it has not.

How evaluation works

Alarms are evaluated two ways, so nothing depends on a single trigger:

Immediately after a relevant scan is processed, for rules driven by checks — no waiting for a polling loop.
On a continuous pass, which sweeps every active alarm each minute. This is what drives server telemetry rules, which have no scan event of their own, and it’s the safety net for everything else.

Transient scan outcomes (blocked, lookup_unavailable) are ignored entirely — they never open or resolve an incident, so a flaky resolver can’t masquerade as a failure.

Absence is treated the same way. A host that has gone dark never auto-resolves its open alarms: silence is not evidence that CPU recovered, and that is precisely the moment a false all-clear would do the most damage.

While a server could not reach WatchTower15

Alarms are never raised for the past. While a server’s data cannot reach WatchTower15, its alarms have nothing new to judge, and an incident already open stays open. When the data the agent kept on the host arrives later, it fills in the charts, but no incident is opened, escalated or resolved for a time that has passed, and no alarm message is sent about it. A breach still going on when the data arrives opens an incident as usual, at the time it opens, and its messages say since when it was breaching (see Incident lifecycle).

The gap review. What the alarms would have shown is worked out afterwards instead. A few minutes after a gap is complete (its cause settled, and the data the agent kept delivered or given up), WatchTower15 reviews it: it evaluates the server’s alarms over the gap minute by minute, as they would have been evaluated live, starting from the incidents open when the gap began. It uses each alarm’s thresholds, consecutive counts and window as they are when the review runs; a change made during the gap is not replayed. Each stretch in which an alarm would have been firing is a finding: when it began and when it would have cleared, the highest severity it reached, and its peak value. A finding is linked to the incident the alarm did have, if any: one already open when the gap began, one raised during the gap (an alarm measured over a window can still fire on its readings from before the gap), or one raised on reconnect, when the breach was still going on as the server came back.

The review opens no incident, adds nothing to the Audit tab, changes no health and sends no alarm message. Its findings show on the server’s Recovered or Lost banner and in the gap report.

Reviewed: the server’s alarms from the server rules, and the alarms of its volumes and system services, on resources that are evaluated: not an unmonitored volume or an inactive system service.
Not reviewed: the agent-reporting rules, Server restarted, listening-port alarms, and alarms on custom metrics and databases.
Only the data that arrived is judged. A gap that lost some of its data is reviewed on what arrived, and when nothing would have fired it says so in the data that arrived. A gap none of whose data arrived is not reviewed, as there is nothing to judge: the host was down, the agent was not running, what the agent kept was dropped or not delivered in time, or its version keeps nothing while it cannot reach WatchTower15.
Not reviewed either: a gap in which the server’s metrics kept arriving and were evaluated live (only its database snapshots or disk usage reports could not be stored), and a gap of a server that has since been revoked, deactivated, archived or deleted. A gap longer than 7 days is reviewed over its last 7 days.

The gap report

A gap report tells the people who would have been notified what the server’s alarms would have shown. It goes to the people and chat destinations the server’s own Agent stopped reporting alarm notifies, through that alarm’s policies, each policy once: by email and to Slack, Teams and webhooks, never by SMS or WhatsApp, since a text message pages someone and this is about the past. A gap gets at most one report, sent once it has been reviewed, when:

it lasted 15 minutes or more;
it was complete (its data delivered or given up) within the last 24 hours;
and it has something new to say:
- an alarm would have fired without already being an incident;
- an alarm was still breaching when the server reconnected, and was raised on reconnect: the report says when the breach began, which that incident’s own email may give only as Breaching since … or earlier;
- an alarm reached a higher severity during the gap than its incident: a warning incident whose value went critical. That critical was never paged, and the report names both levels;
- or data was lost: dropped on the host, kept but not delivered in time, or not kept by an agent version that keeps nothing.

No gap report is sent:

for an alarm that was already an incident at that severity, open when the gap began or raised during it: its own messages went out at the time. A report sent for another reason still lists it;
for time when nothing was collected, on its own: the host was down or powered off, or its agent was not running. When Agent stopped reporting fired, its Resolved message already says so;
for log lines that did not arrive, on their own;
for a gap that was all WatchTower15’s own outage, or time when monitoring was paused, held or awaiting approval: there is nothing for you to act on;
when the server has since been revoked, deactivated, archived or deleted, has no active Agent stopped reporting alarm, or that alarm’s policies reach no email address and no active chat destination.

The email. Its subject is Gap report:, the server and the gap’s length: Gap report: web-01 (2 h 13 min). It has no coloured banner, as it is not an alarm. Then:

A short table of facts: the Resource type and Resource; the Gap, in UTC with your own time in brackets as on alarm emails: 6 Oct 2026, 09:12 – 11:25 UTC (10:12 – 12:25 BST); its Duration, adding in 2 parts when the gap came in separate parts; and its Cause and Data, as on the Agent stopped reporting Resolved email. When there is no finding, Alarms says why: None would have fired., None would have fired in the data that arrived., Not reviewed: no data came back., Not reviewed: the server’s metrics kept arriving and were scored live. or Not reviewed: the metrics store did not answer in time.
What the alarms would have shown: each finding in time order, at most ten, then And 3 more. Each is its alarm and the highest severity it reached, such as Server CPU usage is high — Critical, over 09:40–10:20 UTC (40 min), peak 99.1 %, threshold at or above 95 %. A volume or system service adds its name: Volume is full on /var. Then how it relates to an incident: Already open as an incident., Raised during the gap., Raised when it reconnected., or Still breaching when the gap ended. An alarm that reached a higher level than its incident names both: Already open as a warning incident; reached critical during the gap.
About this report: WatchTower15 never raises alarms for the past. This report shows what this server’s alarms would have shown while its data could not reach WatchTower15.
A View server button, which opens the server’s page.

In Slack, Teams and webhooks the message starts with the same line as the subject, then the gap and its length, the causes and the data, then what would have fired, at most five: Would have fired: Server CPU usage is high, critical 09:40–10:20 UTC (peak 99.1 %); Volume is full on /var, warning 10:05–11:25 UTC (peak 91.2 %, raised on reconnect). With no finding, it says No alarm would have fired., or what the email’s Alarms says. An alarm that reached a higher level than its incident ends (peak 96.4 %, already open as warning). Its times are in UTC. Its link reads View server, and in Teams its card is grey rather than red.

Incident lifecycle

An alarm’s condition is evaluated.
If it’s breached and no incident is open for that alarm, a new incident opens at the computed severity (warning or critical).
If the severity rises from warning to critical, the current incident resolves and a new one opens at the higher level, so history reflects the escalation and it pages again. If it falls from critical to warning, the same incident is lowered in place: it stays open, keeps its acknowledgement, and no email is sent. After a gap in the data, a critical incident is held, not lowered, until there are enough new readings to judge the critical streak.
When the condition clears, the incident auto-resolves, once there is enough new data to judge it.
A resource that is deleted, deactivated or archived has its open incidents cleared — it must not sit critical for ever. Archiving or deleting a server clears its own and its volumes’, ports’ and services’ at once, without a notification.
An alarm that is disabled or deleted, or switched off on its resource, by its profile or because its checks were turned off, has its open incidents cleared without a notification. So do a server’s telemetry and volume alarms when its agent is revoked; they are not evaluated while it stays revoked.

There is at most one open incident per alarm, so a persistent problem is a single actionable incident rather than a flood.

An incident carries the time it opened. One opened on data that arrived late, such as data a server’s agent kept on the host while it could not reach WatchTower15, is dated when it opened, never back to when the breach began. Its messages say since when it was breaching whenever the breach began well before the incident opened or escalated: on data that arrived late, after a threshold was lowered, or when an alarm was switched back on (Breaching since, in the alarm emails). A breach that ended while the data could not arrive is reviewed, not alarmed.

The Incidents page (Alarming → Incidents) has its status filter on the left and a time window on the right, opening on the last 5 days. An incident is listed when it was open at any point in the window: triggered before the window ends, and not resolved before it starts. So a window that ends now always lists everything still firing, however long ago it began. A resource’s overall health reflects its highest open severity.

Every list of incidents — the Incidents page, and the Incidents tab of a server, a database or a domain — has the same Date & Time column: when the incident triggered, in your date format with seconds, such as 06/10/2026, 14:03:22. Its header names your timezone once, as in Date & Time (BST), and a time on the other side of a change of clocks names its own. Hover over a time to see it in UTC too.

Every step appears on the resource’s Audit tab: Incident triggered, Incident escalated or Incident de-escalated (one event for a severity change, not a resolve followed by a new trigger; a de-escalation keeps the same incident), Incident acknowledged and Incident resolved — with who resolved it, or why WatchTower15 did. They are recorded whatever the alarm’s notification settings, so a resolve nobody was notified about still shows.

Acknowledge and resolve

On an incident’s page (Alarming → Incidents), and on a server’s, a database’s or a domain’s Incidents tab, anyone holding incidents.edit can step in:

Acknowledge an open incident, to show that someone has seen it and is on it. An acknowledged incident is still open: it still counts as the alarm’s one open incident, and it still auto-resolves when the condition clears.
Resolve an incident by hand, without waiting for the condition to clear.

Both are recorded on the resource’s Audit tab under your name, where anyone who can view the resource sees them.

If the alarm sends acknowledge or resolve emails, those emails name who acknowledged or resolved the incident. Recipients who can open the Audit page (Admins, and anyone holding audit.view) also see the IP address it was done from; everyone else sees the name only. An incident that clears on its own says WatchTower15 resolved it automatically.

What the alarm emails look like

The subject is the alarm’s level or what happened, then the alarm’s own name:

Critical Alarm: CPU high
Warning Alarm: CPU high
Acknowledged Alarm: CPU high
Resolved Alarm: CPU high

An alarm about one process ends with that process, after a dash, on every email:

Warning Alarm: System service CPU is throttled — nginx.service on web-01
Critical Alarm: A process is running out of file descriptors — postgres (pid 4120) on db-01
Resolved Alarm: Server killed a process (out of memory) — java (pid 8812) and node (pid 9120) on app-02

This holds for every system service alarm, and for A process is running out of file descriptors and Server killed a process (out of memory) where the agent reports the process (above).

If you filter alarm emails into folders, match on these subjects. They carry the name you gave the alarm, not the name of the catalogue rule, so two alarms on the same metric have different subjects.

No email is sent when an incident steps down from critical to warning: the incident stays open at the lower severity and the resource’s Audit tab shows the change. An escalation from warning to critical sends a new Critical Alarm email.

Each email opens by saying what it is about:

A coloured banner on triggered and resolved emails: Critical Alarm (red), Warning Alarm (amber) or Resolved (green). The button at the bottom follows the same colour.
A short table of facts: the Resource type (Server, Domain, Service, Log source, Metric, Volume, Port or System service), the Resource, the Server for a volume, port or system service, the Process or the processes Killed for an alarm about one process on a server, the Alarm, and the State. A triggered metric alarm adds the Value it measured and the Threshold it crossed, for example at or above 90 %. Acknowledge and resolve emails add who acted.
Time: when it happened, in UTC, and in brackets in your own timezone when Profile → Preferences names one other than UTC: Triggered 6 Oct 2026, 13:03 UTC (14:03 BST), or Resolved …, Acknowledged …. A triggered alarm that measures over a window adds the Window, for example last 15 min. See Times & timezones.
Breaching since, on a triggered email, when the breach began well before the incident opened, such as on data that arrived late or after a threshold was lowered: more than its consecutive count, plus two, in minutes (5 minutes for a count of 3). It reads like Time, 6 Oct 2026, 11:21 UTC (12:21 BST), and ends or earlier when every reading the alarm looked back over, at least the last 15 minutes, was already breaching. Alarms that judge the latest readings of a server, volume, system service, custom metric or database show it; an alarm measured over a window never does.
Last check-in, on a triggered Agent stopped reporting email, and Reconnected, Cause and Data on its resolved one, after Time: see Agent reporting.
What changed, for change alarms such as DNS records or WHOIS: a table with the Previous and Current value of each record side by side. A time in it reads 6 Oct 2026, 14:03 UTC.
About this alarm, on triggered and resolved emails: a short box just above the button, saying in a sentence what the alarm watches.

SMS and chat messages (Slack, Microsoft Teams, webhooks) end with when the event happened, in UTC, after the incident’s sentence: … Triggered at 13:03 UTC., Resolved at … or Acknowledged at …. For an alarm about one process, the sentence names it: Warning: System service CPU is throttled — triggered for nginx.service on web-01 (current 55 %). After an out-of-memory kill it adds The kernel killed java (pid 8812) on app-02. Acknowledge and resolve messages name the same process as the alert. After the time, a triggered message adds Breaching since 11:21 UTC. when its email has that row (with the date, 6 Oct, 23:50 UTC, when it is another day), and Agent stopped reporting adds its own sentences.

A server’s alarms and incidents

On a server’s page, the Alarms and Incidents tabs show everything alarmed on that server — on the server itself and on its volumes, listening ports and system services. Anyone who can view the server can read them; see roles and permissions.

An alarm is firing while its incident is open or acknowledged. An inactive alarm (switched off, or on a resource that is not evaluated) never counts as firing, whatever its incidents.

The alarm counts

The header of a server’s page shows N Critical · N Warning · N OK, counting every alarm on the server and on its volumes, ports and system services, default and custom. Each count opens the Alarms tab filtered to it, so a count always equals the alarms it opens. Hovering over them says how many of the firing alarms are acknowledged, and how many alarms are inactive and not counted.

An alarm is inactive when it is switched off, and also when what it watches is no longer evaluated: a system service that is retired or no longer monitored, a port that is retired or stood down, a volume that is decommissioned or not monitored, or a server that is inactive (waiting for approval, held, deactivated or archived) or whose agent is revoked. Such an alarm keeps its settings, reads Inactive and is not counted as OK; its tile says why. On a server that is itself inactive or revoked, the server is given as the reason for every such alarm on its page, its volumes’, ports’ and services’ included, and the counts’ tooltip says so.

When the server is unreachable, rejected, never seen or powered off, its agent token was revoked, its monitoring is switched off or it has no agent, the counts are dimmed and OK reads No data. With no new data, or too little new data after a gap to judge the condition, alarms keep their last state rather than clearing, so the counts are only the last known states — except Agent stopped reporting, which watches the silence itself while the server is unreachable, or rejected with no heartbeat accepted for 5 minutes.

The Alarms tab

Each alarm is a tile showing:

Its live value on a gauge against its warning and critical thresholds — the same figure WatchTower15 evaluates — refreshed every minute while the tab is open. It reads No data when there is no recent sample. A firing alarm says over its threshold, or, while its value is back past the threshold but has not yet moved clearly back, back under 95 % (back above 2 days for an alarm where lower is worse).
Or its state, in words, for an alarm that has no value to plot: Up or Down for System service down and Server network link is down, Writable or Read-only for Volume became read-only, OK or the severity for the agent-reporting alarms. Event alarms say what happened: Server restarted shows the time since the last boot (for example 6d 3h since the last boot), or Restarted when it fires; New port opened reads New or Not new; Port stopped listening reads Closed or No closure. A switched-off alarm reads Off, and so does an alarm on a retired or unmonitored resource, or on an inactive or revoked server, with the reason under it (for example not evaluated: this unit is retired or not monitored).
Its state — Critical, Warning, OK or Inactive — and since when, its conditions and its alarm policies. Policies are named for holders of policies.view; everyone else sees how many there are.
Whether its incident is acknowledged. An escalation from warning to critical opens a new incident, which is not acknowledged; a step down from critical to warning keeps the same incident and its acknowledgement.

Firing alarms come first. The alarms are grouped by what they watch — Compute, Memory, Storage, Network, System services and Availability. Within those, each volume’s, port’s and system service’s alarms are grouped under it, collapsed to a one-line summary such as nginx.service — 2 alarms · OK, with an arrow that opens that volume, port or service. A group opens on its own when one of its alarms is firing, or when you pick any filter other than All. Each alarm has its own address, so a link can open the Alarms tab at that alarm.

The filters are All, Firing, Critical, Warning, OK, Inactive, No policy, Default and Custom. When you open the tab, Firing is selected if anything is firing, and All otherwise.

Filter	Shows
Default	The standard alarm for its rule — the one WatchTower15, or the server’s profile, gave the resource — however you have tuned or renamed it
Custom	An alarm added beyond the standard one: a second alarm on the same rule, a duplicate, or an extra alarm template from the server’s profile

The first alarm you add yourself on a rule that the resource had no alarm for counts as Default.

With alarms.edit you can add alarms (Add alarm) and edit them, and from a tile’s menu Duplicate, Disable or Enable, and Reset to profile an alarm you changed from its profile’s; Delete needs alarms.delete. Without them, every tile is still shown, read-only.

A database’s page has the same tab; see A database’s alarms.

The Incidents tab

The tab is one Incidents card. Under its title, one row holds the filter on the left — Active (open or acknowledged), Resolved or All, each with its count — and the time range on the right. The card lists every incident on the server and on its volumes, ports and system services that was open at any point in that range, newest first and 20 to a page, with an On column naming what each is on: This server, or the volume, port or service, linked to it. An incident on something since removed reads, for example, a removed port.

The time range is the page’s, the one the chart tabs and Logs use: changing it here changes it for every tab, and the counts in the filter are counted inside it.
The filter opens on Active, and your choice is kept in the page’s address, so a link opens the same list. The tab’s own badge counts everything firing now, whatever the range.
With incidents.edit you can acknowledge and resolve incidents here; with incidents.view each row also opens the incident’s page.
An empty list names the range, as older incidents can lie outside it: No incidents for this server in the last 3 hours., or No active incidents … and No resolved incidents … under those filters. A range picked on the calendar is named by its dates. A link beside it widens the range: Show the last 5 days, then Show the last 30 days. Older incidents are reached with Custom.

A database’s and a domain’s pages have the same tab; see A database’s incidents and the domain page.

A database’s alarms

On a database’s page, the Alarms tab shows its alarms as tiles, as a server’s Alarms tab does: the same live values, refreshed every minute while the tab is open, the same states, filters and menu. Anyone who can view the database (database.view) can read it, without alarms.view. The database’s open incidents are on its Incidents tab.

The header of the database’s page shows its alarm counts, Alarms · N Critical · N Warning · N OK, as a server’s does, over the alarms its Alarms tab lists. They refresh every minute. Each count opens the Alarms tab, filtered to it. While the database is paused they are dimmed and OK reads No data.

What differs on a database:

Its sections are Availability, Connections, Queries, Host, Replication, Storage, Schema and Logs. A database has no volumes, ports or services, so there are no groups: every tile is the database’s own.
The values read in their units: Database connections high as a percentage of max_connections, Database replication lag high in seconds, Database aborted connections high and Database slow queries high per second, Database threads running high and Database deadlocks as counts, and Database storage filling up in days to full, or No data while there is no forecast. The three RDS alarms read as percentages from cloud metrics: RDS CPU high its 10-minute average, RDS free storage low and RDS storage burst balance low the latest CloudWatch point; where the database has no such series, No data says why (for example reported only on Amazon RDS, with the cloud metrics read policy).
Some read in words. Database unreachable reads Reachable or Unreachable, and Database replication stopped reads Running or Stopped, or No data on a database that is not a replica. Database error-log errors reads No errors or Errors logged, or No data (reported only while the error log is read (Info → Logs)): see Slow and error logs. No collector reporting shows its state only: OK or its severity. Database restarted shows the time since the database last started, or Restarted when it fires. Database schema changed shows the time since the last change, Changed when it fires, or No change before the first.
A paused database is not evaluated: a note above the tiles says so, and every tile reads Off, with not evaluated: this database is paused under it.
There is no Reset to profile: a database has no profile. With alarms.edit you can add alarms (Add alarm), edit them, and Duplicate, Disable or Enable them; Delete needs alarms.delete.

A database’s incidents

A database’s Incidents tab is a server’s Incidents tab, for the database: the same card, filter and counts, columns, paging and empty states, with This database in the On column. Its time range is the database page’s own, the one its Overview charts, so changing it on either tab changes it on both. Anyone who can view the database (database.view) reads the tab, its open incidents included, without incidents.view. Acknowledging or resolving an incident needs incidents.edit, and opening its page incidents.view.

Notification channels

Incidents are delivered through alarm policies, which route to targets:

Targets — individual users, contact groups, or broadcast to all users / all groups in the organisation.
Channels — email (the default), plus Slack, Microsoft Teams, and generic webhook.

Slack, Teams and webhook messages start with the same line as the email subject, for example Critical Alarm: CPU high.

A gap report goes through the policies of the server’s Agent stopped reporting alarm by email and to Slack, Teams and webhooks only, never by SMS or WhatsApp.

Every delivery attempt is recorded, so you can confirm an alert actually went out — and where.

Uptime, domain & service checks — the scans that feed alarms.
Metrics & telemetry — what the server rules measure.
Security scanning — the certificate/DNS rules in a security frame.
Tagging — targeting policies at groups of resources.