Uptime, domain & service checks

While the server agent watches your hosts from the inside, WatchTower15 watches your domains from the outside — the way your users experience them. A fleet of independent scanner workers runs each check on a schedule; no agent, no cron, nothing to install. Your services are checked by your own servers, from inside your network: see Services.

The check catalogue

Check	Target	What it verifies	Interval
DNS	domain	Resolves A, AAAA, MX, TXT, CNAME, NS, SOA, CAA records and captures the values.	5 min
Domain reachability	domain	Probes four endpoints independently — http/https × www/apex — treating any HTTP response as reachable. A domain’s daily uptime bars are UTC days.	60 s
SSL/TLS certificate	domain	Common name, issuer, validity dates, days-to-expiry, self-signed and chain-valid flags. A certificate counts as expired from the moment it expires, not a day early.	daily
Domain expiry	domain	Registration expiry, registrar and nameservers via WHOIS/RDAP.	daily
Domain info	domain	Registrant/registrar detail and domain status.	daily
Server reachability	server	ICMP ping with an automatic TCP-connect fallback; reports RTT and the method used.	60 s

Services are not in this catalogue: your own servers check them, on each service’s own interval (Services).

How checks are scheduled

A scheduler runs every minute and, for each check type, selects the resources whose next-scan time is due, claims them with a row lock so two schedulers never double-book the same resource, and enqueues them in batches. Workers pick jobs off the queue, run the check, archive the full result, and report a compact summary back. This design scales horizontally — add workers, not complexity — and naturally rate-limits itself so a large estate spreads its load across the minute rather than stampeding.

Status vs outcome — why you don’t get false alarms

Every check records two independent things, and understanding the split is key to trusting your alerts:

• Status is the health of the check itself: queued, running, completed, failed, timeout.
• Outcome is the finding: data_found, no_data, target_not_found, blocked, lookup_unavailable.

Crucially, outcomes of blocked (something actively refused the probe) and lookup_unavailable (a resolver or upstream was momentarily unavailable) are excluded from change detection and alarm evaluation. A flaky resolver or a transient network refusal does not open an incident — only a genuine, classified finding does.

Classification is by cause, not by string-matching

Outcomes are decided by the type of condition encountered, never by scraping an error message. That’s what makes the transient-error exclusion reliable.

Check detail worth knowing

SSL/TLS. The certificate check performs a direct TLS handshake and extracts the certificate chain. If the host is unreachable for a handshake, it falls back to public Certificate Transparency logs so you still get certificate visibility for endpoints that aren’t directly reachable.

Domain reachability. All four endpoint variants are probed independently and redirects are not followed, so you can see exactly which of http://, https://, www. and the apex actually answer — a redirect chain can’t mask a broken variant.

Server reachability. ICMP ping is attempted first; where ICMP is blocked the check automatically falls back to a TCP connect and reports which method succeeded, so a firewall that drops ping doesn’t read as an outage.

Change detection

For DNS, SSL, domain-expiry and domain-info checks, WatchTower15 tracks whether each result changed versus the last successful scan and links the two, so you can see when a certificate rotated or a DNS record moved. The heavy scan payloads are archived to object storage and only the extracted fields plus a pointer live in the database — you keep full history without ballooning storage.

On a domain’s page, each of these checks’ tabs has a Change History → link: scan by scan, what changed, with the old and new values. The DNS tab also lists the scans in which the records changed under the records themselves.

That is not the same as the domain’s Audit tab, which is its part of the audit trail. A detected change appears there as a single Change detected event that names the fields that changed — not their values — and links to the check’s Change History. A domain’s first scan (its baseline) and a flip back to a value already reported record no such event. A DNS change can also open an incident, so the Audit tab can show both: what the scan saw, and what the alarm concluded.

The Domains list

Domains lists your organisation’s domains, 10 to a page by default. Above the list, narrow it by a keyword, by the domain’s name, by Health and by tags; the filter button on the left clears them all.

• Health: All, Healthy, Warning, Critical and Unknown. All is on until you pick one or more of the others, which then show together; turn off the last one and All comes back.
• The filters are remembered for you in each organisation, so they are still there when you come back.

The domain page

Each domain has its own page, reached from Domains: its name and Edit at the top, and the tab bar under them. The breadcrumbs, the name and the tab bar stay pinned to the top of the window as you scroll, with a shadow under them once the page scrolls beneath. A banner under the tab bar says when the domain’s monitoring is switched off. Each tab has its own address, the page’s address followed by the part shown, so a link opens the tab it names:

Tab	Address	What it holds
Overview	—	The domain’s health, and Uptime (last 30 days): the endpoints it monitors, and a bar per UTC day, green for full uptime, amber for a day with any outage, red for a day down throughout and grey for a day with no scans. Hover over a bar for the day and its uptime
Registration	/registration	The Domain Registration card, from the domain expiry check: registrar, registration date, expiry date and days until expiry
Whois	/whois	The Whois/Registration Information card, from the domain info check: registrar, registrant name, email and organisation, whether privacy is on, the TLD and the country
SSL Certificate	/ssl	The certificate’s issuer and common name, when it is valid from and until, whether its chain is valid and whether it is self-signed, and its days until expiry
DNS	/dns	The latest A, AAAA, MX, NS, TXT and CNAME records, then DNS changes: the scans in which the records changed, the first being the baseline
Alarms	/alarms	The domain’s alarms
Incidents	/incidents	Every incident on the domain that was open at any point in the time range, filtered by Active · Resolved · All with a count in each, as on a server’s page. The range opens on the last 3 hours. Needs the incidents.view permission: without it, the tab says your role does not include viewing incidents
Info	/info	Details — the domain’s name, its type (Apex domain, Subdomain or Wildcard), whether its monitoring is active, the checks it gets and whether each is on, the endpoints it checks and its notes — and its Tags
Audit	/audit	The domain’s Audit tab

Each check’s tab — Registration, Whois, SSL Certificate and DNS — says under its card’s title when the check last ran and when it runs next, such as Scanned 5 min ago — Next scan in 55 min, with a Trigger a scan now link that runs the check straight away. Hover over either time to see it in full. Each of their cards ends with a Change History → link to the check’s changes.

Subdomains and wildcards

The registration and WHOIS checks run on an apex domain (such as example.com) only, so the page of a subdomain (such as shop.example.com) or a wildcard (such as *.example.com) has no Registration and no Whois tab: an address to either opens Overview. A wildcard is not checked for uptime either, so its Overview says it has no uptime data. The Info tab gives the domain’s type and the checks it gets.

Services

A service is an endpoint — HTTP, HTTPS, TCP or UDP — that your own servers check, from inside your network. You choose the servers, its collectors: servers running the server agent that can reach the endpoint. WatchTower15 never connects to a service itself, so an internal API, an admin port on a private subnet or a load balancer that only your network can see is checked the same way as a public one. There is no outside-in view of a service: what you see is what your collectors saw.

A service is checked every check interval (1 minute to 24 hours), each check given its timeout (5 to 120 seconds). An HTTP or HTTPS service gets two checks: reachability (the endpoint answers) and response (the answer is the one you expect). A TCP or UDP service gets reachability only.

The six sections

A service is described in six sections, the same on Add service and on Edit: on Add they are steps (Next checks the step, Back keeps what you typed), on Edit they are tabs with one Save.

Section	What it holds
Service	Name, protocol, host and port, check interval, timeout, tags, and Active (off pauses the service).
Parameters	Values the check reads on its collectors — see Parameters.
Request	HTTP and HTTPS: the method, the path with its query, request headers and a body; Verify TLS certificate and Follow redirects (up to 10), both on unless you turn them off. A stored header value is write-only: it is kept until you replace it.
Response	HTTP and HTTPS: response monitoring on or off; the expected HTTP statuses (any of a list), text the response must contain, and an expected content type (well-formed JSON or XML).
Authentication	None, Basic (username and password) or Bearer (a token): a literal, or parameters read on the collectors.
Collectors	The servers that check it, and how — see Collectors.

A host that is a link-local or cloud metadata address (such as 169.254.169.254) is refused. Switching a service’s response checks off, or changing it to TCP or UDP, switches off its response alarms and clears any incident they had open. Switching the checks back on doesn’t re-enable those alarms: turn them on again from the service’s alarms.

Collectors

A service’s collectors are chosen as a pool, as a database’s are: specific servers in your order of preference, or every server carrying a set of tags. A server that gets the tags later joins the pool on its own; one that loses them leaves it. A server can check a service when it is approved and active, has sent a heartbeat in the last 90 seconds, its agent can run service checks (and read parameters, when the service has any; agents upgrade themselves within minutes of a release), and it checks fewer than 200 services.

Saving a pool starts a verify: every server in the pool checks the service once, and the Collectors tab shows a checklist per server — DNS resolved, Destination allowed, Connected, TLS certificate valid, Parameters resolved, Response as expected — with where it stopped and why. A server that has not answered within 2 minutes is likely offline, restarting or upgrading, or cannot reach WatchTower15.

One collector, one standby (the default). WatchTower15 picks one server to run the checks, the active collector, and keeps another warm as the standby: your order first (for specific servers), then the fewest services carried. If the active collector stops, or cannot check the service while the standby can, the standby takes over on its own. A failure counts only once the standby has confirmed it, or after 2 minutes if it does not answer, so one server’s bad moment is not an outage. A pool of one never fails over.

Every-collector mode

Choose Every collector when each server should check the service as its own vantage point: an endpoint reached from several sites, subnets or regions. Every eligible server in the pool checks at the same moments, up to Use at most collectors if you set a limit (WatchTower15 sets none). Alert when at least N of M collectors fail decides when the service counts as unreachable or as failing its response check (by default, half of them); Response slow alerts when any one collector is slow. When some collectors reach the service and others don’t for 5 check intervals in a row, Collectors disagree warns you. There is no active collector to move, so Reassign now is off in this mode.

Parameters

A check can read values on its collectors and insert them as {{param.name}} into the path, a header value, the body or the authentication — an API key, a tenant id, a password. WatchTower15 stores where to read each value, never the value: the collector reads it when it checks, with its own identity, and sends back only the result. A service has at most 20 parameters, read from at most 10 sources.

Source	Where it reads
AWS Parameter Store	A parameter by name or ARN, decrypted, with the server’s own AWS role.
AWS Secrets Manager	A secret by ARN, with the server’s own AWS role.
File on the collector servers	A file by absolute path, read as the agent’s user: the file’s own permissions apply.
URL	An http or https URL answered within 5 seconds, with at most 64 KiB, and without a redirect.

Each parameter has a format and, except Raw, a selector that picks one value:

Format	Selector	Example
Raw	none: the whole content, one trailing newline trimmed	a file holding only a token
.env	a key	DB_PASSWORD
JSON	a path	$.credentials.password
YAML	a path, as JSON’s	$.database.password
XML	a path, or an attribute	/config/api/@user
INI	section.key	database.password

Test parameters asks the collectors to read every parameter and says, per server, whether each one resolved and why not — never the value. The Parameters section also shows what to set up for each source: the IAM statements for the collectors’ role, the setfacl grant for a file, and the services.allow lines below.

The server’s own limits, services.allow. A file outside the agent’s own /etc/wt15-server-agent/services.d/ folder is read only when the server’s /etc/wt15-server-agent/services.allow lists it. That file is yours: the agent reads it and never writes it, the installer never creates it, and WatchTower15 cannot change it. One directive per line, # for a comment:

# a file, or every file under a folder (a trailing /)
file /var/www/orders/current/.env
file /etc/orders/
# if any url line is present, only URLs under a listed prefix are fetched
url http://127.0.0.1:8200/v1/secret/
# if any destination line is present, checks go only to listed destinations,
# and only the parameters a line names are sent there
destination https://orders.internal.example.com:443 api_token tenant

A line it cannot read makes the whole file invalid, and an invalid file refuses everything it would allow: the allow-list fails closed. The agent never reads its own configuration, its state folder, /proc, /sys, /dev, /tmp or /var/tmp, whatever services.allow says.

A value with no other home. Keep it on the collector itself in the agent’s managed file:

printf '%s' "$TOKEN" | sudo /var/lib/wt15-server-agent/wt15-server-agent service-param add <service-uuid> API_TOKEN

The value is read from standard input, never shown, and written to services.d/<service-uuid>.env (readable by the agent only). Choose that file as a File parameter with the .env format and the key; it needs no services.allow line. Run it on each collector.

The service’s page

A service’s page has five tabs:

• Overview — its health and alarm counts, the response time over the page’s time range (each collector a line), and its latest checks.
• Alarms — the service’s alarms, each rule’s state and thresholds.
• Collectors — the pool, each server’s role (Active, Standby or Member) and last check, the last verify’s checklist, and the history of collector changes. Test again makes every server check once, Reassign now moves the check to the next server, and Pause stops it: nothing is checked, no alarm is evaluated and nothing is billed until you Resume.
• Info — the service’s settings at a glance.
• Audit — its part of the audit trail.

A service with no collectors yet reads Unknown — no collector until you choose them (Edit → Collectors); nothing checks it, and it is not billed.

Troubleshooting a check

A failed check names its reason. Reasons on the target’s side are what any collector would see, and they never move the check to another server; reasons on the collector’s side are that server’s alone, and two in a row move the check to the standby.

Reason	Means	Side
dns	The name does not resolve from this server.	collector
blocked_destination	The destination is refused: a link-local or metadata address, or not listed in services.allow.	collector
parameter_unavailable	A parameter could not be read (the Collectors tab names which, and why).	collector
refused	The connection was refused.	target
timeout	No answer within the timeout.	target
no_reply	No usable answer (for UDP, no reply at all).	target
tls_untrusted	The certificate is not trusted.	target
tls_expired	The certificate has expired.	target
tls_hostname	The certificate is for another name.	target
assertion	The status, the text it must contain or the content type did not match.	target
too_many_redirects	More than 10 redirects.	target
body_too_large	The response body is too large.	target
redirect_with_parameter	A redirect to another origin would have carried a parameter.	target

When every collector fails on its own side, the service reads Failing with the reason (such as its collectors cannot read a parameter), and No collector reporting opens once no usable result has arrived for 5 minutes after a check was due.

A parameter that cannot be read says why: file_not_found, permission_denied (grant the agent’s user read access, for example with setfacl -m u:wt15-server-agent:r <path>), not_allowed_locally (list it in services.allow), denied_path, AccessDenied or ParameterNotFound (AWS), format_invalid, select_not_found (with where the selector stopped), not_a_scalar or value_too_large.

A collector cut off from WatchTower15 keeps checking and keeps its results on its own disk, and delivers them when it reconnects, as it does for databases.

Server volumes

The domain checks above work from outside. On each server, the agent also reports its storage volumes — the filesystems mounted on it, such as /, /var or /data — and WatchTower15 watches every volume that holds real storage on its own: whether it is full, filling up, running out of inodes or has turned read-only. See the volume alarms.

Which volumes are monitored

A volume is monitored when it is at least 256 MiB of local storage: ext2, ext3, ext4, XFS, btrfs, ZFS, ReiserFS, JFS, F2FS, FAT (vfat, exFAT) or NTFS. WatchTower15 ignores the rest automatically:

Ignored	Why
Files a container runtime mounts into a container — /etc/hosts, /etc/hostname, /etc/resolv.conf	Single files from the host’s disk, not volumes
Shared-memory mounts (a mount point ending in /shm), and anything mounted beneath /var/lib/docker/, /var/lib/kubelet/pods/, /snap/ or /run/	Mounts a runtime creates for each container, pod or package. /var/lib/docker itself, on a disk of its own, is monitored
The EFI system partition (/boot/efi)	Boot-loader files, not storage anything fills
Filesystems that are not local storage, such as tmpfs, overlay, squashfs, ISO images and network shares (NFS, SMB)	Memory, layers, packages, or another machine’s disk
Volumes smaller than 256 MiB	Too small to be storage anyone fills
Volumes that have been read-only since WatchTower15 first saw them	Read-only by design, so nothing can fill them. A volume that is later seen writable is monitored from then on, and turning read-only after that raises Volume became read-only
Duplicates: a bind mount or a second mount of the same device, or several datasets of one ZFS pool	They share one volume’s space, so only one of them is watched and a full disk raises one alarm, not several. The one already watched stays watched; otherwise the shortest mount point, such as / rather than a bind mount of it

An ignored volume has no volume alarms at all, inodes included. Where the agent reports its space and inodes, they still appear in the server’s disk charts. You can override the choice for any volume on the server’s Storage tab.

New volumes are picked up on their own. The agent reports the server’s volumes with its system-information snapshot, every 5 minutes, so a disk attached later is monitored, with all its alarms, within about 5 minutes of being mounted. A volume that disappears, such as a detached disk, stops being watched.

The Storage tab

On a server’s page, the Storage tab lists the server’s volumes, with a summary such as 6 monitored · 35 ignored, and below them the disk charts. Each volume’s row shows its mount point and health, its device and filesystem, the share of its inodes in use where the filesystem keeps an inode count, its usage bar and percentage, and the space used out of its size. A monitored volume’s row also has Explore usage, which opens its disk usage report. Underneath, the row gives the report’s state while there is no complete report, such as Partial report, Walk scheduled or No report yet. Once a complete report is ready it says nothing: the report itself says when it was taken. The row ends with an icon that opens the volume drawer. Every time names its zone: your display timezone, or the server’s when you switch the page to it.

• The bar is coloured against that volume’s own Volume is full thresholds, which are marked on it and written under it (Warning 85 % · Critical 95 %). A volume that is not monitored stays grey whatever its usage. Disk alarm off means its Volume is full alarm is switched off, which explains a red bar with no incident.
• Read-only. A volume mounted read-only is tagged read-only.
• No data. A volume with no recent reading shows No data over an empty bar, never 0%.
• New. A volume first seen in the last 7 days is tagged New.
• Waiting. A volume WatchTower15 has just found reads Waiting for the agent’s next report until it is classified. A volume that was already over a threshold when volume alarms were introduced reads Alarms arm when this volume next reads below its thresholds, until it does (see volumes that were already there).

Pills above the list choose which volumes it shows, each with its count:

Pill	Shows
Active (the default)	Every monitored volume, and every volume waiting for the agent’s next report
Ignored	Every volume that is not monitored, greyed, with the reason — Ignored — squashfs — snap package, say: one of those in the table above, or Switched off by a user
All	Both, the active volumes first

The choice is kept in the page’s address, so a reload or a shared link keeps it. On a server whose volumes are all ignored — an agent running inside a container, for example — the list says No monitored volumes. 12 are ignored, linking to them; it never reads as a server with no disks.

Switching a volume’s monitoring on or off. With the servers.edit permission, each volume’s drawer has a Monitored switch, which overrides WatchTower15’s choice — to watch a network share, say, or to stop watching a scratch disk:

• Switching a volume on gives it its volume alarms at once, with normal alarming.
• Switching a volume off stops its alarms. Its open incidents are resolved without a notification; if it has any, you are asked to confirm first.
• A volume whose switch someone has set offers Reset to automatic, which hands the choice back to WatchTower15’s rules. One switched on is tagged overridden; one switched off reads Switched off by a user under Ignored.
• Without servers.edit, the drawer says Monitored or Not monitored instead of the switch.
• Every switch is recorded on the server’s Audit tab as Volume monitoring changed, naming the volume and the change, such as /var: On → Off. After Reset to automatic, it records the state WatchTower15’s rules chose.

The volume drawer

The icon at the end of a volume’s row — a pencil with servers.edit, an eye without it — opens the volume’s drawer over the tab, with its own address; the row itself does not. The drawer starts with the Monitored switch and Reset to automatic (above). It then shows the volume’s device, filesystem (and whether it is mounted read-only), size and use, classification — why it is or is not monitored — health and when it was first seen; then its incidents and its alarms. Hovering the device, filesystem or size shows a button that copies it. With alarms.edit you can tune, add or duplicate its alarms there, and with incidents.edit acknowledge or resolve its incidents. A volume that is not monitored lists its alarms read-only: they are inactive while it is ignored. Everyone who can view the server can read the drawer.

The disk charts

Below the volumes, the Storage tab charts the server’s storage over the page’s time range: space and inodes for each volume — Disk Usage (%), Disk Space (GB) (used solid and free dashed, in the volume’s colour) and Inode Usage (%) — then how hard each disk is working, from the agent’s telemetry:

Chart	What it shows
Disk I/O (MB/s)	Bytes read and written per second
Disk IOPS	Reads and writes completed per second. A cloud volume has an IOPS limit (3,000 by default on an AWS gp3 volume): a flat line at it is a volume being throttled
Disk latency (ms)	The average time each read or write took, waiting in the queue included. A healthy cloud volume or NVMe drive answers in about a millisecond; tens of milliseconds, sustained, is a disk that is struggling
Disk busy (%)	The share of the time the disk had at least one request in flight — see below
Disk queue depth	The average number of requests in flight. A queue that rises with latency is a disk that cannot keep up

Each disk keeps one colour, with reads solid and writes dashed. Every value is an average over the agent’s 15-second telemetry interval, so a short burst reads lower than its peak. Where a point covers several samples, it shows the busiest, or for latency the slowest, so the latency chart can read above Server disk latency is high, which scores each minute’s average. A disk that did no reads in an interval has no read latency, so its line has a gap there rather than a 0.

An ignored volume is not drawn. The space and inode charts, here and on the Overview, leave out every volume that is not monitored, whether it was excluded or switched off in its drawer, so the charts list the same volumes as the Active filter. A volume still being classified is drawn. Switch a volume back on to draw it again; the disk I/O charts count each disk whatever its volumes.

Partitions and software RAID arrays are not drawn: their I/O is already counted at the disk (see which disks are reported). The Overview tab’s Disk I/O chart is drawn the same way.

Disk busy is not saturated. Busy is the share of time the disk had at least one request in flight. A spinning disk serves one request at a time, so at 100% busy it is full. NVMe drives and cloud volumes, such as AWS EBS, serve many requests in parallel: at 100% busy they can still have plenty of room. Read busy with latency and queue depth. A disk that is 100% busy and still answers in a millisecond or two is coping; latency and queue rising together is a disk that cannot keep up. That is why disk latency has an alarm, and busy does not.

Disk IOPS, Disk latency, Disk busy and Disk queue depth need agent 1.1.28 or later, which agents install on their own; with an older agent, each says No device data — this agent predates 1.1.28.

Reading a disk report

A volume’s disk usage report shows what is filling it. Open it with Explore usage on the volume’s row on the Storage tab: it opens in a drawer over the tab, with its own address. Only monitored volumes are walked, so only they have the button. The report is read-only — nothing on the page starts a walk; see when reports are taken. Its first line says when the report was taken and how long the walk took.

The ranked list. The report opens at the top of the volume and lists its largest folders and files, each with its size and a bar. The bars compare the items at that level with each other — the largest is the full bar — not with the whole volume.

• Click a folder to go into it. The breadcrumb above the list shows where you are and takes you back up, and so does the browser’s Back button. The folder is kept in the page’s address, so a link opens the report at that folder.
• A file large enough to rank among the folders is shown with a file icon, and does not open.
• N smaller items, the last row of a level, is everything too small to list on its own, summed so that the level still adds up. It is shaded and does not open.
• A lock marks each folder in /home, and /root: contents not collected. Only their name and total size are reported (see home folders). A /home with more than 200 folders ends in N more folders.
• The search box (Find a path…) finds a folder or file in the report by its path.

Unaccounted space. Above the list, the report says how much of the volume’s used space no file or folder explains. Usually that is files that were deleted while a program still holds them open: their space is only freed when the program lets go of them, often by restarting it. The report links to Deleted but still open, under Quick culprits, which names those programs. When the figure is small — under about 2% of the used space — it is a quiet grey line. It points elsewhere in two cases:

• when the report could not see the server’s real /tmp and /var/tmp, the figure includes them, and the report names them rather than pointing at deleted files;
• when part of the volume sits beneath another mount or a subvolume that the walk does not enter, the report says so.

Mounts inside the volume read as notes, such as there is also a separate volume at /data, or a bind mount counted at its source. System and container mounts are counted in a single line, never listed.

The panels.

• Quick culprits: the space taken by Docker, journald (the systemd journal), Package caches and Old kernels, each with its largest paths, and Deleted but still open: deleted files a program still holds open, by program name, never by process ID.
• Largest files and Recently changed files (last 7 days), with their full paths. Inside a locked home folder, a path stops at the folder, followed by file names not collected.

File names are shown exactly as they are on the server, as plain text. Invisible control characters and characters that reverse the direction of text are shown as visible codes, such as \u202E, so a file name cannot disguise itself.

What the drawer says while a report is due. A line above the report says where the walk stands; when it failed or ran out of time, the previous report is shown below it.

You see	Means
No report for /data yet	The volume has not been walked. One is taken when a volume is first seen, every day at about 04:30 UTC, and when a disk alarm opens on it
Scheduled for 04:52 BST	A walk is booked for that time, in your display timezone
Waiting for the agent — it picks up new tasks within ~30 s	The time has come, and the agent has not started yet. The line says when the agent was last seen
Running — started 12 s ago	The agent is walking the volume; a walk takes seconds to a few minutes, and the drawer updates on its own. A server walks one volume at a time, so Still running — this server may be walking another volume first can follow
Partial: the walk stopped at …	The walk stopped at a limit, which the line names; the report shows what it found, and its totals still add up
This filesystem is not walked	A network share or other non-storage filesystem that a user switched on as monitored. It is never walked, so this recurs every day
The last walk failed: this path is not a mounted directory now	The volume was not mounted when the walk started
The walk failed on the host	The walk could not read the volume, with the reason’s code
No result in time	No result arrived within an hour of the walk’s time
This report could not be read	The stored report is damaged or in a form the page does not know. It does not mean the volume is empty
This server’s agent does not produce disk reports yet	The agent is older than 1.1.24; reports start once it updates itself

Reports and alarms. When a report was taken because a disk alarm opened, the drawer says so (Walked because …) and names that incident. A volume is walked at most once every 6 hours, so an alarm can open after the latest report: the drawer then says how much older the report is, for example taken 3 h before this alarm opened.

The Network tab

On a server’s page, the Network tab shows, from top to bottom, the server’s network interfaces, its listening ports and its network charts.

Network interfaces

The list has one row per interface the agent reports — loopback and the interfaces a container runtime creates for each container are left out (see which interfaces are reported) — under a count such as 2 up · 1 down. Each row shows the interface’s name, MAC address, IPv4 address and mask, broadcast address and IPv6 address, then:

Column	Shows
Link	Whether the interface has a link (a carrier): a green Up, a red Down, or a dash where it is not reported
In now, Out now	The latest receive and transmit rate in the page’s time range
Errors now	Receive and transmit errors per second, added together, at the latest sample
Drops now	Receive and transmit packets dropped per second, added together, at the latest sample

Hovering over a rate says when it was sampled. A dash means there is no sample: for Errors now and Drops now, for example, an agent older than 1.1.28.

Link is not the dot. The dot and the down tag before an interface’s name are its administrative state, from the agent’s system-information snapshot every 5 minutes: whether it is switched on. Link is whether it has a carrier, from telemetry. The two can disagree: a cable pulled from a switched-on interface shows a green dot with Link: Down, and that is what Server network link is down alarms on. An interface switched off on purpose is greyed and tagged down, with a dash in every rate and in Link: switching it off is not a fault.

Link is reported for physical network cards, bonds and VLAN interfaces that are switched on, and for a VLAN only while the interface it runs on is switched on too. A bridge, docker0, a tunnel, a VLAN on a switched-off interface, and an interface whose link state is not known or has not been reported for 5 minutes show a dash (see link state). The column can trail the real state by up to about two minutes.

Network charts

Below the listening ports, Network telemetry charts the server’s interfaces over the time range you choose:

Chart	What it shows
Network	Bytes received (solid) and sent (dashed) per second, one colour per interface
Network errors (/s)	Errors per second, one line per interface, receive solid and transmit dashed. Errors point at a failing network card, a bad cable or a misconfigured link; sustained, they raise Server network interface has errors
Dropped packets (/s)	Packets dropped per second, drawn the same way. Some drivers count frames they ignore, such as those of an unknown protocol, as drops, so a steady low rate can be normal. Dropped packets have no alarm
TCP connections	Connections by state: Established (which counts connections in CLOSE_WAIT too), Time-wait and Orphan
TCP retransmits (%)	The share of segments sent that were retransmissions: a cheap measure of packet loss between this server and its peers. It has no alarm, because retransmits to internet clients depend on the clients’ networks
Connection problems (/s)	Listen overflows (incoming connections dropped because the application’s accept queue was full), Failed attempts (connections this server opened that failed) and UDP buffer errors (datagrams dropped on a full socket buffer)
Connection tracking (%)	How full the kernel’s connection-tracking table is. On a server without connection tracking the card says Connection tracking is not loaded on this server

Every value is an average over the agent’s 15-second telemetry interval, so a short burst reads lower than its peak. Network errors and Dropped packets need agent 1.1.28 or later; with an older agent, each says No device data — this agent predates 1.1.28. The four TCP charts need agent 1.1.31 or later (No TCP data — this agent predates 1.1.31.). They count the server’s own network namespace: a container with a network namespace of its own keeps its own TCP figures.

When a listen overflow alarm fires (Server is dropping incoming connections), the application accepts connections more slowly than they arrive, or its backlog is too small. Raise the listening socket’s backlog (nginx’s listen … backlog=, the application’s own setting, capped by net.core.somaxconn), or add workers.

When the connection-tracking alarm fires (Server connection tracking table is filling), raise net.netfilter.nf_conntrack_max (with sysctl, and in /etc/sysctl.d/ to keep it), or reduce what is tracked. A full table drops new connections with nothing in any application log.

Listening ports

The agent reports the ports each server listens on. On a server’s page, the Network tab lists them below the network interfaces, one row per port:

• the protocol and port, where it listens — all interfaces, the server itself only (loopback), or particular addresses, which it names — its IP families and the process listening;
• whether it is open or closed, and since when, when it was first seen, and its health;
• a New tag while its New port opened incident is current, and Alarms off when its alarms are switched off;
• a history strip over the page’s time range — open, partly open, closed, or no data — and the share of the range it was open.

Ephemeral ports — ports the kernel hands out at random, see ephemeral port — are tagged ephemeral and listed without a history. They are dropped 24 hours after they close, unless you track them, when the tag reads ephemeral · tracked.

Filter the list by Open, Closed or All. Show retired adds the ports retired after 30 days closed, so how long a port was open stays answerable for every port the server has had.

The list says when it cannot be complete: when the server’s agent is too old to report ports (older than 1.1.21), when no list has arrived yet or the latest one is old (The last port list is from 3 h ago), or when the server listens on more ports than are tracked — 512 — in which case new ones are not added until one retires. If the port history cannot be read for a moment, the list says History is unavailable right now and shows the ports without their strips; it never shows them as closed.

The port drawer. Clicking a port opens its drawer, with its own address: the port, where it listens, its process, its state and since when, when it was first seen, its history and availability over the page’s time range, its two alarms and its incidents. It also offers these actions:

Action	Needs	What it does
Forget this port	servers.edit	For a closed port only — an open one would come back with the next report, so on an open port the drawer says Forget is offered once the port has closed. Retires the port now instead of 30 days after it closed, and resolves its open incidents without a notification. Its history stays, under Show retired. If it listens again, it counts as a new port and raises New port opened. An ephemeral port you have not tracked is removed altogether
Ignore (alarms off)	alarms.edit	Switches the port’s alarms off; Alarms off — turn on switches them back. The port stays listed, greyed, with Alarms off, and keeps its history and incidents
Mark expected	servers.edit	Says the port must keep listening; the button then reads Expected, and clicking it again clears it. If the port then stops, Port stopped listening alarms at once, instead of only for a port that had been listening for 24 hours. It applies from the next closure. Not offered on an ephemeral port until you track it
Track this port	servers.edit	For an ephemeral port only; the button then reads Tracked. Treats it like any other port: it gets a history and both alarms, and is kept after it closes. It counts towards the 512 tracked ports. Stopping tracking makes it an ephemeral port again: its open incidents resolve without a notification, and it is dropped 24 hours after it closes

Forget this port, Mark expected and Track this port are recorded on the server’s Audit tab as Port forgotten and Port settings changed. Ignore is recorded as an alarm change. A retired port offers no actions. Everyone who can view the server can read the list and the drawer.

System services

The agent reports the server’s systemd services. On a server’s page, the System services tab lists them. Monitored, the default, shows the watched services; All lists every service the agent reported, and Needs attention only the watched services with an alarm firing. Filter by name… narrows the list, and a line above it sums it up, for example 12 monitored · 2 need attention · 131 on this server.

Each row shows:

• the service’s name and description;
• its state — Running, Active (an active service that is not a running process, such as a one-shot service that stays active, with its sub-state, for example exited), Failed, Stopped, Starting or Not present — and since when;
• its start-up setting, such as enabled or disabled;
• an uptime strip over the page’s time range — up, partly up or restarted, down, or no data — and the share of the range it was up;
• the memory it uses now, as systemctl status shows it — a dash, never 0, when systemd does not account memory for it;
• its CPU now, as a share of one core (a busy multi-threaded service can read above 100 %). A dash says why when you point at it: the service is not running, the agent is older than 1.1.32, CPU accounting is off for the service, or there was no reading in the last 5 minutes;
• how many times systemd restarted it within the range;
• whether it is watched.

A service near one of its limits carries one tag after its name, at the level its alarm warns at, the most serious first: out-of-memory kills in the time range (for example 2 OOM kills), then memory at 85 % or more of its MemoryMax= (memory 91 % of limit), tasks at 80 % or more of its TasksMax=, then CPU throttled in half its periods or more. The tag reads the latest values, so it shows even when you switched the alarm off.

A service that is not watched is greyed and shows its state from the agent’s latest list, no more than 5 minutes old, with a grey strip and No history: not monitored · state as of 14:27 BST: history is kept only for watched services. If the history cannot be read for a moment, the tab says History is unavailable right now and shows the services without their strips, memory and restarts. Manage on profile → opens the server’s profile, where collection is switched on or off.

When the tab has nothing to show, it says why: the server’s agent is too old (older than 1.1.22), the host does not run systemd, systemd could not be reached, the server’s profile does not collect system services, or none of its services is watched.

The service drawer. Clicking a service opens its drawer, with its own address. It starts with a summary laid out as systemctl status prints it, under the service’s name and description:

• Loaded: whether systemd loaded the service, its unit file, and its start-up setting;
• Active: its state, since when and how long ago;
• Main PID with its process name, and Tasks with their limit;
• Memory, now, and CPU, the CPU time it has used;
• Restarts: systemd’s own count, which a manual systemctl restart resets;
• CGroup: its processes, each with its process ID and name — up to 16, then … and N more. When the agent counted the processes but could not name them in time, it shows how many there are.

Under the summary, Activation says how the service runs — as a daemon, once, or started by a timer, socket, path or D-Bus — and what down means for it (see system service alarms).

Main PID, Tasks, CPU and the processes come with the agent’s list of services, every 5 minutes, so the drawer says when they were reported, for example Main PID, Tasks, CPU and processes as of 14:27 BST, from the last inventory. Memory and restarts are measured every 15 seconds. The summary needs agent 1.1.26 or later; a line the agent sent nothing for — from an older agent, an older systemd, or with accounting switched off — is left out, never shown as 0. Only process names are shown, never command lines.

For a watched service, the drawer then shows its history over the page’s time range — its uptime strip, how much of the range it was up and how often it restarted, a Restarts chart, a Memory (MB) chart, a CPU (% of one core) chart and a Service limits (%) chart — then its alarms and its incidents. Service limits draws only the lines the service has: CPU throttled under a CPUQuota=, Memory of limit (working set) under a MemoryMax= and Tasks of limit under a TasksMax=. The working set leaves out page cache the kernel can drop, so it can read lower than Memory (MB). A red OOM kills in range tag beside the state counts the kernel’s out-of-memory kills in the service over the time range. An empty chart says why: no data in the window, no limit set, CPU accounting off, or an agent older than 1.1.32. With alarms.edit you can tune its alarms there. A service that is not watched has no history and no alarms, and the drawer says so.

Performance and Capacity

The Overview’s Telemetry card has two views, chosen next to its title. Performance, where the Overview always opens, is the full set of charts. Capacity keeps the time range and shows how close the server is to its limits: CPU Usage (%), CPU per core (%), Load Averages, Processes CPU (%), Memory Usage (%), Memory Usage (MB), Disk Usage (%) and Disk Space (GB), beside a Volumes table. The view is kept in the page’s address, so a reload or a shared link opens the same view.

The Volumes table lists the server’s monitored volumes, from the latest sample whatever the time range, in GB:

Column	What it shows
Volume	The mount point; it opens the volume on the Storage tab
Current	The space used now
Max	The volume’s size
Free	The space free now
% Used	The share used, coloured by the volume’s own alarm thresholds

Ignored volumes are not listed; the line under the table counts them and links to them on the Storage tab. A figure the agent has not reported shows —.

Saturation

CPU and memory percentages say how busy a server is, not whether its work is waiting. Four charts at the bottom of a server’s Overview tab answer that, from the agent’s telemetry:

Chart	What it shows	Each point
CPU by mode (%)	Where CPU time went — user, nice, system, IRQ, SoftIRQ, I/O wait and steal — stacked, as a share of all cores	The step’s average
Pressure stall (%)	The share of time in which at least one task (some), or every non-idle task at once (full, dashed), was stalled waiting for the CPU, memory or I/O	The step’s peak
Memory activity (/s)	Pages swapped in and out per second, and major page faults (pages read back from disk) per second	The step’s peak
Tasks	Threads running or ready to run, and threads waiting on I/O	The step’s peak

The CPU Usage (%) chart adds Busiest core and Quietest core: one core near 100% while the whole machine is quiet means a single-threaded process is the limit. That chart counts I/O wait as idle and shows each step’s peak, so it can differ from the top of CPU by mode.

CPU per core (%), next to CPU Usage (%) on the Overview, draws every core’s own line, each step’s peak, for the first 64 cores by the kernel’s numbering (Busiest core and Quietest core still cover every core). Up to eight cores each have a colour; above eight every core is one thin line in the same colour, so a pinned core shows as the top line, and the tooltip names the 16 busiest cores. It needs agent 1.1.53 or later, which agents install on their own; with an older agent the card says No per-core data — this agent predates 1.1.53. Beyond 30 days its points are hourly, like the other charts, and they begin when the server’s agent reached 1.1.53. The Memory Usage (%) chart shows a red tag, such as 3 OOM kills, when the out-of-memory killer killed processes in the time range; its tooltip lists when.

What each signal means, and when it needs action:

• Steal is time the hypervisor gave to other virtual machines while this server was ready to run. A little, now and then, is normal on shared hardware. Sustained steal means a busy neighbour on the same physical host or, on a burstable instance in standard credit mode, CPU credits running out: the server then shows close to 100% steal, while the cloud console shows only the instance’s baseline CPU. When Server CPU steal is high fires, move to a larger or non-burstable instance type, or give a burstable instance unlimited credits.
• Pressure is the share of time work waited for a resource instead of running. Low single figures are normal on a busy server; act when it stays high, as the pressure alarms do at 20% for three minutes in a row. CPU pressure means more runnable work than cores: find the busy process on the Processes tab, or add cores. Memory pressure means the server is short of memory and is reclaiming or swapping; it rises before the out-of-memory killer runs, so add memory or run less before processes are killed. I/O full means nothing on the server could make progress because of storage: the Storage tab’s disk charts show which disk, as its latency and queue depth rise together. I/O some on its own is routine on a busy database.
• Tasks waiting on I/O are threads blocked until an I/O request, usually a disk’s, completes. The load average counts them as well as running threads, so a high load with idle CPUs often means slow storage, not busy cores. A rise during a backup or a deploy is normal; a count that stays up, with I/O pressure, points at the disk.
• I/O wait, in CPU by mode, is idle CPU time while a disk request was pending. It falls as cores are added and says little about how long work waited, so it has no alarm: I/O pressure answers that question.
• Memory activity: constant swap-in is thrashing; a full swap with no swap-in is not.

When the charts are empty. They need agent 1.1.27 or later, which agents install on their own; with an older agent each says No saturation data — this agent predates 1.1.27. A server whose kernel does not report pressure stall information shows No pressure data on the Pressure stall chart only.

Long time ranges. For a time range that starts more than 30 days ago, these four charts, like the server’s other Overview charts, show hourly or coarser points. Their history starts when the server’s agent began reporting them.

Kernel limits

The Processes tab shows how close a server is to the limits behind “Too many open files” and “Cannot fork”, from the agent’s kernel limits (agent 1.1.31 or later):

Chart or column	What it shows
Limits (%)	Threads against the server’s limit on process IDs, File handles (host) against fs.file-max, and Open files (worst process): the process nearest its own open-file limit. Where the server has no host-wide file-handle limit, as most current Linux servers have none, that line is absent and the card says No host file limit. The card also names the worst process group, such as Worst: nginx (www-data) 92 %
Zombie processes	Processes that have exited but whose parent has not collected them
Open files column	Each process group’s open files as a share of its own limit. It sorts like the other columns; a group close to its limit is always listed, however little memory or CPU it uses

What each limit means, and how to raise it:

• A process’s open files — its soft ulimit -n. A process at its limit fails with “Too many open files”. For a systemd service raise LimitNOFILE= in the unit (systemctl edit <unit>, then restart it); for a login session, /etc/security/limits.conf. Also look for a leak: a share that only ever climbs is one. A process is running out of file descriptors watches it.
• The host’s file handles — fs.file-max. Most current distributions leave it effectively unlimited, so there is nothing to watch. Where it is set, raise it with sysctl -w fs.file-max=… and in /etc/sysctl.d/.
• Threads — the smaller of kernel.pid_max and kernel.threads-max. At the limit the server cannot start a process or a thread (“Cannot fork”). threads-max grows with memory, so a small instance reaches it first. Raise kernel.pid_max or kernel.threads-max with sysctl, or find what is spawning threads. Zombies each hold a process ID until their parent collects them, so many of them count against this limit.

Gaps in a server’s charts

When a server’s data did not reach WatchTower15 for a while, the charts on its page mark that time by its cause, and show what became of the data. Every chart on the server’s Overview (its Telemetry card), Storage, Network and Processes tabs, and in a system service’s drawer, is marked. Custom metric charts, dashboard widgets and the log histogram are not.

Mark	Looks like	Means
WatchTower15 not receiving data from most agents	Amber	WatchTower15 was not receiving data from most of its agents at the time (the interruption notice). It says nothing about whose side the fault was on.
WatchTower15 outage — not your server	Amber	WatchTower15 answered this server’s agent with an error (5xx) or with Too many requests (429).
No connection from the host	Slate stripes	The agent was running but could not connect. The tooltip names the error, such as DNS lookup failed.
No report from the agent	Slate stripes	The agent has not said why.
Rejected by WatchTower15	Slate cross-hatch	WatchTower15 refused the agent’s data because the host’s clock was more than 5 minutes off.
Host down, Agent not running, Powered off	Solid slate	Nothing was collected: the host was down or powered off, or its agent was not running.
Held, paused or awaiting approval — nothing collected on purpose	Blue stripes	Monitoring was switched off or held, the agent’s token was revoked, or the server was waiting for approval.
Not reaching WatchTower15 — the cause is known on reconnect	Slate stripes	The server is unreachable now: from its last heartbeat to now. While WatchTower15 is rejecting it, this band reads Rejected by WatchTower15 · host clock, cross-hatched; while WatchTower15 is not receiving data from most agents, it is that amber band.
Catching up — still on the host	Light blue	Data the agent kept on the host while it could not reach WatchTower15, still being delivered.
Delivered late	A thin line along the bottom of the chart	Data the agent kept on the host and delivered once it could: it is on the chart.
Data lost	A red wash over the cause	Data that will not arrive: dropped on the host, kept but not delivered within the time the agent keeps data, or, while the server is unreachable, collected by an agent that is not keeping data on the host.

• A gap whose data all arrived has no band: the data is there. What arrived late carries the Delivered late line only.
• Time when nothing was collected has no red wash, as there was nothing to lose: a host that was down or powered off, an agent that was not running, a server that was held or paused. The Lost banner still counts the time a host was down or its agent was not running.
• These charts show the agent’s telemetry. When the agent kept checking in but some of its data could not be stored, they mark that time only if telemetry was part of it. A fault that hit only database snapshots, disk usage reports or system information draws nothing on them and shows no note on a window of your own (below). Data of those kinds dropped on the host still counts in the Lost banner and in the Agent card’s Dropped on the host, without a red wash.

The tooltip. Hovering over a mark names it, with its time window, even where the chart has no values. Inside a red wash, the cause adds · data lost: No connection from the host · DNS lookup failed · data lost · 06/10/2026, 09:12 – 09:52 UTC, then Delivered late · 06/10/2026, 09:52 – 11:25 UTC.

The legend. A key above the charts lists the marks in the time window, and only those.

A window inside a gap. On a time window that falls entirely inside a gap, such as dates picked inside it, or the last hour of an outage that is still going on, the charts still draw the band, the legend and the tooltip rather than reading empty. A chart that has its own message when the agent reports no such data, such as the per-disk, saturation or kernel-limit charts, keeps that message, and the legend above it names the gap.

A window of your own dates. A relative window, such as the last 3 hours, is read again every minute and fills in as the data arrives. A window of dates picked on the calendar is not, so while data inside it is still on the host, a note under the time range says Data for this window is still arriving (1 h 40 min of buffered data still on the host). The charts fill in when you reload. The time in brackets is all the data the server still has to deliver, not only this window’s. A minute or two after it has arrived, the note reads The buffered data for this window has arrived. with Reload, which loads the charts again. It also says so when data the agent kept reaches a window you picked while the server was out of reach. The note shows on the Overview’s Telemetry card and on the Storage, Network, Processes and System services tabs.

The marks use the same colours as a database’s gaps.

Related

• Alarms & alerting — turning these findings, and each volume’s state, into incidents.
• The server’s page — the tabs that show a server’s volumes, disk and network charts, interfaces, ports, system services and saturation charts, and the banners that say what happened in a gap.
• Metrics & telemetry — the metrics behind the saturation, disk and network charts, which systems report pressure stall information, and which interfaces report link state.
• Security scanning — the certificate/DNS/expiry checks framed as security posture.
• Core concepts — the status/outcome model.