Metrics & telemetry

The agent reads standard /proc and /sys paths through the gopsutil library, as its unprivileged service user. Host metrics come from world-readable interfaces. Per-process metrics for processes owned by other users rely on the agent’s two read capabilities, CAP_SYS_PTRACE and CAP_DAC_READ_SEARCH (see the privilege model). Nothing here needs root.

There are two collection cadences: a lightweight telemetry push every 15 seconds, and a heavier system-information snapshot every 300 seconds. Both, and the per-process limit, are set by WatchTower15. The disk usage report, which shows what is filling each volume, has no cadence of its own: it is taken about once a day per volume, and when a volume needs a look.

Collection settings

How often the agent reports, and how many processes it reports on, are the same on every profile. WatchTower15 sets them, and they are not shown on or changed from the profile pages:

Setting	Value	What it means
Telemetry interval	15 seconds	How often each host pushes the telemetry sample, including per-process metrics.
System-information interval	300 seconds	How often the fuller system-information snapshot is pushed.
Per-process metric limit	30 processes	Each host reports its top 30 processes by memory and its top 30 by CPU. Everything below that cut is summed into one summary row, so it is still counted.

If a profile needs a different value, contact support. WatchTower15 reviews each request and applies an agreed change to that one profile; its agents pick it up on their next configuration poll.

Telemetry — every 15 seconds

A compact, high-frequency vitals sample for time-series charts and metric alarms. The agent sends it as OpenTelemetry (OTLP) metrics:

Metric	Unit	Attributes	Description
system.cpu.utilization	%	—	Aggregate CPU utilisation across all cores. Steal counts as busy, and I/O wait as idle
system.cpu.mode.utilization	%	state: user, nice, system, iowait, irq, softirq, steal	CPU time by mode, as a share of all cores. Idle is left out: it is 100 minus the rest. Guest time is counted inside user and nice, as the kernel counts it. steal is time the hypervisor gave to other virtual machines while this host was ready to run; iowait is idle time while a disk request was pending
system.cpu.core.utilization.max / .min	%	—	The busiest and the quietest logical core’s utilisation. One core pinned at 100% while the others idle shows here, where the whole-machine figure averages it away
system.cpu.core.utilization	%	cpu: cpu0, cpu1, …	Each logical core’s utilisation, for the first 64 cores by the kernel’s numbering, so a host keeps the same set of series; .max and .min still cover every core. Agent 1.1.53 and newer
system.cpu.logical.count	cores	—	Number of logical CPUs
system.cpu.load_average.1m / .5m / .15m	—	—	Load averages
system.processes.running	threads	—	Threads running or ready to run, at the moment of the sample. It includes the agent’s own thread reading it, so it never falls below 1
system.processes.blocked	threads	—	Threads waiting for I/O to complete, usually on a disk, at the moment of the sample. The load average counts these as well as running threads
system.pressure.stall	%	type: cpu, memory, io; state: some, full	Pressure stall information: the share of the time in which at least one task (some), or every non-idle task at once (full), was stalled waiting for the CPU, memory or I/O. There is no cpu full line. Not reported when the host’s kernel does not provide pressure stall information: see supported systems
system.memory.usage	bytes	state: used, free, cached	Memory by state. free is memory available to a new workload (MemAvailable), which includes reclaimable page cache
system.memory.utilization	%	—	Memory in use as a percentage of total
system.memory.oom_kills	kills	—	Processes the kernel’s out-of-memory killer has killed since the host booted, as the kernel counts them (oom_kill in /proc/vmstat). Every kill counts, including kills inside a container or service that reached its own memory limit. It starts again from 0 when the host reboots, and kills while the agent was stopped or restarting show at its next push. Not reported when the kernel keeps no such count (Linux older than 4.13)
system.memory.page_faults.major.rate	faults/s	—	Major page faults per second: pages that had to be read back from disk or swap
system.paging.usage	bytes	state: used, free	Swap by state. Not reported on a host with no swap configured
system.paging.utilization	%	—	Swap in use as a percentage of total swap
system.paging.operations.rate	pages/s	direction: in, out	Pages swapped in and out per second. Reported on a host with no swap too, where it reads 0
system.filesystem.usage	bytes	mountpoint, state: used, free	Space per mounted filesystem
system.filesystem.utilization	%	mountpoint	Space in use per mounted filesystem, the same figure df reports
system.filesystem.inodes.usage	inodes	mountpoint, state: used, free	Inodes per mounted filesystem. Not reported for a filesystem that keeps no inode count
system.filesystem.inodes.utilization	%	mountpoint	Inodes in use per mounted filesystem, the same figure df -i reports
system.filesystem.read_only	—	mountpoint	1 when the filesystem is mounted read-only, 0 when it is read-write. Not reported when the agent cannot read the mount’s options
system.filesystem.mounts.omitted	filesystems	—	How many filesystems the 50-filesystem limit left out. 0 on almost every host
system.disk.io.rate	bytes/s	direction: read, write	Disk throughput, all reported disks. Partitions and software RAID (md) arrays are left out: their I/O is already counted at the disk
system.disk.device.io.rate	bytes/s	device, direction	Disk throughput per disk. Partitions and md arrays are left out, as in the total
system.disk.device.operations.rate	operations/s	device, direction: read, write	Reads and writes completed per second, per disk (IOPS)
system.disk.device.utilization	%	device	How busy each disk is: the share of the interval in which it had at least one request in flight. On NVMe drives and cloud volumes, which serve many requests at once, 100% means never idle, not saturated
system.disk.device.await	ms	device, direction	Average time per completed read or write, waiting in the queue included, per disk — what iostat reports as r_await and w_await. Not reported for a direction with no operations in the interval: no operations means no latency, not a latency of 0
system.disk.device.queue.depth	requests	device	Average number of requests in flight on each disk over the interval — what iostat reports as aqu-sz
system.disk.devices.omitted	disks	—	How many disks the 32-disk limit left out of the per-disk metrics. 0 on almost every host
system.network.io.rate	bytes/s	direction: receive, transmit	Network throughput, all reported interfaces
system.network.device.io.rate	bytes/s	device, direction	Network throughput per interface
system.network.device.errors.rate	errors/s	device, direction: receive, transmit	Receive and transmit errors per second, per interface
system.network.device.dropped.rate	packets/s	device, direction	Packets dropped per second, per interface. Some drivers count frames they ignore, such as those of an unknown protocol, as drops
system.network.device.up	—	device	1 when the interface has a link, 0 when it has lost it. Reported only for physical network cards, bonds and VLAN interfaces that are switched on: see link state
system.network.devices.omitted	interfaces	—	How many interfaces the 32-interface limit left out of the per-interface rates. 0 on almost every host
system.network.tcp.connections	connections	state: established, time_wait, orphan	TCP connections by state, in the host’s network namespace. established counts connections in CLOSE_WAIT too
system.network.tcp.retransmit.ratio	%	—	The share of TCP segments sent that were retransmissions. Not reported for an interval in which the host sent none
system.network.tcp.listen_overflows.rate	overflows/s	—	Incoming connections dropped per second because a listening socket’s accept queue was full
system.network.tcp.failed_attempts.rate	attempts/s	—	Connections this host opened that failed, per second
system.network.udp.receive_buffer_errors.rate	errors/s	—	UDP datagrams dropped per second because a socket’s receive buffer was full (DNS, syslog, StatsD)
system.network.conntrack.usage	entries	—	Entries in the kernel’s connection-tracking table. Not reported when connection tracking is not loaded
system.network.conntrack.utilization	%	—	The connection-tracking table’s fill against nf_conntrack_max. Not reported when connection tracking is not loaded
system.processes.count	—	—	Total running processes
system.processes.zombies	processes	—	Processes that have exited but whose parent has not collected them. Each holds a process ID until it does. Not reported for an interval whose open-files scan ran out of time
system.processes.file_descriptors.max_utilization	%	—	The process nearest its own open-file limit: its open files as a share of its soft ulimit -n. Processes with no limit are left out. Not reported for an interval whose open-files scan ran out of time
system.filedescriptors.usage	file handles	—	File handles open on the host
system.filedescriptors.utilization	%	—	File handles open as a share of the host’s limit, fs.file-max. Not reported when the host has no such limit, as most current Linux hosts have none
system.threads.usage	threads	—	Every thread on the host
system.threads.utilization	%	—	Threads as a share of the host’s limit on process IDs: the smaller of kernel.pid_max and kernel.threads-max

The same push carries the state of the host’s system services, three metrics per watched service.

The CPU modes, the busiest and quietest core, each core’s own figure, pressure stall and the per-second rates each cover the whole interval since the previous push, so a short spike between two pushes is counted rather than missed. The first push after the agent starts has none of them yet. They, the out-of-memory kill count and the two thread counts need agent 1.1.27 or later, which agents install on their own. The server’s saturation charts draw them. Each core’s own figure needs agent 1.1.53 or later; the CPU per core (%) chart draws it.

Every disk and network figure is an average over the interval. Throughput, operations, errors and drops per second, and each disk’s busy share, latency and queue depth, cover the whole 15 seconds since the previous push, not a snapshot. A burst anywhere in the interval counts for its share, so a short spike reads lower than its peak: 60 MB/s for 2 seconds of an otherwise quiet interval reads as 8 MB/s. The first push after the agent starts has none of them yet, and a disk or interface that appears is measured from its second push. These averages, the per-disk operations, busy share, latency and queue depth, the per-interface errors, drops and link state, and the 32-device limits need agent 1.1.28 or later. An older agent measured one second in each interval, so its disk and network peaks read higher, and it counted each partition beside its disk. The server’s Storage tab and Network tab draw them.

Which filesystems are reported

The agent reports real storage and leaves out mounts that are not:

• Pseudo and in-memory filesystems, such as proc, sysfs, tmpfs, overlay and squashfs (snap packages), and anything on a loop device.
• Mounts that a runtime creates for each workload. These are anything mounted beneath /var/lib/docker/, /var/lib/kubelet/pods/, /snap/ or /run/, and any mount point ending in /shm. The directory itself on its own volume is still reported — for example, /var/lib/docker on a dedicated disk.

Telemetry covers at most 50 filesystems per host. / is always included, and the rest are the largest by size. A host with more reports how many it left out as system.filesystem.mounts.omitted. The smaller filesystems still appear in the per-disk inventory, with their usage shown as unknown.

Inode metrics, the per-workload exclusions and the 50-filesystem limit need agent 1.1.19 or later, and the read-only state needs agent 1.1.23 or later.

WatchTower15 alarms on a narrower set than the agent reports: see which volumes are monitored.

Inodes

Every file, directory and symlink uses one inode. A filesystem that runs out of inodes cannot create files. Writes fail with “No space left on device”, even while df -h shows free space. The usual cause is a directory that collects millions of small files, such as PHP sessions that are no longer being cleaned up.

The agent reports inode usage for each filesystem, from the same reading as disk space. What the total means depends on the filesystem:

Filesystem	Inode total
ext2, ext3, ext4	Fixed when the filesystem is created. WatchTower15 records it as the filesystem’s inode limit
XFS, ZFS	Dynamic. Inodes are allocated as needed, so the total grows as the filesystem fills and is not a limit. Usage can fall without anything being deleted
btrfs, FAT, exFAT	Not tracked. These filesystems keep no inode count, so nothing is reported and they are never alarmed on

On a server’s page, the Inode Usage (%) chart, on the Overview and Storage tabs, draws one line per mounted filesystem. Each filesystem keeps the same colour in every disk chart. A bind mount (the same filesystem at a second path) draws a second, identical line. In the Storage tab’s list of volumes, each volume shows the share of its inodes in use beside its space. A volume shows no inode figure when its filesystem keeps no inode count (btrfs, FAT or exFAT), or when no inode figures have been received, for example from an agent older than 1.1.19. Each monitored volume’s Volume is running out of inodes alarm watches its inodes.

Pressure stall information

Pressure stall information is the Linux kernel’s own measure of how much time work spent waiting for the CPU, for memory, or for I/O, rather than running. CPU and memory percentages say how busy a host is; pressure says whether that is slowing anything down. A host can read 100% CPU with nothing waiting, or 40% with every request queued behind memory reclaim.

The host’s kernel must provide it and have it switched on. Linux has provided it since 4.20; some distributions build it in but ship it switched off. Where it is missing, system.pressure.stall is not reported at all — never reported as 0, which would read as “no pressure” — and the three Server … pressure is high alarms never fire on that host. Every other metric on this page is still reported, the out-of-memory kill count included.

System	Pressure stall information
Amazon Linux 2023	Reported
Ubuntu 22.04 and 24.04	Reported
Debian 12	Reported
Amazon Linux 2 (kernel 5.10)	Not reported: the stock kernel does not provide it
RHEL 8 and RHEL 9 (kernels 4.18 and 5.14)	Not reported unless the kernel is booted with psi=1. The kernel has the feature but ships it switched off. Tested on Rocky Linux 8 and 9, which build the same kernels

Inside a container, the agent reports pressure only when the container can read /proc/pressure.

Which disks and interfaces are reported

The kernel counts a disk request at every layer it passes through: a partition, a software RAID array or an LVM volume, and the disk beneath. The agent counts it once, at the disk:

• Partitions, such as nvme0n1p1 or sda1, are left out: their I/O is already part of their disk’s.
• Software RAID arrays (md…) and LVM and other device-mapper volumes (dm-…) are left out: their I/O is counted at the disks they are built on.
• Loop devices (snap packages and disk images), RAM disks (ram…, zram…) and optical and floppy drives (sr…, fd…) are left out.

So system.disk.io.rate is the I/O that reached your disks, counted once: the sum of its disks. An agent older than 1.1.28 counted each partition beside its disk, so a partitioned disk — the root disk of a typical cloud server — was counted twice, and its total read about double the real figure.

Network interfaces are reported except loopback and the interfaces a container runtime creates for each container or pod: veth…, Docker’s br-… networks, and cali…, cni…, lxc… and vif…. A bridge you configured, such as br0, and docker0 are reported, with their throughput, errors and drops, but have no link state.

At most 32 disks and 32 interfaces per host carry per-device series: the busiest, by the bytes they moved in the interval, with ties in name order (nvme2n1 before nvme10n1). A host with more reports how many it left out as system.disk.devices.omitted and system.network.devices.omitted. Nothing is lost from the totals: system.disk.io.rate and system.network.io.rate still count every disk and interface. Which devices are kept follows the load from push to push, and one that becomes one of the busiest is reported in that same push. Link state is not limited: it is reported for every interface that has it.

Kernel limits

The limits behind “Too many open files” and “Cannot fork”, and how close the host is to them. They need agent 1.1.31 or later and the processes telemetry group, and the server’s Processes tab draws them.

• Open files are counted per process, against the process’s own limit. On a current Linux host the limit that bites is each process’s soft ulimit -n (LimitNOFILE= in a systemd unit), not the host-wide fs.file-max, so system.processes.file_descriptors.max_utilization is the figure to watch. The agent counts each process’s open file descriptors and reads its limit. It counts them and nothing else: no file name, socket address or command line is read or sent. Agent 1.1.40 or later also says which process is nearest its limit, by its process ID and its short command name (below), so the alarm can name it.
• The count has a time budget of half a second per push, so a host with millions of open files never delays its telemetry. If the scan runs out of time, that push carries neither the worst process’s share nor the zombie count, rather than a figure that could miss the worst process. On typical servers it takes a few milliseconds.
• system.filedescriptors.utilization is not reported where fs.file-max is effectively unlimited, as systemd 240 and later set it. There is no host-wide limit to approach, and a share of an unlimited figure would read 0 % for ever. The usage is still reported.
• Threads are measured against the smaller of kernel.pid_max and kernel.threads-max. threads-max scales with memory, so on a small instance it is the limit reached first.

TCP and connection tracking

TCP, UDP and connection-tracking figures need agent 1.1.31 or later and the network telemetry group, and the server’s Network tab draws them.

• The host’s network namespace only. A container with a network namespace of its own (Docker’s default bridge mode) keeps its own TCP counters, which are not included. Connection tracking and the listen overflows of a published port are in the host’s namespace and are counted.
• Rates and the retransmit share cover the whole interval since the previous push, like the disk and network rates; the first push after the agent starts has none of them yet.
• Connection tracking is reported only where it is loaded. The kernel loads it for NAT, container networking and stateful firewall rules. A host without it reports neither system.network.conntrack.* metric, and its chart says so.

Link state

system.network.device.up says whether an interface has a link — a carrier: a cable plugged in and a switch port answering, or a cloud network card attached. It is the kernel’s own reading (operstate): up is reported as 1, down and lowerlayerdown as 0. Any other state, such as the unknown that some virtual network cards report, is not reported.

It is reported only for interfaces where a lost link is a fault:

• Physical network cards, cloud ones (such as ENA and virtio) included, bonds and VLAN interfaces. Bridges, tunnels, WireGuard and docker0 have none: they go down by design, docker0 when its last container stops.
• Switched on. An interface that is administratively down — switched off on purpose with ip link set eth1 down — has no link state, so it never reads as a lost link. A VLAN also needs the interface it runs on to be switched on: switching off eth1 does not report eth1.100 as down. A VLAN whose interface is switched on but has lost its link reads 0.
• Every such interface, whatever the 32-interface limit. The limit keeps the busiest, and an interface without a link moves nothing, so a limited link state would leave out exactly the interface that is down.

If the agent cannot read an interface’s state, it reports nothing for it, never a guess. Link state is read at every push, the first one included. A server’s Network tab shows it in the interface list, and Server network link is down alarms on it.

System information — every 300 seconds

A fuller inventory snapshot, split into a host record plus per-disk and per-interface breakdowns, the host’s listening ports, and the list of its system services.

Host snapshot

os_name, os_version, kernel_version, hostname, supplier_identifier, cpu_model, cpu_cores, total_memory_mb, uptime_seconds.

Clock

The server’s own clock, shown on its Info tab’s Clock card. timezone is on every snapshot; a field the agent cannot read is left out, and the last value sent is kept.

Field	Description
boot_time	When the host booted, in Unix seconds, by its own clock (btime in /proc/stat). The uptime WatchTower15 shows is measured from uptime_seconds on its own clock instead, so a wrong server clock never changes it
timezone	The zone’s name, e.g. Europe/London, from /etc/localtime (its link into zoneinfo, or /etc/timezone beside a copied file), or from $TZ on a host without /etc/localtime. Empty when nothing names the zone
timezone_abbreviation, utc_offset_seconds	The zone’s abbreviation and offset from UTC now, e.g. BST and 3600, read from the same source as the name
clock_synchronised	Whether the kernel holds the clock synchronised (adjtimex, read-only)
clock_max_error_ms, clock_est_error_ms	The kernel’s maximum and estimated error of the clock, in milliseconds
time_service	The time daemon systemd is running: chronyd, systemd-timesyncd, ntpd… or none. Read from systemd’s list of units, never by activating a service. Not sent inside a container
in_container	Whether the agent runs in a container (Docker, Podman, systemd-nspawn)

Connection

How the agent reaches the WatchTower15 API:

Field	Description
connection_type	direct, or proxy when the agent’s API traffic goes through a forward proxy, whether that is set with --proxy or by HTTPS_PROXY
connection_proxy_url	Only with proxy: the proxy’s scheme and host, e.g. http://proxy.corp.example
connection_proxy_port	Only with proxy: the port the agent connects to, with the scheme’s default written out (80 for http://, 443 for https://)

Proxy credentials are never sent. The agent strips any user name, password, path or query before reporting, and WatchTower15 rejects a proxy URL that contains credentials. The API returns the three fields with a server’s details, so the proxy’s host and port are visible to members of your organisation. The server’s Info tab shows them as Connection: Direct, Proxy with the proxy’s scheme, host and port, or Not reported when the fields are empty.

Agents older than 1.1.18 do not report a connection, and neither does an agent behind a proxy that is not HTTP(S), such as socks5://. For those servers, the three fields are empty, which means unknown. They are never assumed to be direct.

Per-disk (one row per mounted partition)

mount_point, device, filesystem_type, total_gb, inodes_total, read_only. How full each mount is comes from the system.filesystem.* telemetry metrics above, not from this snapshot.

inodes_total is the size of the filesystem’s inode table. The agent sends it for every filesystem that keeps an inode count. WatchTower15 keeps it only for ext2, ext3 and ext4, where it is a fixed limit (see Inodes). The inventory leaves out the same mounts as telemetry, but it is not held to the 50-filesystem limit.

read_only is whether the filesystem was mounted read-only when the snapshot was taken (agent 1.1.23 or later). It is how WatchTower15 tells a volume that is read-only by design from one that became read-only.

Per-interface (one row per non-loopback interface)

interface_name, mac_address, ipv4_address, ipv4_broadcast, mask, ipv6_address, is_up. Throughput, errors and drops come from the system.network.* telemetry metrics above. is_up is the interface’s administrative state, whether it is switched on, not whether it has a link: that is link state.

Listening ports

The ports that services on the host are listening on, so that you can see what each server exposes and be told when that changes. With every snapshot, every 5 minutes, the agent sends one entry per protocol, address and port that something is listening on:

Field	Description
protocol	tcp or udp
family	ipv4 or ipv6
address	The address the socket is bound to: 0.0.0.0 or :: for every address, 127.0.0.1 or ::1 for the server itself only, or one particular address
port	The port number
process_name	The name the kernel gives the process that owns the socket, such as nginx or sshd — at most 15 characters. Never its command line or arguments
process_started_at	When that process started

It also sends the host’s ephemeral port range (net.ipv4.ip_local_port_range), so that ports the kernel hands out at random are recognised as ephemeral.

• Listening means waiting for connections: TCP sockets in the LISTEN state, and UDP sockets bound with no fixed peer. Established connections — who is talking to whom — are not reported.
• Collected on every host running agent 1.1.21 or later. There is no setting to turn it off.
• The process’s name and start time are all that is sent about it. Command lines can carry passwords and tokens, so they are never read for this and never leave the host. Finding the owner uses the same CAP_SYS_PTRACE read capability as per-process metrics. If the agent cannot find it within two seconds, the port is still reported, without a name.
• Up to 256 entries per host. A host with more reports the first 256 and says that the list was cut.
• Where a port listens is part of what it is. A port bound to every address, to the server itself only, or to particular addresses counts as a different port each time — see bind scope.
• Inside a container, the agent sees only that container’s own ports. On a Docker host, a container’s published port is seen when Docker’s proxy (docker-proxy) listens for it on the host, or when the container uses host networking.
• A server’s ports are listed on its page, in the Network tab, with each port’s history and the actions you can take on it: see listening ports. They are watched by the port alarms.

Per-process metrics

Alongside the host vitals, the agent reports per-process usage on the same telemetry cadence. This powers the Processes tab on a server’s page: two process charts, Processes CPU (%) and Processes Memory (MB), and the Top processes table — current usage per process, with inline sparklines for CPU and memory, which you can sort and filter by name. Everything below the per-process limit is summed into the table’s Other row.

Metric	Description
process.memory.usage	Resident memory, summed per process group
process.cpu.utilization	CPU percentage, top-style — a multi-process or multi-threaded workload can exceed 100% on a multi-core host
process.count	How many processes share that group
process.file_descriptors.utilization	The group’s process nearest its own open-file limit, as a share of that limit (ulimit -n). Reported for the 10 groups closest to their limit (agent 1.1.31 or later)

Points are aggregated by (executable name, owner) rather than per PID. That is a deliberate cardinality and privacy choice: full command lines are never emitted, so a command line carrying a secret or a personal path can never leave the host, and a process ID is never used to group or label a point. The view ranks by the heaviest processes on each of memory and CPU, so a memory hog can’t hide a CPU hog or vice versa, and since agent 1.1.31 also by open files: the 10 process groups closest to their open-file limit are always listed, however little memory or CPU they use. The table’s Open files column shows their share.

Which process an alarm is about

Two metrics name one process each, so that A process is running out of file descriptors and Server killed a process (out of memory) can say which process their notifications are about. They need agent 1.1.40 or later. Each point’s value is a process ID, and its one attribute, process.executable.name, is the kernel’s command name for the process: at most 15 characters, never its command line or arguments.

Metric	Unit	Description
process.file_descriptors.max_utilization.pid	process ID	The process behind system.processes.file_descriptors.max_utilization: the one nearest its own open-file limit. When two are equally near, the one with the lower process ID. Sent with every push that carries that share
process.oom_kill.pid	process ID	For 15 minutes after the kernel’s out-of-memory killer kills a process, one point per command name killed, at most five, each with the process ID of its latest kill. Read from the kernel’s own log, so it is sent only where the agent watches system services and can read their out-of-memory kills: Linux 4.19 or later, the agent outside a container, and the system journal readable

Both are kept for 30 days, with no long-term history.

System services

The systemd services on the host — what systemctl status nginx shows — so that WatchTower15 can tell you when a service that should be running is not. A system service is a unit on one of your servers. It is not a Service, the endpoint you add under Services and your own servers check.

The agent reads them from systemd itself, over the system’s D-Bus, read-only. It asks systemd what exists and what state it is in, and nothing else: it cannot start, stop or restart a service, whatever the host’s permissions would allow. Reading this needs no extra privilege.

The list of services — with the system-information snapshot

With the snapshot, every 5 minutes, the agent sends one entry for each systemd service unit on the host, running or not, up to 512:

Field	Description
unit_name	The unit’s name, such as nginx.service. An alias is reported under the name of the unit it points to
description	The unit’s own description, such as A high performance web server, at most 255 characters
load_state, active_state, sub_state	What systemd reports for the unit, such as loaded, active, running or failed
unit_file_state	Whether the unit is installed to start: enabled, disabled, static, masked, …
activation	How the unit runs, which decides what up means for it: daemon, oneshot_remain, oneshot, timer, socket, path or dbus — see below — or none for a service that is not watched
monitored	Whether the agent watches this service
healthy	Whether a watched service is up right now
state_changed_at	When the unit last changed state. Not reported by systemd 219

For each watched service (see below), the entry also carries what systemctl status prints about it, so that the server’s page can show the same summary (agent 1.1.26 or later):

Field	Description
main_pid	The process ID of the service’s main process
tasks_current, tasks_max	How many tasks (processes and threads) the service has, and its limit
cpu_usage_nsec	The CPU time the service has used since it started
fragment_path	The unit file, such as /usr/lib/systemd/system/nginx.service
processes	How many processes the service has, and up to 16 of them, each as its process ID and the name the kernel gives it — the main process first

A field systemd does not report is left out, never sent as 0: a service with no main process, no task limit, or CPU accounting switched off, or an older systemd that lacks the property. A process’s name is all that is sent about it — never its command line or its environment, which can carry passwords and tokens. The agent reads the processes from the service’s control group under /sys/fs/cgroup and each name from /proc/<pid>/comm, read-only. Services that are not watched carry none of this.

Left out of the list: template units (foo@.service — their instances, such as postgresql@15-main.service, are listed), temporary units created by systemd-run, the per-login user managers (user@1000.service), and the one-per-connection units of sockets that start a service for every connection. Those carry the connecting client’s IP address in their name, so the agent never sends them.

The watched services — every 15 seconds

For the services that are expected to run, the agent sends these metrics with every telemetry push:

Metric	Unit	Attributes	Description
system.systemd.unit.up	—	systemd.unit.name	1 when the service is up, 0 when it is not
system.systemd.unit.restarts	restarts	systemd.unit.name	How many times systemd has restarted the service automatically (Restart=). A manual systemctl restart starts the count again from 0. Not reported before systemd 235
system.systemd.unit.memory.usage	bytes	systemd.unit.name	The memory the service uses — what systemctl status shows as Memory, page cache included. Not reported when systemd does not account memory for the unit
system.systemd.unit.cpu.utilization	%	systemd.unit.name	The CPU the service used over the last interval, as a share of one core: a busy multi-threaded service can read above 100 %. Agent 1.1.32 and newer
system.systemd.unit.cpu.throttled	%	systemd.unit.name	The share of scheduling periods in which the service was held back by its CPU quota. Only for a service with a CPUQuota=, and only while it uses CPU
system.systemd.unit.memory.utilization	%	systemd.unit.name	The service’s working set as a share of its memory limit (MemoryMax=). The working set is the memory in use without the page cache the kernel can drop, so it can read lower than Memory above. Only for a service with a memory limit
system.systemd.unit.tasks.utilization	%	systemd.unit.name	The service’s processes and threads as a share of its task limit (TasksMax=). Only for a service with a task limit
system.systemd.unit.oom_kills	kills	systemd.unit.name	How many processes of the service the kernel’s out-of-memory killer has killed, counted from the kernel’s log since the agent started counting. 0 on almost every host
system.systemd.units.omitted	services	—	How many services the 50-service limit left out. 0 on almost every host

Which services are watched. The agent watches the services systemd is set up to run, and leaves the rest in the list only:

• services that are enabled and wanted by a target, such as sshd, nginx, NetworkManager or docker — daemons, up while they are running;
• one-shot services that stay active, such as the firewall units ufw, nftables and iptables (RemainAfterExit=yes) — up while active;
• other enabled one-shot services, and services started by an enabled timer, socket or path unit, or on demand over D-Bus — these stop by design between runs, so they are up unless their last run failed;
• services that systemd runs from an init script in /etc/init.d.

CPU, limits and out-of-memory kills (agent 1.1.32 and newer). These are read from each service’s own control group (cgroup v2, and cgroup v1 on older distributions). A service only reports what it has:

• CPU is not reported when CPU accounting is off for the service. On cgroup v2 it is always on. On cgroup v1 (for example Amazon Linux 2 or CentOS 7) turn it on with CPUAccounting=yes in the service’s unit file, or for every service with DefaultCPUAccounting=yes in /etc/systemd/system.conf.
• Throttling is not reported for a service without a CPUQuota=, nor while a service with one is idle.
• Memory of limit and tasks of limit are not reported for a service without a MemoryMax= or a TasksMax= of its own.
• Out-of-memory kills are not reported when the service has no memory accounting (MemoryAccounting=), on a kernel older than 4.19, when the agent runs inside a container, or when the agent cannot read the system journal. An agent installed before 27 July 2026 lacks the permission to read it: reinstall the agent, or add CAP_DAC_READ_SEARCH to the AmbientCapabilities= of wt15-server-agent.service.

A metric that is not reported is left out, never shown as 0.

It does not watch disabled services (unless an enabled timer, socket or path unit starts them), masked services, and services whose start condition is not met on this host (for example cloud-init outside a cloud). Up to 50 services are watched per host, daemons and active one-shots first. A host with more reports how many were left out as system.systemd.units.omitted.

A few things follow from watching the unit rather than the program:

• A service stopped with systemctl disable --now stops being watched, and is not reported as down.
• For a firewall unit, up is the unit’s state, not whether any rules are loaded. systemctl stop nftables is seen. ufw disable bypasses systemd and leaves ufw.service active, so it still reads up.
• A service started from an init script that dies without systemd noticing can still read up: that is how systemd tracks init scripts. Stopping it with systemctl stop is seen.
• A service that exits on its own, when idle or after running once at boot while still being wanted by a target, reads down. It is only alarmed on once it has been seen up.
• In the first 10 minutes after the host boots, a service that has not started yet reports no data rather than down, so a slow start is not mistaken for a failure.
• A service that is down for less than 15 seconds between two samples is not seen as down. If systemd restarts it, the restart count still records it.

Supported systemd versions

systemd 219 and newer, which covers every current distribution plus Amazon Linux 2 and CentOS 7.

systemd	What differs
219–229 (Amazon Linux 2, CentOS 7)	Read through a compatibility path, with the same result
older than 235	No restart count. System service flapping then counts how often the service fell from running
cgroup v1 without memory accounting (for example systemd 219)	Memory per service is not reported. It is left out, never shown as 0

A host whose agent runs without systemd, such as inside a container, reports no system services.

Where this is collected, and how to stop it

System services are collected on every host running agent 1.1.22 or later whose profile has Capture all Telemetry turned on, which is the default, or the System services option ticked. The list is a software inventory of the host: service names and descriptions can name the software you run, including internal products. It is stored with your organisation’s data.

To stop collecting system services on a profile, turn off Capture all Telemetry and tick only the options you want to keep — CPU, Memory, Swap, Disk, Disk I/O, Network, Load and Processes — leaving System services unticked. With Capture all Telemetry off, a profile collects system services only when System services is ticked. Turning telemetry off for the profile stops them too. When collection stops, any system-service incidents on that profile’s servers resolve.

A server’s system services are listed on its page, in the System services tab: see system services. They are watched by the system service alarms.

Disk usage reports

When a volume fills up, the next question is what is filling it. Rather than you logging in to the server to run du, the agent walks each monitored volume at set moments and reports which folders and files take the space. Disk usage reports need agent 1.1.24 or later.

What a report contains

A report describes one volume, as it was when the walk finished:

Part	What it holds
The volume	Its size, used and free space, and inode counts
The folder tree	The largest folders at each level, and any single file big enough to rank among them, each with its size, how many files and folders it holds and its newest modification date. Everything smaller is summed into one N smaller items entry per folder, so the sizes always add up. A folder or file is listed when it holds at least 1% of the folder it is in and at least 64 MiB (on a volume larger than about 67 GB, at least a thousandth of the volume)
Largest files	The 50 largest files on the volume: folder, name, size and modification date
Recently changed files	The 50 largest files modified in the last 7 days, in the same form
Quick culprits	The space taken by Docker’s storage, the systemd journal, package manager caches and kernels other than the running one; and files that were deleted but are still held open by a program, which keep their space until the program lets go. Those are listed by program name, never by process ID, with up to 5 of the files’ paths
Mounts inside the volume	Other filesystems mounted beneath it, which the walk does not enter. Storage volumes and bind mounts are listed; system mounts such as /proc are only counted
Unaccounted space	The part of the volume’s used space that no file or folder explains: usually deleted files still held open, and the filesystem’s own records

Sizes are the space files take on disk, the figure du reports, not their nominal size: a sparse file counts what it really occupies, and a file with several hard links counts once.

The agent reads names, sizes and dates. It never opens a file or reads its contents. What leaves your server is folder and file names, their sizes, modification dates and counts, the names of programs holding deleted files open, and the volume’s size and usage. Reports are stored with your organisation’s data: one per volume, replaced by each new walk, and deleted when the volume goes away or its server is deleted. To read one, open the volume on the server’s Storage tab: see reading a disk report.

Home folders

/home and /root are treated as private:

• Each folder directly inside /home is reported with its name and total size only. Nothing inside it is reported, at any depth: no folder name and no file name. A large file inside /home/alice appears in the largest-file lists as /home/alice, with its size and date but without its name.
• /root is treated as a user folder: its total only.
• Up to 200 folders in /home are listed by name, however small; beyond that, the rest are summed as N more folders.
• A file placed directly in /home, outside any user’s folder, is reported like any other file.
• The rule holds whether /home is a folder on the root volume or a volume of its own, and a volume mounted inside a user’s folder is reported by its total only.

The rule follows the path

Only folders at /home/<name>, and /root, are private. If your users’ homes really live somewhere else and are reached through a symbolic link, for example /home → /data/home, the agent does not follow the link: it walks them at their real location, /data/home/…, and reports the names inside them like any others. The same applies to any other place user files are kept.

When reports are taken

• Once a day, between 04:30 and 05:30 UTC, for every monitored volume. Each volume has its own fixed minute in that hour, so servers that share a profile do not all start at once.
• When a volume is first seen: a new server’s first system-information snapshot, or a disk attached later. Several new volumes on one server are spaced 10 minutes apart.
• When a volume alarm opens or gets worse: Volume is full, Volume is running out of inodes or Volume is filling up opening, or rising from warning to critical. Not when an alarm eases or resolves, and not for Volume became read-only, which a walk cannot explain.
• At most once every 6 hours per volume, whatever the reason. An alarm that opens within 6 hours of a report uses that report rather than taking a new one, so the report can be a few hours older than the alarm.

Nothing else starts a walk: not a restart or an upgrade of the agent, not a change to its profile, not opening a page in the dashboard. There is no button to run one, and there is no setting that turns reports on or off: every server running agent 1.1.24 or later takes them. An older agent is skipped until it updates itself, which it does within minutes of a release.

• A server walks one volume at a time; several wait their turn.
• A new report replaces the volume’s previous one. If a walk fails, or no result arrives within an hour of its scheduled time, the previous report is kept.
• A walk that reaches its 10-minute limit sends what it has, marked as partial.
• A volume is walked only while it is mounted. A network share, such as NFS or SMB, is never walked.

How a walk stays light

• The lowest priority. The walk runs on a thread of its own at the lowest CPU priority (nice 19) and in the idle IO priority class. Only that thread is lowered: the agent’s heartbeats, telemetry and log shipping keep their normal priority, so monitoring carries on while a walk runs, even on a busy disk.
• Paced. At most 10,000 files and folders a second, however fast the disk. A volume of 200,000 files and folders takes about 20 seconds.
• Bounded. A walk stops after 10 minutes or 5 million files and folders, whichever comes first. The agent bounds its own memory use at 128 MiB, and on servers installed with agent 1.1.24 or later its service unit also caps it at 256 MiB.
• One filesystem. A walk never crosses into another mounted filesystem and never follows a symbolic link. It never walks network or FUSE filesystems, container overlay layers or in-memory filesystems.

On a disk that is already saturated, where the server’s IO scheduler honours priorities, an idle-priority walk may make little progress and stop, partial, at its 10-minute limit. That is intended: the walk gives way to your workload.

It sees the server’s real /tmp. The agent’s own /tmp and /var/tmp are private (PrivateTmp=yes, see the privilege model). To report the server’s real ones, the walk looks at the filesystem the way the server’s init process sees it, through /proc/1/root, using the two read capabilities the agent already holds. That gives it no new access. If that view is not available, the report says so and lists /tmp and /var/tmp as not seen.

What the agent does not collect

Some kinds of data would need privileges the agent does not hold, so it does not collect them:

Data	Would require
Raw packet / flow capture	CAP_NET_RAW + CAP_NET_ADMIN
Filesystem integrity monitoring	Out of scope for the agent

The agent’s capability bounding set allows only CAP_SYS_PTRACE and CAP_DAC_READ_SEARCH, so it can never acquire the capabilities above. See the privilege model.

Related

• Configuration reference — the agent’s local settings.
• Alarms & alerting — turning these metrics into alerts.
• Logs — the agent’s other data stream.