Server agent overview

The wt15-server-agent is a single, statically-linked Go binary that collects host telemetry, system information and logs, and forwards them to WatchTower15. Its defining characteristic is its security posture: it runs as an unprivileged service user, never as root, under a hardened systemd unit that grants exactly two read-oriented Linux capabilities and nothing else.

What the agent does

• Reports host telemetry every 15 seconds (CPU, memory, disk, network, load).
• Reports a fuller system-information snapshot every 300 seconds (OS, kernel, per-disk and per-interface detail, and the host’s listening ports).
• Reports the host’s systemd services — the list with each snapshot, and whether each service expected to run is up, how often it restarts and how much memory it uses, every 15 seconds. It reads them from systemd read-only and can never start or stop one.
• Reports what is filling each volume — about once a day, and when a volume is first seen or a disk alarm opens, it walks each monitored volume and reports its largest folders and files by name, size and date, never their contents, and never anything inside a user’s home folder (see disk usage reports).
• Ships logs — tails profile-defined files and the systemd journal, and forwards them for search.
• Self-identifies with tags collected from cloud, Docker and Kubernetes metadata (see auto-tagging).
• Enrolls itself using an organisation token and keeps its configuration in sync with a server profile.

The privilege model

The installer needs root once, to create a dedicated, login-less wt15-server-agent service user and the systemd unit. From then on the agent runs as that user, never as root, under a hardened unit. The relevant directives:

# /etc/systemd/system/wt15-server-agent.service (excerpt)
[Service]
User=wt15-server-agent
NoNewPrivileges=yes
AmbientCapabilities=CAP_SYS_PTRACE CAP_DAC_READ_SEARCH
CapabilityBoundingSet=CAP_SYS_PTRACE CAP_DAC_READ_SEARCH
ProtectSystem=strict
ReadWritePaths=/var/lib/wt15-server-agent /var/log/wt15-server-agent.log
PrivateTmp=yes

• NoNewPrivileges=yes: the process and its children can never gain privileges through setuid/setgid binaries.
• CapabilityBoundingSet: the process can hold only the two capabilities below, and can never acquire any other (not CAP_SYS_ADMIN, CAP_NET_RAW or CAP_NET_ADMIN, for example).
• ProtectSystem=strict with ReadWritePaths: the whole filesystem is read-only to the agent, except its own state directory and its own log file.
• PrivateTmp=yes: the agent gets a private /tmp.

The two capabilities are both about reading:

Capability	Why the agent holds it
CAP_SYS_PTRACE	Reading other users’ process details under /proc, for per-process metrics, to name the process behind each listening port and, since agent 1.1.31, to count each process’s open files against its limit (kernel limits: the files are counted, never named or read), and for the disk usage report, seeing the server’s filesystems as its init process does, real /tmp included. It also allows inspecting other processes.
CAP_DAC_READ_SEARCH	Reading files regardless of their permissions: the /proc entries of other users’ processes, and the log files your profile names, even where they are not readable by the service user. It also lets the disk usage report list every folder of a monitored volume, without opening any file. In effect, the agent can read any file on the host.

Except a service parameter. A file a service reads as a parameter, and the server’s services.allow, are opened with CAP_DAC_READ_SEARCH set aside for that one read, so the file’s own permissions apply, and only where services.allow lists the file (or it is in the agent’s own services.d/). A configuration WatchTower15 sends can never turn the agent into a reader of any file: see service checks.

Neither capability allows writing: ProtectSystem=strict keeps every path except the two above read-only. The service user also belongs to the adm and systemd-journal groups, for reading the journal and conventional log files.

The uninstall unit. The installer, in install mode and with --upgrade, also adds a systemd path unit, wt15-server-agent-uninstall.path. It watches for one file, /var/lib/wt15-server-agent/uninstall-requested, which the agent writes only when WatchTower15 deletes its server, and then runs wt15-server-agent-uninstall.service, as root, to remove the agent. The unit’s script, /usr/local/lib/wt15-server-agent/uninstall, and both units are owned by root and sit outside every path the agent can write. The script runs only when that file exists, reads nothing the agent wrote (the file’s existence is the whole request, and every path in the script is fixed), and can only remove the agent: the agent, or anything that takes it over, can ask for nothing but its own removal.

What the agent does not do is still out of scope: raw packet capture, filesystem integrity monitoring, and anything that needs to change the host.

Why this matters

The agent is not root, cannot escalate, and cannot change the host apart from asking for its own removal. That is what lets it pass security review on hosts where a root-level agent would not. The two read capabilities are named here so that your review can weigh them explicitly: they give the agent broad read access to the host.

How the agent talks to the backend

The agent makes outbound HTTPS only — there are no inbound ports on your host. Every request carries an HMAC signature so the backend can authenticate it without a session:

Header	Purpose
X-Agent-Id	the server’s UUID
X-Agent-Timestamp	Unix time; must be within ±300 s of WatchTower15’s clock, so a server whose clock is more than five minutes off is refused
X-Agent-Signature	HMAC-SHA256 over the id, timestamp and body hash, keyed by the agent secret

Two response headers drive self-management: X-Agent-Config-Etag signals a changed profile configuration, and X-Agent-Required-Version signals an available upgrade.

Each request gets one of these answers:

Code	Meaning
2xx	Stored. Data discarded on purpose is answered 2xx too: from a server waiting in Pending, whose monitoring is switched off, or archived by a person or by teardown automation, from an agent on a deleted or revoked server that is answered 200 (below), replayed telemetry or logs that the server’s profile has since switched off, a system-information snapshot or disk usage report older than the one already stored, and a replayed heartbeat or database snapshot.
403	A missing header, an unknown server, a bad signature, or a clock more than five minutes off.
409	Replayed database data that WatchTower15 refuses: collected after another collector took the database over, while the database was paused, by a server that was not its collector, or of a database that was deleted. data.code is stale_epoch, and data.refused names each database, the reason and the time range refused. Nothing in the request was stored: the agent sends it again at once, without that data.
410	The server was deleted or its access revoked, and its agent has reported that it keeps data while it cannot reach WatchTower15 (below). data.code is server_removed, and data.removal says which: deleted or revoked.
413	The body is over its size cap.
422	The request is not valid.
429	Replayed data is being paced (data.code is replay_shed), the server’s replay budget for the day is spent (replay_budget_spent), or the server was archived after its agent went silent and is waiting to be reactivated (server_reactivating: see Archived servers).
500	Anything unexpected, a database failure included. Nothing was stored: retry as for a 503.
503	A data store did not store the data. Retry later.

A 429, 500 or 503 carries a Retry-After header: the seconds to wait before trying again. The agent keeps the data meanwhile (see When WatchTower15 cannot be reached) and sends what it kept later, marked X-WT15-Replay: 1, with X-WT15-Batch naming the records it carries. The answer to each such request carries X-WT15-Replay-Interval, the seconds to wait before the next.

Once a server is deleted or its access revoked, its agent is answered 410. It then deletes the data it kept on the server and its last profile, keeps its credentials (so a restart cannot enroll it again as a new server), stops collecting, and writes one error line saying so. From then on it only checks in once an hour, is answered 410 again, and should be uninstalled. A server removed before its agent ever reported that it keeps data gets an empty 200 instead: its agent keeps running, and what it sends is discarded, until you uninstall it.

When the server was deleted (data.removal is deleted), agent 1.1.48 or later also asks for its own removal: it writes uninstall-requested beside its credentials, once, and one error line naming the uninstall command. Where the installer added its uninstall unit, systemd then removes the agent within seconds; otherwise the agent stays idle as above. It reads data.removal on every 410, so a server whose access was revoked first and that was deleted later reaches its idle agent at its next hourly check-in. An agent whose server was only revoked stays idle. See Deleting a server.

[wt15-agent] 2026-10-18T08:40:22Z error: WatchTower15 deleted this server. Removal requested: systemd removes the agent within seconds where its uninstall unit is installed; otherwise run: curl -fsSL https://api.watchtower15.com/api/agent/install-script | sudo sh -s -- --uninstall

The agent’s own log, /var/log/wt15-server-agent.log, stamps each line in UTC, whatever the server’s timezone:

[wt15-agent] 2026-10-06T14:03:22Z error: heartbeat failed: connection refused

When WatchTower15 cannot be reached

The agent keeps collecting when it cannot reach WatchTower15, whatever the cause: a network or DNS outage, a firewall or proxy change, or WatchTower15 itself being unavailable. Everything it collects is written to the server’s own disk first and sent from there, and deleted once WatchTower15 has stored it. When WatchTower15 answers again, the newest telemetry goes first, so the server’s charts are live at once, and what was kept follows, oldest first, one request every few seconds at the pace WatchTower15 sets. The gap in the charts fills in as it arrives, and the server reads Catching up meanwhile (see Gaps in a server’s charts).

Data	What is kept
Telemetry	Every sample, for up to 72 hours, in up to 64 MB, plus 24 MB for each database the agent collects, or collected in the last 72 hours. Past either limit the oldest samples are dropped first, and the agent log names the time range dropped.
System information	The latest snapshot.
Database snapshots	The latest one per database, while it is recent. An older one is replaced by a fresh snapshot rather than sent.
Disk usage reports	Each report, until it is delivered or its task expires.
Heartbeats	Never kept: a heartbeat says the server is alive now.
Logs	Not copied: the agent notes which parts of each log file are still to send, as positions, never lines, for up to 72 hours, in up to 64 MB, and sends them from the file when WatchTower15 answers again. The journal is read again from the last entry WatchTower15 stored. See tailing behaviour.

• Never below the free-disk floor. The agent writes nothing to its store while the filesystem has less than 10 % or less than 512 MB free, nor when the disk refuses a write (full, out of quota or read-only). It keeps telemetry in memory instead: everything still reaches WatchTower15 while it answers, but an outage then keeps only its last two minutes. The agent writes one error line when this starts, tries the disk again every five minutes, and writes a notice once it is back on disk.
• Kept beside the credentials, in /var/lib/wt15-server-agent/spool/, readable only by the agent’s service user and root. See What the agent keeps on the host.
• The clock corrects itself. WatchTower15 refuses a request whose timestamp is more than 300 s from its own clock. The agent compares its server’s clock with the Date of every answer, and once the two differ by more than 290 s it corrects the timestamp of its requests and the times of the data it collects by the difference, until they differ by less than 120 s. It never changes the server’s clock: it writes one error line asking for the server’s time to be fixed, and a notice when the clocks agree again.
• Database data refused. WatchTower15 refuses the database data the agent kept for the time after another collector took the database over, or while the database was paused (see Collector pools). The agent leaves it out, delivers the rest at once, and --status counts it on its Refused line.
• One error line per outage. The agent log gets an error line when an outage starts, for example cannot reach WatchTower15 (dns_failure: lookup api.watchtower15.com: no such host); keeping data on this host, a warning every hour while it lasts, a notice when WatchTower15 answers again, and another once the kept data is delivered. Failed attempts in between are not logged as errors, so an alert on the agent’s errors fires once per outage. --status shows the same state at any time (see Verify the install).

The agent writes its store as it collects, without forcing each write to disk:

Situation	What the agent writes
Connected	Each telemetry sample once, every 15 seconds (a few KB compressed); each system-information and database snapshot when it is taken; one small bookkeeping file rewritten per sample; about 120 bytes for each log file that grew, every 15 seconds, and the log positions at most once a minute
WatchTower15 out of reach	Nothing more: the data is already on disk. A small outage journal when an outage starts, every hour and when it ends, and the agent’s state file at most once a minute
Kept data delivered	The delivered files are deleted
Under the free-disk floor, or the disk refuses writes	Nothing in the store
The agent stops or upgrades	One flush to disk
The server is removed from WatchTower15	The store is deleted

On a server without databases these writes come to about 54 MB of pages written a day, plus about 6.4 MB a day for each database the agent collects, plus about 20 MB a day for a log file growing at 10 lines a second (about 23 bytes a line), with the filesystem’s own journal on top.

Lifecycle at a glance

1. Install — the one-line installer (run as root) downloads the binary (verified against a published X-Content-SHA256), creates the service user, and writes the unit.
2. Enroll — the agent calls the enrollment endpoint with your organisation token, optionally collects self-reported tags, and stores its credentials locally. If Require approval for new servers is on (Organisation → Settings), the server then waits in Pending until it is approved, from Approve in the ⋯ menu at the top right of its page. If your organisation has a server limit and it is reached, the server is held instead: it waits in Pending, tagged Waiting — server limit reached, is not monitored, and is approved from its page’s ⋯ menu once a slot is free. See held servers. And if your organisation has no payment method (and is not a courtesy account), or has an overdue invoice, the server waits for that first, tagged Waiting — payment required: it is approved from its page’s ⋯ menu once a payment method is added. See without a payment method.
3. Enroll again — an agent with no credentials, on a host WatchTower15 already knows, continues that host’s server instead of enrolling a new one: the same server, with its Audit tab, charts, alarms, tags, profile and server-limit slot. That is what happens after an uninstall and a new install, an OS reinstall or a deleted credentials file, when three things match: your organisation, the host’s instance ID, and the host name WatchTower15 shows for the server (on AWS EC2 the instance’s Name tag when the agent can read it, on DigitalOcean the droplet’s name). It happens at once, even if an agent still runs with the old credentials: that agent is refused from then on. An archived server is reactivated first, as Reactivate would, so it needs a payment method and a free server slot; without them the agent enrols a new server, which waits as any new one does. Continuing needs an instance ID: the agent detects one on AWS EC2, on DigitalOcean and in a Docker container. Elsewhere (bare metal, on-premises virtual machines, other clouds), install it with --supplier-identifier=<a stable ID>, the same on every install of that host (see Installer options). A paused, waiting, revoked or deleted server is never continued, and neither is a server in another organisation or under another host name: the agent enrols a new one. The server keeps its own profile, whatever the new install names; you change it on its page’s Info tab, from (change) on the Profile card.
4. Run — it pushes telemetry and system information on their intervals, ships logs, and polls its configuration etag. The first telemetry or logs to arrive is recorded as First data received. While it cannot reach WatchTower15 it keeps collecting, and sends what it kept once it can (see When WatchTower15 cannot be reached).
5. Upgrade — when the backend advertises a newer required version, the agent updates itself (see Upgrades).
6. Archive — a server you no longer need monitored, but whose history you want to keep, is archived: by you, from Archive in its page’s ⋯ menu; by your teardown automation; or by WatchTower15, once its agent has not checked in for the number of days your organisation sets on Organisation → Settings → Server Monitoring: Archive servers unseen for the last X days, from 2 to 30, 7 by default. An archived server stops being monitored and billed, moves from the Servers list to its Archived view, and no longer counts toward your server limit. Its page stays viewable, and the host is left untouched. Reactivate brings it back, once a payment method is set and a server slot is free, as approving does. A server archived for silence is reactivated by itself, within the hour, when its agent reports again, and installing the agent again on its host brings it back at once (step 3). See Archived servers.
7. Delete — a server you no longer need, history included, is deleted from Delete in its page’s ⋯ menu, after a page that says what will happen. It leaves every list, stops being billed, and its open incidents are resolved without a notification. Its agent removes itself from the host where it was installed, or upgraded with the installer, from agent 1.1.48 on; otherwise it stops sending and waits to be uninstalled. Nothing restores a deleted server: running the installer again enrols a new one. See Deleting a server.

The server’s Audit tab — the Audit tab, the last on the server’s page — records its lifecycle: how it enrolled (automatically or manually approved) and with which agent version, whether it was held at the server limit, its approval, first data received, agent upgrades and reboots, any revocation or token rotation, each archive and reactivation, as Server archived and Server reactivated, and any re-enrolment, as Server re-enrolled. It also records each gap of 5 minutes or more in which nothing was heard from the agent, as Connectivity lost and Connectivity restored, with its cause and what became of its data, once the gap is complete: its cause settled and the data the agent kept delivered or given up. Those two rows describe the gap as it stood then; the server’s banner and its Agent card follow anything the agent reports about it later, such as data it dropped. The Agent card on the server’s Info tab shows its Enrollment Date and Approved By; how it enrolled is on the Audit tab.

Archived servers

Archiving retires a server from monitoring without deleting it: a host you decommissioned, or the server an Auto Scaling instance leaves behind when it is terminated. Only an approved server can be archived, whether its monitoring is on or paused. A server waiting in Pending is approved or deleted instead, and one whose access was revoked is deleted. Archiving or reactivating a server yourself needs the permission to edit servers (servers.edit).

• By hand. Archive, in the ⋯ menu at the top right of the server’s page, opens a page that says what will happen before anything does: monitoring and billing stop, how many open incidents will be resolved, what stays viewable and for how long, and whether the server collects a database or has custom metrics linked to it. Archive server archives it and opens its page; Cancel changes nothing and takes you back to the server’s page.
• For silence. Each day at 05:10 UTC, WatchTower15 archives every approved server whose monitoring is on and whose agent has not checked in for longer than your organisation’s Archive servers unseen for the last X days, on Organisation → Settings → Server Monitoring: from 2 to 30 days, 7 by default. A server whose agent has never checked in is never archived this way. When many servers are due at once, some wait for the next day’s run. A change to the setting applies from the next run: lowering it archives the servers it now covers, and raising it brings back none already archived.
• When WatchTower15 was not receiving data. A server that went silent while WatchTower15 was not receiving data is not archived automatically: archive it yourself. This holds when at least half of your organisation’s monitored servers last checked in during the same interruption, as they would if the fault were on WatchTower15’s side. Once the interruption is over, its Agent stopped reporting alarm fires for it as for any silent server, unless it powered off.
• By teardown automation, with one call: see Teardown automation.
• What stops. Nothing the agent sends is stored, nothing is scanned, and the server is not billed. Its open incidents, and those of its volumes, listening ports and system services, are resolved without a notification, and its alarms are not evaluated. Another collector in its database’s collector pool takes over a database it collects, and a database with no other collector stops being collected. Archived servers are no longer offered where you pick a server: dashboard widgets, the Logs page’s server filter, custom metric links and a database’s collector.
• What stays. Its page, its details, its alarms, its incidents and its Audit tab. Its raw charts and logs stay for 30 days after its last data, and its hourly charts for 10 years. Its alarms can still be edited, on its Alarms tab; they apply once it is reactivated. Custom metrics linked to it stay linked and are billed as before, because they are pushed with their own token. The host is left untouched: a still-running agent keeps its credentials, and nothing is removed from the machine.
• Reactivate. Reactivate, in the bar of the server’s page, asks you to confirm, then monitoring and billing resume with no reinstall. If its agent is still running, it reports again within a minute. It needs a payment method and a free server slot, as approving does: while either is missing, Reactivate is disabled and the page says which. A reactivated server whose host is gone reads Unreachable after 5 minutes and its Agent stopped reporting alarm fires, as for any silent server, and it is archived again after your organisation’s setting.
• Back by itself, after silence. A server archived for silence is reactivated within the hour of its agent reporting again, once a payment method is set and a server slot is free. While either is missing it stays archived, its page says when its agent reported again, and it is tried again every hour. A server archived by a person or by teardown automation stays archived whatever its agent sends: its agent is ignored until someone reactivates it.
• What its agent kept. Until it is reactivated, a returning agent is told to try again shortly (429, server_reactivating), so an agent that keeps data on the host keeps what it collects and delivers it once the server is reactivated, oldest first, as after any outage (see When WatchTower15 cannot be reached). While a missing payment method or a full server limit holds it archived, data older than 72 hours ages out on the host.
• A clock more than 5 minutes off. A returning host whose clock is more than 5 minutes off WatchTower15’s is refused before its return is noted. An agent that corrects its clock is noted once it has, usually within a minute; otherwise the server stays archived until the host’s time is fixed or someone reactivates it.
• Keeping it archived. A server archived for silence comes back whenever its host starts again. To keep one archived whatever its host does, delete it, or reactivate it and archive it yourself: a person’s archive stays archived.
• Organisations stay apart. A host enrolled in another organisation is a new server there. The server it leaves behind goes silent and is archived by its own organisation’s setting. Nothing one organisation does archives or changes a server in another.
• Deleting it. An archived server is deleted from its page’s ⋯ menu, where Delete is then the only action, or from Delete it → in its Archived banner: see Deleting a server.

Deleting a server

Deleting a server removes it from WatchTower15 for good: it leaves every list, and nothing restores it. To keep its history viewable instead, archive it. Deleting a server needs the permission to delete servers (servers.delete); without it, the server’s ⋯ menu shows no Delete.

• The Delete page. Delete, in the ⋯ menu at the top right of the server’s page, opens a page that says what will happen before anything does: what stops, how many open incidents will be resolved, what is removed and what is kept, what happens on the host, and how to monitor the host again. To confirm, type the server’s name as the page shows it (its display name, else its hostname, else its UUID) and click Delete server: the Servers list then opens. Cancel returns to the server’s page. For an approved server that is not archived, the page also offers Archive instead → to members who can edit servers.

• What stops. Nothing its agent sends is accepted, nothing is scanned, and the server stops being billed. It leaves the Servers list and no longer counts toward your server limit. Its volumes, listening ports and system services stop being monitored with it, and their open incidents and its own are resolved at once, without a notification.

• What is removed. Its disk usage reports, and its links to custom metrics: the metrics stay, no longer linked to it. Another collector in its database’s collector pool takes over a database it collects, and a database with no other collector stops being collected.

• An archived server is already unmonitored, unbilled and outside your server limit, and its incidents were resolved when it was archived. Deleting it removes it from the Servers list’s Archived view, with the reports, links and collector roles above. Its Delete page says when it was archived in place of what stops.

• What is kept. Its details, its incidents and its audit trail, and its charts and logs for as long as an archived server’s (see What stays under Archived servers). Its telemetry, logs and scan results are not deleted: they age out as they would. A deleted server can no longer be opened, so its events stay on the Audit page, where Resource finds it by name: Server deleted, by the person who deleted it, and an Incident resolved for each incident, with the reason the server was deleted.

• On the host. WatchTower15 never reaches into your host, so what happens there depends on the server’s agent, as the table below says. The Delete page reads the agent’s last reported version, says which applies, and shows the uninstall command:

  curl -fsSL https://api.watchtower15.com/api/agent/install-script | sudo sh -s -- --uninstall

  It removes the agent and everything the installer created: see Uninstall. Until an agent is uninstalled, its credentials and your organisation’s install token (in /etc/wt15-server-agent/agent.env) stay on the host.

• Monitoring the host again. Nothing restores a deleted server: run the installer again, and it enrols the host as a new server, with a history of its own. On a systemd host whose agent asked for its removal, the installer removes the old agent first, saying WatchTower15 deleted this host’s previous server: removing the old agent first. An agent older than 1.1.48 never asks: uninstall it first, or the new install carries on as the deleted server instead of enrolling a new one. In a container, remove the container and its volume, then start a new one.

The server’s agent	What happens on the host
1.1.48 or later, installed, or upgraded with --upgrade, by the installer of 1.1.48 or later	It removes itself within a minute: the agent, its units, its files, its log, the install token and its service user. When the server’s access was revoked first, within the hour, as a revoked server’s agent checks in once an hour.
1.1.48 or later, installed earlier and only self-updated since	It stops sending and waits, checking in once an hour, with its removal requested. Run the uninstall command, or --upgrade: the uninstall unit it adds then removes the agent.
1.1.36 to 1.1.47	It stops sending and waits, checking in once an hour. It does not remove itself: run the uninstall command.
Older than 1.1.36, which keeps nothing while it cannot reach WatchTower15	It is not told: it keeps running, and everything it sends is ignored. Run the uninstall command.
In a container	From 1.1.36, it stops sending and waits; an older agent keeps running, ignored. Remove the container and its volume: the uninstall command removes nothing there, and says so.

Teardown automation

When an Auto Scaling group scales in or Terraform destroys an instance, the server it leaves behind would wait for the daily archive. Your automation can archive it at once, with one call authenticated by your organisation’s install token, as enrollment is:

POST https://api.watchtower15.com/api/agent/archive
Content-Type: application/json

{"org_token": "<install token>", "supplier_identifier": "i-0abc123def4567890"}

• What it archives. Every approved, unarchived server of your organisation whose supplier ID is supplier_identifier: a host installed more than once has a server for each install. Only the token’s own organisation is ever read.
• The answer. 200 with {"data":null}, whether it archived one server, several or none, so it says nothing about which instance IDs are enrolled. A 422 means the body is not valid, or the token matches no organisation (errors.org_token: Invalid organisation token.). A 5xx archived nothing: retry it, up to 3 times. Read any 200 as done.
• What it records. The same archive as one made by hand, with By teardown automation on the server’s Audit tab. A server archived this way stays archived if its agent ever reports again: only a server archived for silence comes back by itself.
• Which identifier. A server’s supplier ID is the one given at install (--supplier-identifier, WT15_SUPPLIER_IDENTIFIER), else the agent’s Docker container ID, else the cloud instance ID. A container install on an Auto Scaling instance therefore matches teardown only if it was installed with --supplier-identifier=<instance id>.
• The install token. Anyone who holds it can archive any of your approved servers by instance ID, as they can enroll servers. Regenerate it (Organisation → Settings → Server Monitoring) to stop a leaked one. Then replace WT15_ORG_TOKEN in each host’s /etc/wt15-server-agent/agent.env (in a container’s environment, for a container install) and on the machine that runs Terraform: until you do, teardown gets the 422 and the server waits for the daily archive. The hook reads the file when it runs, so the agent needs no restart.
• Behind an IP allow-list. A host or Terraform runner that makes the call allows the three addresses of api.watchtower15.com too (see the Restricted egress tab under Install).

Neither example below writes the token into a script, a command line or a Terraform variable.

Auto Scaling: a termination lifecycle hook

Give the group a lifecycle hook on autoscaling:EC2_INSTANCE_TERMINATING, and the instances’ role autoscaling:CompleteLifecycleAction. Each instance runs the script below as root, started at boot (by a systemd service, for example). It waits until the instance metadata (IMDSv2) says the instance is being terminated, archives its server, then lets the termination go on. It reads the token, the API’s address and any proxy from the agent’s own configuration, so nothing is copied into the image, and it sends the call through the proxy when the agent uses one.

#!/bin/sh
# Archives this instance's WatchTower15 server when its Auto Scaling group terminates it.
ASG_NAME="my-asg"           # your Auto Scaling group
HOOK_NAME="wt15-archive"    # its EC2_INSTANCE_TERMINATING lifecycle hook
ENV_FILE=/etc/wt15-server-agent/agent.env

imds() {  # instance metadata, IMDSv2, never through a proxy
  t=$(curl -sf --noproxy '*' -X PUT http://169.254.169.254/latest/api/token \
    -H 'X-aws-ec2-metadata-token-ttl-seconds: 60') &&
    curl -sf --noproxy '*' -H "X-aws-ec2-metadata-token: $t" "http://169.254.169.254/latest/$1"
}
setting() { sed -n "s/^$1=//p" "$ENV_FILE"; }

until [ "$(imds meta-data/autoscaling/target-lifecycle-state)" = "Terminated" ]; do sleep 5; done

instance_id=$(imds meta-data/instance-id)
id=$(setting WT15_SUPPLIER_IDENTIFIER); [ -n "$id" ] || id=$instance_id
api=$(setting WT15_API_URL); [ -n "$api" ] || api=https://api.watchtower15.com

set --  # curl's proxy option, as the agent uses it
proxy=$(setting WT15_PROXY)
if [ -n "$proxy" ]; then
  host=${proxy#*://}; host=${host%%/*}; host=${host##*@}
  case "$host" in
    *:*) ;;  # the URL names its port
    *) port=$(setting WT15_PROXY_PORT)
       [ -n "$port" ] || case "$proxy" in https://*) port=443 ;; *) port=80 ;; esac
       proxy="${proxy%/}:$port" ;;
  esac
  set -- -x "$proxy"
fi

for attempt in 1 2 3 4; do  # the call, then up to 3 retries of a 5xx
  code=$(printf '{"org_token":"%s","supplier_identifier":"%s"}' "$(setting WT15_ORG_TOKEN)" "$id" |
    curl -s -o /dev/null -w '%{http_code}' "$@" -X POST "$api/api/agent/archive" \
      -H 'Content-Type: application/json' --data-binary @-)
  case "$code" in 5??|000) sleep 5 ;; *) break ;; esac
done
[ "$code" = 200 ] || echo "WatchTower15 archive answered $code" >&2

aws autoscaling complete-lifecycle-action --lifecycle-action-result CONTINUE \
  --lifecycle-hook-name "$HOOK_NAME" --auto-scaling-group-name "$ASG_NAME" \
  --instance-id "$instance_id" --region "$(imds meta-data/placement/region)"

The token travels to curl through a pipe, never on a command line. A server the call could not archive waits for the daily archive; the termination goes on either way.

Terraform: a destroy-time provisioner

A local-exec provisioner with when = destroy runs on the machine that runs Terraform. It takes the token from WT15_ORG_TOKEN in that machine’s environment, and the instance ID from self.id, so it matches a server whose install did not set its own supplier ID. A destroy-time provisioner can read only self, count.index and each.key, and runs only while it is still in the configuration when the instance is destroyed.

resource "aws_instance" "web" {
  # …

  provisioner "local-exec" {
    when        = destroy
    environment = { SUPPLIER_IDENTIFIER = self.id }
    command     = <<-EOT
      for attempt in 1 2 3 4; do
        code=$(printf '{"org_token":"%s","supplier_identifier":"%s"}' "$WT15_ORG_TOKEN" "$SUPPLIER_IDENTIFIER" |
          curl -s -o /dev/null -w '%%{http_code}' -X POST https://api.watchtower15.com/api/agent/archive \
            -H 'Content-Type: application/json' --data-binary @-)
        case "$code" in 5??|000) sleep 5 ;; *) break ;; esac
      done
      [ "$code" = 200 ] || echo "WatchTower15 archive answered $code" >&2
    EOT
  }
}

%%{ is Terraform’s escape for curl’s %{. The command always succeeds, so a failed call never stops a destroy: the server waits for the daily archive instead.

The server’s page

Each server has its own page, reached from Servers. A bar pinned to the top of the page stays in view while you scroll, with a shadow under it once the page scrolls beneath:

• the breadcrumbs and the server’s name;
• how long the server has been up, or what replaces it. Once the server is unreachable — no heartbeat for 5 minutes — it reads Last seen … in amber. A server WatchTower15 is rejecting for its clock reads Rejected · clock 7 min fast (or slow) in red, a server powered off cleanly Powered off …, one whose agent has never checked in Never seen, and one with no agent enrolled No agent. While a server is catching up, a blue Catching up tag sits beside its uptime;
• its alarm counts — N Critical · N Warning · N OK — covering every alarm on the server and on its volumes, listening ports and system services. Each count opens the Alarms tab, filtered to it;
• on an archived server, Reactivate, if you can edit servers and its agent is approved;
• the ⋯ menu, with what you can do to the server (see Changing a server);
• the tab bar.

A server that is waiting for approval, whose agent token was revoked, whose monitoring is switched off, or that is archived says so in a banner under the bar. The waiting-for-approval banner tells members who can approve it to Approve it from the ⋯ menu at the top right. So does a server that cannot reach WatchTower15, is still delivering what it kept, or has just come back from a gap, with what its alarms would have shown meanwhile: see Connectivity banners.

The Archived banner, the last under the bar, says when the server was archived, that monitoring is off, and how long its raw charts, logs and hourly charts stay, with See Audit → and, if you can delete servers, Delete it →, which opens the Delete page. For a server archived for silence it says that the server is reactivated within the hour if its agent reports again, once a payment method and a free server slot allow it; once its agent has reported again, it gives that date, and adds that what its agent keeps meanwhile is delivered then. Any other archive says that its agent, if still running, is ignored. While Reactivate is disabled, a line under the banner says why: a missing payment method, or the server limit and how many servers are in use. An archived server’s header reads Last seen …, muted, as a server whose monitoring is switched off does.

Each tab has its own address, so a link opens the tab it names:

Tab	What it holds
Overview	A details card with the server’s name (its display name, or its hostname when it has none), which you can edit in place, then its hostname, supplier ID, IP address and health, and every telemetry chart, sharing one cursor and zoom. Under Telemetry, a Custom metrics card charts the first eight custom metrics shown on the server. The alarms behind the health are on Alarms, and the profile and tags on Info
Logs	The log search, limited to this server’s logs and kept live — Live · every 15 s — while the tab is open. Open in Logs opens the same view on the Logs page. Shown with the logs.view permission
Storage	The disk charts, from space to each disk’s latency and busy share, and the server’s volumes, each with its alarms, incidents and disk usage report
Network	The server’s network interfaces, each with its link and its current rates, errors and drops, its listening ports, and the network charts
Processes	The two process charts and the Top processes table
System services	The server’s systemd services, each with its uptime and memory, and a systemctl status-style summary
Custom metrics	Every custom metric shown on the server, grouped by metric source, each chart saying why it is there. Shown once at least one metric is linked to the server, its tags or its metric source, and only with the custom_metrics.view permission
Alarms	Every alarm on the server and on its volumes, ports and system services, with each alarm’s live value
Incidents	Every incident on the server and on its volumes, ports and system services that was open at any point in the page’s time range, with an On column saying which. Under the card’s title, the Active · Resolved · All filter, with a count in each, sits on the left and the time range on the right (The Incidents tab)
Info	Everything the agent reports about the host — OS, kernel, CPU, memory, boot time and how it connects — with the server’s Notes, which you can edit in place; then the server’s Clock: its timezone, whether its clock is synchronised, how accurate it is and how far it is from WatchTower15’s (Times & timezones). Below them, two columns. On the left, a Tags card listing each of the server’s tags as Tag Group › Tag, each tag opening Servers filtered to it, with (change) beside its title to choose the server’s tags, and under it the Agent card: the agent’s version, last seen and enrollment, and its connection to WatchTower15 (The Agent card). On the right, a Databases card when the server collects or runs databases, listing them, and under it the Profile card: the server’s Current profile, whose link opens its profile in a new tab, with a Default pill when it is your organisation’s default profile and (change) beside it to move the server to another profile, then the telemetry it collects (All telemetry, or each type it collects), its log files and folders, and its schedules. Hovering the hostname, display name, operating system, kernel, CPU model, IP address or supplier ID shows a button that copies it
Audit	The server’s audit trail: its lifecycle, changes, alarm changes, incidents and health changes, newest first

The time range offers the last 1h, 3h, 12h, 1d or 5d, then Custom: a length of your own, such as the last 2 weeks, or dates on the calendar. The range you choose applies to every tab with charts, and to Logs and Incidents, and stays the same as you move between tabs. Opening another server starts again on the last 3 hours (an archived server opens on the day before its last check-in). That day shows as Custom, so an archived server’s charts open on its last data, and any other range can be picked as usual. The footer of each selector’s Custom panel switches the whole page between your timezone, the server’s and UTC: see A server’s own timezone.

Changing a server

There is no separate page for editing a server: you change it where its page shows it. Every change needs the permission to edit servers (servers.edit), except deleting, which needs servers.delete. An archived server’s name, notes, profile and tags cannot be changed until it is reactivated (Reactivate is in the bar), and its ⋯ menu offers only Delete.

The ⋯ menu, at the top right of the server’s page, offers:

• Approve, while the server is waiting for approval. It is disabled, with the reason under it, while your organisation needs a payment method or has reached its server limit.
• Disable Monitoring or Enable Monitoring, depending on whether the server is monitored. Enable Monitoring is unavailable, with the reason under it, on a server that has not been approved yet, on one whose agent token was revoked (re-install its agent to monitor it again), and while your organisation needs a payment method or has an overdue invoice.
• Archive, on an approved server that is not archived. It opens the Archive page, which says what will happen first: see Archived servers.
• Delete, which opens the Delete page: see Deleting a server.

Editing in place. On the Overview tab, hovering the server’s Name shows a copy button and an edit button. Edit turns the name into a text field with ✓ (save) and ✕ (cancel) beside it: Enter saves, and Esc or a click anywhere else discards your change. Clear the name and save to have the server go by its hostname again. The hostname itself comes from the agent and cannot be edited. The Notes on the Info tab’s System card are edited the same way, over several lines: Ctrl+Enter (⌘+Enter on a Mac) saves.

Profile and tags. On the Info tab, (change) beside the Profile card’s current profile opens a panel listing your organisation’s profiles, the default one marked (default). Under the list, it shows what the chosen profile collects — its telemetry, logging and schedules — laid out as the card shows them, so you can compare before you switch. Save moves the server to that profile; its agent picks up the new configuration on its next poll, with no restart. Changing the profile also needs the permission to view profiles (server_profiles.view). (change) beside the Tags card’s title opens a panel where you choose the server’s tags; Save applies them.

Connectivity banners

One connectivity banner shows at a time, under the approval, revoked and switched-off ones: the first of these that applies. An archived server shows none, as it has no connectivity state.

Banner	Shown when	What it says
Rejected (red)	WatchTower15 is rejecting the agent’s data for the host’s clock	WatchTower15 is rejecting this server’s data: its clock is 7 min fast. Set the host’s time with NTP. While the agent keeps data on the host (below), it adds Its data is kept on the host meanwhile.
Unreachable (amber)	The server is unreachable	When it was last heard from, such as Not heard from since 10:12 BST (12 min ago)., that the host may be down or its agent cannot reach WatchTower15, and what to check. While the agent keeps data on the host: Data collected meanwhile is kept on the host for up to 72 hours and delivered when it reconnects. Otherwise: Data from this time may be missing from charts. While WatchTower15 is not receiving data from most agents, it points to that notice instead of the host
Catching up (blue)	The server is catching up	Catching up. 1 h 40 min of buffered data is still arriving from the host (12 MB). The charts fill in as it lands. When the time is not known, the size alone. While WatchTower15 is holding back the delivery, it adds Delivery is paused for now and resumes on its own.
Recovered (green)	All of the latest gap’s data arrived	Reconnected after 2 h 13 min (No connection from the host: DNS lookup failed). All data recovered. Time the server was powered off, held, paused or awaiting approval is named, as it is not a loss: All data recovered (1 h powered off). Then, once the gap is reviewed, what its alarms would have shown: No alarm would have fired.
Lost (amber)	Some of the latest gap’s data will not arrive	Reconnected after 3 h 5 min. Then what was lost, largest first: 40 min was dropped on the host; 2 h 13 min was not kept: this agent version keeps nothing while it cannot reach WatchTower15; 1 h was not collected: the host was down (or the agent was not running). Then The rest was recovered when some was, and any powered-off, held or paused time in brackets, as on Recovered. Then, once the gap is reviewed, what its alarms would have shown in the data that arrived

• The Recovered and Lost banners are about the latest gap of 5 minutes or more in which nothing was heard from the agent. They show once the gap is complete, a few minutes after the server reconnects and once what the agent kept has been delivered, while the server is connected (or only delayed), and for 24 hours after the gap ended. See Audit → opens the server’s Audit tab, where the gap is recorded, and ✕ hides the banner in this tab; a newer gap shows again. A gap made only of time the server was powered off cleanly, held, paused or awaiting approval is not a loss, and shows neither.
• What Lost counts: data the agent dropped on the host, data it kept but could not deliver in time, all of a gap for an agent version that keeps nothing, and the time the host was down or its agent was not running, when nothing was collected.
• The cause in brackets is the one that took most of the gap while the agent was running, such as No connection from the host with its error, Rejected by WatchTower15 or WatchTower15 outage — not your server. It is left out when the agent was not running at all.
• The banners promise data kept on the host only while the agent keeps it now: its version keeps data, and its host’s disk can take it. While the disk is full or cannot be written, the agent keeps only the last 2 minutes, in memory (see When WatchTower15 cannot be reached), and the banners make no such promise. During an outage this is what the agent last reported before it. The Agent card’s Keeps data on the host says which applies.

What the alarms would have shown

A few minutes after a gap is complete, WatchTower15 reviews it: it works out what the server’s alarms would have shown while its data could not arrive. No alarm is raised for that time. The Recovered and Lost banners then add one sentence:

The review found	The sentence
Nothing	No alarm would have fired.
Nothing, in a gap that lost time	No alarm would have fired in the data that arrived.
One alarm	One alarm would have fired: Server CPU usage is high, critical for 40 min.
More	3 alarms would have fired: Server CPU usage is high, critical for 40 min; Volume is full on /var, warning for 1 h 20 min; and 1 more.

• At most two alarms are named, the most severe first, then the longest. A volume or system service adds its name (on /var), and the time is how long the alarm would have been firing.
• In the data that arrived. A gap that lost time, through data dropped or not delivered, or time the host was down or its agent was not running, is reviewed only on the data that reached WatchTower15, so the banner cannot say that nothing happened in the rest.
• An alarm that was an incident at the time says so in brackets: (already open) when its incident was open as the gap began, (raised during the gap), or (raised on reconnect) when it was still breaching as the server came back. (already open as warning) means the alarm reached a higher level during the gap than its open incident: it went critical while its incident was a warning, and that critical was never paged. (raised during the gap as warning) and (raised on reconnect as warning) mean the same for an incident raised then. The gap report names both levels and when.
• No sentence while the review is still to run, or when it had nothing to judge: none of the gap’s data arrived, the server’s metrics kept arriving and were evaluated live, or the data could not be read.

The Agent card

The Agent card on the Info tab shows the agent’s version, when it was last seen and its enrollment, and its connection to WatchTower15:

Row	Shows
Keeps data on the host	Whether the agent keeps what it collects while it cannot reach WatchTower15: Yes — up to 72 hours; Not now — the host’s disk is full or cannot be written, so only the last 2 minutes are kept; Unknown — the agent has not reported where it keeps data; or No — this agent version keeps nothing while it cannot reach WatchTower15.
Buffered data	What the agent has kept and not yet delivered: None, 1 h 40 min · 12 MB · being delivered (or the size alone), or Unknown until it reconnects while the server is unreachable or rejected.
Last connection error	The most frequent error of the latest gap in the last 30 days in which the agent’s requests failed, how many times it happened and when it last did: DNS lookup failed · 412 times · 02/10/2026, 14:03 UTC. Otherwise None in the last 30 days.
Dropped on the host (30 days)	How much time’s data the agent collected but did not deliver over the last 30 days, dropped on the host or kept but not delivered in time: None or 40 min.

Buffered data and Dropped on the host show only for an agent version that keeps data. The errors read DNS lookup failed, Connection refused, Connection timed out or no route, TLS handshake failed, Proxy refused the connection, WatchTower15 answered with an error (5xx), Too many requests (429), Rejected: host clock, Rejected: agent credentials and No answer, or the connection dropped: the kind of error, never its message, which can hold host names and addresses. How far the server’s clock is from WatchTower15’s is on the Clock card.

Connectivity

Every server whose agent reports has a connectivity state, which WatchTower15 works out from the agent’s heartbeat, sent every 30 seconds:

State	Meaning
Connected	A heartbeat in the last 90 seconds
Catching up	A heartbeat in the last 90 seconds, while the agent is still delivering data it kept on the host. It lasts for a short time after the agent reconnects, usually under a minute, as it follows what the agent reports with each heartbeat, so a page that refreshes every minute can miss it. Only an agent version that keeps data reaches it
Delayed	The last heartbeat is between 90 seconds and 5 minutes old: a few were missed
Unreachable	No heartbeat for 5 minutes or more
Rejected	In the last 5 minutes WatchTower15 refused the agent’s heartbeat because the host’s clock is more than 5 minutes off WatchTower15’s, and has accepted none since. The next accepted heartbeat ends it; an agent that corrects its clock usually does so within a minute. A server refused before its agent was ever accepted reads Rejected, not Never seen
Powered off	The host shut down cleanly and its agent said so (agent 1.1.21 or later)
Never seen	The agent enrolled but has never checked in

A server that is waiting for approval, whose agent token was revoked, that has no agent, whose monitoring is switched off, or that is archived has no connectivity state: nothing is accepted from an agent there.

Unreachable starts at 5 minutes everywhere — the server’s page, the Servers list, heat maps, the Silent agents and Agent activity widgets and the notice bar — the moment Agent stopped reporting warns. Changing that alarm’s thresholds changes when you are alerted, not when a server reads Unreachable.

A rejected server counts with the unreachable ones — in the notice bar’s count, on heat maps and in the Silent agents and Agent activity widgets — once 5 minutes pass without an accepted heartbeat, or at once if none was ever accepted. For one that was accepted before, Agent stopped reporting warns from the same 5 minutes. A server catching up counts as connected; Agent activity also counts it on its own.

Finding them on Servers

The Connectivity filter on Servers is a list to pick from: choose one or more states — Connected, Catching up, Delayed, Unreachable, Rejected, Powered off and Never seen — to show the servers in them. With none chosen it shows every server; clear it with its ✕. It is kept with your other filters. The Last Seen column turns amber with a warning sign for an unreachable server, and red with a ban sign for a rejected one, with the age of its last accepted heartbeat (Rejected · 12 min ago, or Rejected · never accepted). It reads Powered off with a power icon for a server that was powered off, Never seen for an agent that never checked in, and adds a blue Catching up tag to a server catching up.

View: Active · Archived, after the search, switches the list between the servers you monitor and the archived ones. Archived lists only archived servers, each tagged Archived (its title gives the date), and its Connectivity filter is disabled: an archived server has no connectivity state, and the states you chose are kept for Active. The view is kept with your other filters in this browser tab. Clear filters, a tag’s link from a server’s Info tab and the notice’s View servers all open Active.

The notice at the top of every page

While any of your servers is unreachable, a notice says how many: 3 servers cannot reach WatchTower15: nothing heard from them for 5 minutes or more. The count includes rejected servers once 5 minutes pass without an accepted heartbeat. View servers opens the Servers list’s Active view on Unreachable and Rejected. You can dismiss it in this tab; it comes back when the number changes. It is shown to members who can view servers. See the notice bar.

What to check on the host

• Is the host running?
• Is the agent running? systemctl status wt15-server-agent says.
• Can the host reach WatchTower15 over HTTPS (port 443)? Check its firewall, proxy and DNS.
• Is the host’s clock right? A Rejected server’s clock is more than 5 minutes off WatchTower15’s: set its time with NTP. Its Clock card shows how far off it is.

An agent that keeps data on the host delivers what it collected while the server was unreachable once it reconnects, and the charts fill in; otherwise data from that time is missing from them. Either way, the charts mark the gap by its cause: see Gaps in a server’s charts. A server you have retired, but that still reads Unreachable, is best archived from its page’s ⋯ menu, or deleted; otherwise it is archived once it has been silent for your organisation’s setting.

When WatchTower15 is not receiving data

WatchTower15 also watches all of its agents together. When it is not receiving data from most of them, every organisation with an active agent sees a notice at the top of every page: WatchTower15 is not receiving data from most agents since 10:12 BST. We are looking into it. (the time in your display timezone). It cannot be dismissed while it lasts. Meanwhile the count of unreachable servers is hidden, since most servers then look unreachable whatever their own state, and an unreachable server’s banner points to this notice instead of the host.

After such an interruption of 15 minutes or more ends, a notice says for 24 hours when WatchTower15 was not receiving or checking data from most agents, and that charts and alarms may have a gap in that window. When at least one of your active agents is a version that keeps data on the host, it says instead that charts fill in from agents that keep data, and that alarms may have a gap in that window. You can dismiss it in this tab. Shorter interruptions are not announced.

These notices say what WatchTower15 observed. They never say whose side a fault was on: the same pattern appears when one large organisation loses its own network.

When WatchTower15 itself is down

If WatchTower15 cannot be reached at all, it cannot tell you so itself. The WatchTower15 status page is hosted outside WatchTower15 and checks the console and the API every minute. It is linked from the bottom of the app’s menu, from both interruption notices and from these docs’ sidebar.

In this section

• Installation — one-line and unattended installs, installer options, and hosts behind a forward proxy.
• Configuration reference — every environment variable and its default.
• Metrics & telemetry — the exact fields collected.
• Fleet & profiles — configure many hosts centrally.
• Upgrades — how self-update works.