Skip to content
WatchTower15Docs

Configuration reference

The agent is designed to need almost no local configuration — most behaviour comes from the server profile it’s assigned to, served from the backend and refreshed automatically. The settings below are the local overrides, mostly used at install time or for special environments.

Environment variables

Section titled “Environment variables”

The agent reads its settings from /etc/wt15-server-agent/agent.env, the unit’s environment file. The installer writes that file from its options, and rewrites it when you reinstall. To change a setting on an installed host, edit the file and run sudo systemctl restart wt15-server-agent.

Exporting a variable in your shell before running the installer does not reach the agent, except for the variables the installer copies into the file: WT15_ORG_TOKEN, WT15_PROXY, WT15_PROXY_PORT, WT15_DISABLE_LOGGING, WT15_DISABLE_TELEMETRY and WT15_SUPPLIER_IDENTIFIER, WT15_PROFILE, which it writes as WT15_PROFILE_UUID or WT15_PROFILE_SLUG, and WT15_TAGS. All variables are optional unless noted.

Variable Default Description
WT15_ORG_TOKEN — Organisation install token used at enrollment. Usually passed as the installer argument instead. Keep it in a secret store, not an image layer.
WT15_API_URL the API the installer was downloaded from Base URL of the WatchTower15 API. Set by the installer; you do not normally change it.
WT15_PROFILE_UUID / WT15_PROFILE_SLUG org default Assigns the host to a profile at first enrollment (the UUID wins if both are set). A uuid or slug your organisation does not have stops the enrollment: the agent logs profile "<value>" not found in this organisation; not enrolling and exits, with no retries, rather than joining the default profile. The installer writes one of them, whichever the value is, from --profile= or WT15_PROFILE, and only when one is given (see Installer options).
WT15_SUPPLIER_IDENTIFIER auto-detected The provider’s ID for the host. Detected on AWS EC2 (the instance ID) and DigitalOcean (the droplet ID), and in a Docker container it is the container’s ID, new when the container is re-created; elsewhere it is empty unless you set it. An agent installed again continues its server only on a host that has one, so on other hosts set a stable ID: the provider’s instance or server ID, not /etc/machine-id, which an OS reinstall regenerates. Give the same value on every install of that host: the installer keeps it in /etc/wt15-server-agent/agent.env, which uninstalling removes. Set by --supplier-identifier=.
WT15_CREDENTIALS_PATH /var/lib/wt15-server-agent/credentials.json Where the agent stores its credentials.
WT15_DISABLE_TELEMETRY 0 Set to 1 to disable telemetry collection and sending. Set by --no-telemetry.
WT15_DISABLE_LOGGING 0 Set to 1 to disable log collection and shipping entirely. Set by --no-logging.
WT15_LOG_FILES — Comma-separated list of log files to tail on this host, added to the profile’s log files and folders: both are read. An entry can be a pattern with *, ? or […], which cannot hold a , (the comma separates the entries). Nothing checks a pattern here before the agent reads it: a malformed one is not read, and the agent’s log says why, once: log source pattern <pattern> is malformed: <reason>; nothing is read from it, the reason being the message the profile would show.
WT15_LOG_MAX_LINES 1000 The most log lines the agent sends in one request. An upper bound only: a value above 1,000, the ingest endpoint’s per-batch cap, is read as 1,000.
WT15_LOG_STATE_PATH logs-state.json beside the credentials file Only where the agent looks for an older agent’s log checkpoint file, to take over its positions at its first start. The agent keeps its own log positions in spool/logs/ (below).
WT15_AGENT_LOG_PATH /var/log/wt15-server-agent.log The agent’s own diagnostic log. It is the agent’s complete record; errors, warnings and notices are also sent to the systemd journal.
WT15_AGENT_LOG_LEVEL info How much the agent writes to its own log. info records start-up, enrollment, configuration changes, errors, and the warnings and notices of an outage: one error when it starts (not one per failed attempt), a warning every hour while it lasts, and notices when it ends and when the kept data is delivered. debug also records each successful heartbeat and data push, and each failed attempt, which helps when troubleshooting. error records errors only, so it hides the hourly still cannot reach WatchTower15 warning and the recovery notices. Any other value is read as info.
WT15_PROXY — HTTP(S) forward proxy for WatchTower15 API traffic only, e.g. http://proxy.corp.example:3128. Set by the installer’s --proxy=. Percent-encode any special character in credentials (@ becomes %40). See Behind a forward proxy.
WT15_PROXY_PORT — The proxy’s port (1-65535) when WT15_PROXY does not include one. Set by --proxy-port=. If WT15_PROXY includes a different port, the agent refuses to start rather than pick one. Setting it without WT15_PROXY is also an error.
WT15_TAGS — The tags the server gets at its first enrollment, joined by commas: <group>=<tag>, <group-uuid>=<tag> or <tag-uuid>, at most 20 (WT15_TAGS=Environment=production,Team=billing-api). Set by the installer from --tag= (see Tags at install). Unlike WT15_TAG_<Name>, they are always applied, whatever the profile’s server-led tagging and its filter, several tags of one group are allowed, and a group or tag that does not exist yet is created. A tag WatchTower15 cannot use (a deleted name, another resource type’s group) is left out, never a refused enrollment, and the agent logs install tag <tag> dropped: <reason>.
WT15_TAG_<Name> — In a container with server-led tagging on, each becomes the tag <Name>: <value> (prefix removed), its name and value stored to the naming rule like any reported tag. The agent reads no other container variable as a tag.
HTTPS_PROXY / NO_PROXY — Standard proxy variables. They are a fallback only, used for WatchTower15 API traffic when WT15_PROXY is not set. A database collector’s calls to AWS (Secrets Manager, SSM Parameter Store, and cloud metrics from CloudWatch and the RDS API) always use them, never WT15_PROXY. They are never used for cloud metadata (the instance metadata service at 169.254.169.254).

Profile-driven settings (served from the backend)

Section titled “Profile-driven settings (served from the backend)”

These are not local environment variables — they are delivered as part of the agent’s configuration document (fetched on enrollment and whenever the X-Agent-Config-Etag changes). Manage them on the profile:

Setting Typical value Controls
Telemetry on/off & options all metrics Whether host telemetry is pushed, and all of it or a chosen subset.
Log files & folders profile list Which paths the agent tails and ships. A log file path can be a pattern.
Server-led tagging on/off + filter Whether the agent self-reports tags (from a container, only WT15_TAG_<Name> variables), and the include/exclude filter applied.

Because log paths and tagging live on the profile, you can change an entire fleet from one place without touching a single host — the change propagates on the next configuration poll. Each agent also keeps the last configuration it received on its host, so a restart while WatchTower15 is out of reach does not leave it without a profile (see The profile kept on the host).

The configuration document also carries the agent’s collection intervals and per-process limit. Those are set by WatchTower15 rather than on the profile — see Collection settings. So are the limits on what the agent keeps while it cannot reach WatchTower15.

The credentials file

Section titled “The credentials file”

At enrollment the agent stores its identity (the server’s agent ID, the agent secret it signs requests with, and its self-reported tag baseline) in /var/lib/wt15-server-agent/credentials.json. The file is owned by the wt15-server-agent service user and readable only by that user and root (mode 0600); the secret is stored as-is, so treat the file like a password. This is what lets the agent sign requests and compute tag deltas without re-enrolling. Deleting it forces a fresh enrollment on next start. On a host WatchTower15 already knows by its instance ID and host name, that enrollment continues the same server at once (see Lifecycle at a glance), and the agent keeps the data it held for it. Otherwise it is a new server, and the agent deletes the data it kept for the old one.

An agent whose every request is refused with 403 Invalid agent signature, after another install enrolled with its host’s instance ID and host name, has had its server taken over by that install. Delete its credentials.json and restart it, which takes the server back (and refuses the other agent in turn), or install it again with its own --supplier-identifier.

The profile kept on the host

Section titled “The profile kept on the host”

The agent keeps the last configuration WatchTower15 sent it in /var/lib/wt15-server-agent/config-cache.json, beside the credentials file and with the same protection (mode 0600, owned by the wt15-server-agent service user). It holds what the configuration document carries: the profile’s settings and log paths, any disk usage tasks, and the connection settings of the databases the agent collects, with references to their credentials. It never holds a password or a key.

The agent writes the file when its configuration changes, and reads it once at start-up, before it contacts WatchTower15. An agent restarted while it cannot reach WatchTower15 (a network outage, a firewall change) therefore collects with the profile it last received rather than with none. The first configuration it fetches from WatchTower15 replaces it.

  • A damaged or unreadable file is ignored. The agent writes one error line at start-up naming the reason (it reaches the journal too, so an alert on agent errors sees it), starts as if there were no file, and replaces it at its next fetch.
  • The start-up line says where the configuration came from (see Verifying the running version). Its saved … ago is when the configuration last changed, not when WatchTower15 last confirmed it.
  • Deleting the file is safe. A running agent is not affected. At its next start it fetches its configuration as usual and writes the file again; only a start while WatchTower15 is out of reach then finds no profile to use.
  • A removed server’s agent deletes it. Once the server is deleted or its access revoked in WatchTower15, its agent deletes this file and the data it kept, keeps its credentials, and stays idle, checking in once an hour, until it is uninstalled (see How the agent talks to the backend). A deleted server’s agent, from 1.1.48, also asks its uninstall unit to remove it (see Deleting a server).

What the agent keeps on the host

Section titled “What the agent keeps on the host”

The agent keeps its state in /var/lib/wt15-server-agent, owned by the wt15-server-agent service user and readable only by that user and root (the directory is mode 0700):

Path What it holds Written Size
wt15-server-agent The agent binary At install and at each self-update about 15 MB
wt15-server-agent.prev The previous binary, kept by a self-update as its backup At each self-update about 15 MB
credentials.json The server’s identity and agent secret (above) At enrollment, and when the server’s self-reported tags change under 1 KB, plus the tags
config.json The capability switches given at install (--no-logging, --no-telemetry) At every start under 1 KB
config-cache.json The last configuration WatchTower15 sent (above) When the configuration changes a few KB
databases.json The databases the agent collects, with references to their credentials, never a password When a configuration is applied a few KB
systemd-oom.json How far the agent has read the kernel’s out-of-memory kills, each service’s count, and the processes killed in the last 15 minutes (their short command names and process IDs, at most five) When it counts new kernel entries under 1 KB
spool/ The data waiting until WatchTower15 has stored it (When WatchTower15 cannot be reached) Every 15 seconds about 4 MB at most while connected; during an outage up to 64 MB of telemetry, plus 24 MB for each database collected, or collected in the last 72 hours
spool/logs/files.json Where each log file and the journal stand: how far each file has been read and stored, a checksum of its first and last bytes, and the journal’s place. Positions, never log lines At most once a minute while logs are sent, and at once when a file is first seen, rotated or let go a few KB; one small entry per log file
spool/logs/segments/ The parts of log files still to send, as byte ranges and times, never log lines (When WatchTower15 cannot be reached) Every 15 seconds while a log file grows about 4 MB at most while connected; during an outage about 0.7 MB a day for each log file that grows all the time, up to 64 MB
spool/services/ The results of the service checks this server runs, waiting until WatchTower15 has stored them: state.json (how far they were delivered) and segments/ (the results, never a parameter’s value) Every 15 seconds while it checks a service small while connected; during an outage up to 32 MiB, the oldest dropped first
status.json The agent’s state, printed by --status When its state changes, at most once a minute during an outage a few KB
disk-scan-state.json Which disk usage reports are done, and when each volume was last walked When a walk finishes or a report is delivered a few KB
uninstall-requested The agent’s request to be removed, written when WatchTower15 deletes the server: its uninstall unit then removes the agent. One line saying when and why; nothing reads it Once, when the server is deleted (agent 1.1.48 or later) under 1 KB
db-<database-uuid>-schema.jsonl.gz The last accepted schema of a database this server collects as the active collector, names in clear, for schema-change history. It never leaves the host. After a schema pass that found the schema changed, or had no earlier state, once WatchTower15 has accepted it gzip; about 11 MB before compression at the 60,000-column limit, capped at 32 MiB before compression
db-<database-uuid>-schema.pass.gz, db-<database-uuid>-schema.baseline.gz, db-<database-uuid>-schema.jsonl.gz.tmp A schema pass waiting for WatchTower15’s answer: the changes it sends, its baseline without names, and the next state When a pass is ready to send; kept until WatchTower15 accepts or refuses it up to about 3 MB for the first two together

spool/ holds FORMAT (the server the data belongs to), manifest.json (what has been delivered), segments/ (telemetry), state/ (the latest system-information, database and disk usage snapshots), journal.json (the outages not yet reported to WatchTower15), refused.json (the database data WatchTower15 refused, by database and time range), database_shares.json (when each database stopped being collected here, for the telemetry limit), logs/ (the log positions above) and services/ (the service check results above). On a server without databases its writes come to about 54 MB of pages written a day, plus about 6.4 MB a day for each database the agent collects, plus about 20 MB a day for a log file growing at 10 lines a second (about 23 bytes a line), with the filesystem’s own journal on top. Nothing is forced to disk on each write.

  • File-integrity monitoring. Exclude /var/lib/wt15-server-agent/spool/: its files change every 15 seconds by design, spool/logs/files.json and spool/logs/segments/ included. status.json also changes in normal operation, the db-<database-uuid>-schema files after a schema pass, and the binary at each self-update.
  • Database schema state. The db-<database-uuid>-schema files exist only on a database’s active collector, mode 0600. The agent deletes a database’s files when this server stops being its active collector (a handover, a pause, the database deleted, or this server leaving its pool), and all of them when the server is removed from WatchTower15. A restart keeps the state and drops a pass still waiting for an answer: the next pass reports its changes. A damaged state file, or one over its cap, is deleted, and the next pass starts again from WatchTower15’s baseline.
  • Nothing is kept anywhere else. Outside this directory the agent writes only its own log in /var/log; its configuration in /etc/wt15-server-agent is written by the installer (and services.d/ by you, below), as are the uninstall script in /usr/local/lib/wt15-server-agent and its two units. The uninstall commands remove them all.

Service checks: services.d/ and services.allow

Section titled “Service checks: services.d/ and services.allow”

A server that is a service’s collector reads the service’s parameters itself. Two places under /etc/wt15-server-agent are yours, and WatchTower15 never writes either:

Path What it is Owner and mode
services.d/ The agent’s managed parameter files, one <service-uuid>.env per service, written by service-param add (below). A File parameter may read anything here without a services.allow line. wt15-server-agent, 0750; each file 0600
services.allow Your limits: which files outside services.d/ a parameter may read (file), which URLs a parameter may fetch (url), and where checks may go and which parameters each destination may receive (destination). The agent reads it and never writes it; the installer never creates it. root:wt15-server-agent, 0640

The Services page has the services.allow syntax and an example. A line the agent cannot read makes the whole file invalid, and an invalid or unreadable file refuses everything it would allow.

Every parameter file, and services.allow itself, is read with the agent’s file-reading capability set aside, so the file’s own permissions apply: grant the agent’s user read access to a file it should read, for example setfacl -m u:wt15-server-agent:r /var/www/orders/current/.env. The agent never reads its own configuration (except services.d/), its state directory, /proc, /sys, /dev, /tmp or /var/tmp as a parameter.

service-param add writes a value into services.d/, for a value with no other home:

Terminal window
printf '%s' "$TOKEN" | sudo /var/lib/wt15-server-agent/wt15-server-agent service-param add <service-uuid> API_TOKEN

It runs as root, reads the value from standard input (without echoing it at a terminal), writes API_TOKEN="…" into services.d/<service-uuid>.env, replacing an earlier API_TOKEN line, and never prints, logs or sends the value. The key is letters, digits and underscores, not starting with a digit; the service’s uuid is shown in its Parameters section. Run it on each of the service’s collectors.

Section titled “Related”