Logs overview & shipping

WatchTower15 includes centralised log collection built on the same least-privilege agent you already run. It tails the log files you nominate and the systemd journal, ships them over an authenticated, durable pipeline, and stores them in a log store built for fast full-text search — ready to search, chart and explore.

Two ways in

	Ships via	Best for
Servers	the agent, tailing files and journald	any machine you can install the agent on — it handles batching, offsets, rotation and back-off
Log sources	the ingest API, one bearer token	anything else: a Lambda, a CI job, a container, a script

Both land in the same store, search the same way, and bill the same way. In the viewer they are both sources, listed together.

A monitored database’s slow query and error logs are parsed and keyed before they leave your server, and included in the database’s price (see Slow and error logs). Read them on the database’s own Logs tab, or in log search beside your servers’ lines, by choosing the database as a source. Either way they are not log volume: they are never billed as such and never count toward your log limits. Shipping their files as log files is separate, and billed as log volume like any other.

Why it’s built this way

Logs are the highest-volume, spikiest data a monitoring platform handles, so the pipeline is designed for durability and cost, not just speed:

• The searchable store is heavily compressed, so a long hot window stays affordable and full-text search stays fast as your volume grows.
• Durable object storage sits in front of it as a buffer and replay source, so a burst or a momentary store outage never loses lines.
• The store is never exposed publicly. All queries go through the WatchTower15 API, which enforces your organisation scope and the logs.view permission.

The pipeline, end to end

  agent (tail files + journald)
    │  gzip NDJSON, HMAC-signed
    ▼
  POST /api/agent/logs/ingest ──▶ object storage (durable archive, date-partitioned)
                                       │
                                       ▼
                                  forwarder ──▶ log store (hot, searchable)
                                                       │
                                                       ▼
                                         WatchTower15 API  ◀── your dashboard
                                         (org-scoped, logs.view)

1. The agent tails each configured source and batches new lines.
2. It gzips them as NDJSON and POSTs to the ingest endpoint with an HMAC signature (the same auth used for telemetry).
3. The endpoint archives the raw batch to object storage and hands off a pointer.
4. A forwarder loads the batch and inserts it into the log store.
5. You query it through the API — the store is never contacted directly by the browser.

What the agent ships

Log files and folders

Which paths a host tails is profile-driven (see fleet & profiles). Set the file and folder list on the profile and every host on it ships the same sources; the WT15_LOG_FILES environment variable is added to the profile’s sources on that host. Each source is labelled with a log_source — by default the file’s basename — which becomes a first-class field you can filter on.

A log file path can name many files with wildcards (below). A log folder path cannot: the profile refuses one holding *, ? or [ with A log folder path cannot hold wildcards: add a log file pattern instead, for example /var/www/*/storage/logs/*.log. Write a log file pattern instead. A folder path saved with a wildcard before this check is still read as written, which normally finds nothing, and the profile’s next save asks you to change it.

Wildcards in log file paths

A log file path can be a pattern that names many files: each day’s laravel-2026-10-13.log, or the same log in every app on the server. Put a wildcard anywhere in the path, in the file name or in a folder:

Wildcard	Matches
*	any run of characters, including none
?	any one character
[abc], [a-z]	one character from the set or the range
[^a-z]	one character not in the set or the range
\	the next character as itself: \*, \?, \[

• Each wildcard matches within one folder level. /var/www/*/storage/logs/laravel-*.log reads /var/www/shop/storage/logs/laravel-2026-10-13.log, never a file a folder deeper. ** is not supported: write a pattern per level, or use a log folder.
• A wildcard also matches a name starting with a dot: /var/log/app/*.log reads .hidden.log too.
• Every matching file is read, including the files that appear later. The agent looks again every 15 seconds, so a new daily file is found within one look.
• Each file is a first sight. Its last 72 hours are imported, within the usual caps, whether it matched when you saved the pattern or appeared later: a new file whose lines carry a time the agent can read is imported from its first line; one with no readable time is followed from the moment it is found. A pattern’s files are imported on their own: their rotated copies (app.log.1) are imported only when the pattern matches them too.
• A file at exactly the saved path wins. A path holding *, ? or [ is read as that one file when a regular file has that very name; otherwise it is a pattern. A symlink with that name does not count.
• Put \ before a *, ? or [ that is part of a file name. /var/log/app\[1.log reads the file app[1.log.
• One file, two patterns: it is read once, under whichever comes first: the WT15_LOG_FILES entries, then the profile’s log files in their order (the journal first, then by path, ignoring case, as the profile lists them). A file you name in full is always read as that file, whatever pattern also matches it.
• Limits, per pattern. Each look reads at most 1,024 folders and keeps at most 256 files. Past 256 it keeps the files it already reads that were written in the last 72 hours, then the newest of the rest. A file it already reads that the limit leaves out is still read until it goes quiet, so no line of it is lost. The agent’s log notes a limit when it is first reached, and again only after the pattern has been back under it.

The log_source of a pattern’s files

All the files one pattern reads share one log_source: the pattern’s file name as you wrote it, wildcards included. The daily files of /srv/shop/storage/logs/laravel-*.log are all laravel-*.log. When a folder in the pattern is a wildcard, the folder each file was found in goes in front, so apps stay apart: under /srv/*/storage/logs/laravel-*.log, the files of /srv/app1/ are app1/laravel-*.log and those of /srv/app2/ are app2/laravel-*.log. A path whose wildcards are all escaped keeps its escape: /var/log/a\*.log reads the file a*.log, labelled a\*.log.

Two patterns, one name. When two patterns on a host would give the same log_source (the same file name, the same number of wildcard folders), each gets its own folder in front: the folder at the first level, counted from /, where their paths differ. For example, a profile with these four patterns:

Pattern	log_source
/srv/shop-api/storage/logs/www-data/laravel-*.log	shop-api/laravel-*.log
/srv/pricing/storage/logs/laravel-*.log	pricing/laravel-*.log
/srv/shop-api/storage/logs/*.log	*.log
/srv/pricing/storage/logs/import-*.log	import-*.log

The two laravel-*.log patterns first differ at their second folder, so each takes its own (shop-api, pricing); the other two clash with nothing and keep their file names. With three or more alike, those that still share a name are told apart again at the next folder where they differ. A wildcard folder at that level adds nothing, since the folder it matched is already in front. WT15_LOG_FILES patterns count too, on the hosts that set them. The names follow from the list of patterns alone: adding or removing a pattern can change them for the files the agent finds from then on, while the files it already reads keep theirs.

A log_source is at most 128 bytes: past that, the folder names in front are dropped, then the file name keeps its last 128 bytes.

On the Log Line page, Stream File shows the path a pattern’s line came from: the pattern with its wildcard folders filled in (/srv/app1/storage/logs/laravel-*.log), or the pattern as written when none of its folders is a wildcard (/srv/shop-api/storage/logs/www-data/laravel-*.log for shop-api/laravel-*.log, /var/log/a\*.log for a\*.log). It shows nothing when the name does not say which path it was: two of the profile’s paths giving that name, a name the 128-byte limit shortened part-way, or a name taken because of a WT15_LOG_FILES pattern, which the profile cannot see.

Patterns and symbolic links

A pattern reads real files through real folders, so that someone who can write in a folder it reaches cannot point it at a file they could not read themselves:

• A symlinked file is never read through a pattern, even when it points at a regular file. Name the file it points to instead.
• A wildcard folder matches real folders only. Under /var/www/*/logs/*.log, a site whose folder is a symlink is skipped: write that site’s path out.
• A fixed folder below a wildcard is followed when it is a symlink only if the user who owns the link also owns the folder it points to, or root owns the link. For example, a deploy whose /var/www/shop/storage is a link owned by deploy to a shared folder owned by www-data is skipped under /var/www/*/storage/logs/*.log. Write that app’s path without the wildcard (/var/www/shop/storage/logs/*.log), or point the pattern at the shared folder.
• The folders before the first wildcard are followed as in any path: /srv/app/current/storage/logs/*.log works with current a symlink to the live release.

The rules hold each time a file is opened, not only when it is found: a matched file, or a folder below a wildcard, swapped for a symlink later is not read through it.

The agent’s own log (/var/log/wt15-server-agent.log) names each link a pattern skipped, log source pattern <pattern> skipped the symlink <path> (a pattern follows no symlink there; see the docs), and each file it did not read because its path came to lead through a symlink, log source pattern <pattern> did not read <path>: its path now leads through a symlink (a pattern follows no symlink there; see the docs). It names the first 16 per pattern, then writes once log source pattern <pattern> skipped more symlinks (not named). A pattern that matches nothing is noted once too: log source pattern <pattern> matches no file on this host (will read matching files once they appear).

A wildcard reads what others put there

A pattern reads every regular file it matches, whoever wrote it. Under /home/*/logs/*.log, any user of the server can add a file to their own logs folder and have its lines shipped to your organisation’s logs. Point wildcards at folders only trusted users can write to.

What the profile refuses

Saving a profile checks each log file path that holds *, ? or [, and shows the reason under the row it refuses. Every save checks all of the profile’s rows, so a row saved before this check that it now refuses must be changed before the profile saves again.

Refused	Example	Message
A pattern not starting with /	logs/*.log	A pattern must be an absolute path (start with /).
An empty, . or .. folder level	/var/log//*.log, /var/log/../*.log	A pattern cannot hold an empty, . or .. folder level.
**	/var/log/**/x.log	** is not supported: a wildcard matches within one folder level. Use a pattern per level, or a log folder.
A class opening with [!	/var/log/app[!0-9].log	[! is not supported: use [^...] to exclude characters.
A [ with no ] in its folder level	/var/log/app[0-9.log	Unclosed [ in the pattern: a bracket class must close within one folder level. Put \ before a [ meant literally.
A class starting with ] or -, ending with -, or empty	/var/log/app[-a].log, /var/log/app[^].log	Malformed [ ] class in the pattern: put \ before a ], - or ^ meant literally.
A \ ending a folder level	/var/log/app\/*.log	A \ at the end of a folder level escapes nothing.

In a pattern, [!a] would mean “! or a”, not “anything but a”, which is why it is refused: write [^a].

The systemd journal

On hosts with systemd, the agent also ships the journal via journalctl, labelled log_source = journald. It reads through the systemd-journal group the service user already belongs to — no elevated privilege.

• Each entry keeps its own time: the moment journald recorded it, not the moment the agent read it.
• It resumes from what WatchTower15 has stored. The agent moves its place in the journal only once WatchTower15 has stored the entries before it, so a failed send or a restart loses no entry.
• The first time, the last 72 hours. When the agent first reads a server’s journal, it also imports the entries from the last 72 hours, as it does for a log file. They count as log volume.
• Every systemd version from 219. That includes Amazon Linux 2 and RHEL 8, whose systemd is older than 242. If such a server’s journal has not reached WatchTower15 before, the agent’s first read of it imports its last 72 hours, as log volume.
• Entries of every user are kept, those of programs running as a regular user (such as ec2-user or ubuntu) included.

Tailing behaviour you can rely on

• The last 72 hours on first sight. When the agent first reads a log file, it also imports the lines already written in it during the last 72 hours. A first sight happens when you install the agent, add a file to a profile, move a server onto a profile, or when a new file appears in a watched folder or starts to match a pattern. The import is capped:

  • 64 MB per file. A file’s plain rotated copies (syslog.1, syslog-20261005) share its 64 MB. Compressed copies (.gz and the like) are never imported.
  • 256 MB per server per UTC day, across all its files and the journal.

  Imported lines count as log volume. Each keeps the time written in it. The agent reads six time formats: ISO 8601 with a zone (2026-10-06T09:13:03.123Z), ISO-like times without one (2026-10-06 10:13:03, read in the server’s time zone), syslog (Oct 6 10:59:27), the Common Log Format of web servers ([06/Oct/2026:10:13:03 +0100]), auditd (msg=audit(1759741983.123:456)) and PHP ([06-Oct-2026 10:13:03]). A file with no readable time in its first 100 lines is not imported: the agent follows it from its end.

  A file’s new lines appear once its import has finished. Imports are sent paced, so a full 64 MB takes about half an hour, and longer when several files start at once. A small file takes a few minutes. Other files’ new lines are not held up.

• Files are followed, not names. The agent follows each file by its identity, so a rename or a rotation never loses or repeats a line. A rotated file is read to its end before the agent lets it go. Its last lines, and its imported copies, keep the file’s log_source: syslog, not syslog.1. A copy made by copytruncate is recognised as the file’s earlier lines and never imported again.

• Nothing is sent wrong. Before sending, the agent checks that the bytes it noted are still there. A line that can no longer be read (a file truncated, or rotated away and compressed while the agent was stopped) is recorded as lost in the agent’s log, never sent with other content.

• Nothing skipped after a restart. Lines written while the agent was stopped are sent when it starts again, if they are less than 72 hours old. After a longer stop, older lines are recorded on the server as lost, not sent. Each of these lines carries the time written in it when that time is plausible, otherwise the time the agent read it. They are sent paced, about 200 lines a second for the whole server, before that file’s new lines. So a file that wrote r lines a second during a stop of T seconds shows its new lines about r × T / 200 seconds after the agent starts again:

  • up to about a minute after a short restart or an upgrade;
  • about 15 minutes after an hour’s stop, at 50 lines a second;
  • about 2.5 hours after an hour’s stop, at 500 lines a second.

• Whole lines only. A partial final line is held until its newline arrives, so you never see torn log entries.

• Ordered. A new line carries the time the agent read it, distinct for every line of a server, which keeps cursor-based paging stable even for bursts written in the same millisecond. An imported line carries the time written in it, kept within the precision it was written at: a line written to the second stays inside that second. Rarely, two imported lines from different files with the same written time share a timestamp, as lines from a log source with whole-second times can. When a server’s clock is off, the times are moved onto WatchTower15’s clock.

• Each line once. WatchTower15 remembers how far each file, and the journal, is stored, so a retried range is stored once. After a rare store timeout, one batch can be stored twice.

• Shown in your timezone. Each line’s time is shown to the millisecond in your display timezone, which the page names, and Copy copies it with its offset, so a pasted time means the same moment anywhere.

Log levels

The agent classifies each line’s severity by scanning it for the usual level keywords — emergency, alert, critical/fatal, error, warning/warn, notice, info, debug. Lines that don’t match a known keyword are stored with no level (and are still fully searchable). Levels power the histogram and level facets.

Turning logging off

Set WT15_DISABLE_LOGGING=1 to stop a host collecting and shipping logs entirely while leaving telemetry untouched — useful for a host under investigation or one with strict egress rules. Switching logging off on the profile does the same for every host on it.

Nothing written while logging is off is imported when it comes back on, from files or from the journal. The agent starts again from the end of each file and from the newest journal entry. Lines it had read but not sent when logging went off are not sent either.

Related

• Log sources — pushing logs from things that are not servers.
• Log ingest API — the endpoint, payload and limits.
• Search & explore — query syntax, histograms, facets, context.
• Slow and error logs — a database’s slow statements and error log, readable in log search as a source, never billed as log volume.
• Retention, throttling & limits — how long logs are kept and the ingest limits.
• Fleet & profiles — choosing what to ship.