Retention, throttling & limits

This page documents the operational envelope of the log pipeline — how long logs stay searchable, the limits that apply at ingest, and how the store is secured.

Retention

The searchable hot window is retained for 30 days. Lines older than that age out of the log store automatically. The 30-day window applies uniformly across all servers and sources in your organisation. Times are kept as UTC instants and shown in your display timezone.

The raw batches the agent ships are also written to a durable object-storage archive, partitioned by date. The archive exists for durability and replay — if the hot store ever needs to be rebuilt or a batch re-processed, the source of truth is intact.

Need longer than 30 days?

Extended and tiered retention is an infrastructure-level concern rather than a per-account toggle today. If your compliance regime requires a longer searchable window, raise it with us — the durable archive already keeps the underlying data, so extending the searchable window is a provisioning change, not data recovery.

Ingest limits

Limit	Value	Behaviour on breach
Lines per batch	1000	The ingest endpoint returns 413. The agent never sends more than 1,000 lines in one request, and splits a request the endpoint refuses as too large.
Body size	5 MB on the wire, 20 MB decompressed	413. The decompressed cap is what stops a small gzip bomb expanding to gigabytes.
Single line size	64 KB	Longer lines are cut at 64 KB, not rejected: by the agent for the lines it ships, by the ingest API for a log source’s.
Request signature clock skew	±300 s of WatchTower15’s clock	Requests outside the window are rejected — keep host clocks in sync (NTP); a server’s Info tab shows whether its clock is synchronised and how far it is from WatchTower15’s.
Body encoding	gzip	Batches are compressed on the wire; the endpoint counts lines without parsing each one on the request thread.

The agent sends at most the lower of WT15_LOG_MAX_LINES and 1,000 lines in one request, so it never trips the per-batch line cap.

Log ingestion limit per billing period

Some organisations have an account limit on how many GB of logs they can send in a billing period, from their servers and log sources together. Most have none. When an organisation reaches it, log ingestion stops until the next billing period starts, and logs sent in the meantime are discarded, not queued:

• servers’ agents carry on as normal and move past the discarded lines;
• log sources are answered 429, with a Retry-After header and data.code log_volume_limit_reached.

Ingestion restarts automatically when the next billing period starts. See log ingestion: stopped.

Back-pressure & delivery guarantees

The pipeline favours not losing lines over lowest possible latency:

• If the durable archive write or the log store fails, the ingest endpoint returns 503 and the agent keeps its place in the file and sends the same lines again.
• A retried range is stored once. WatchTower15 remembers how far each file, and each server’s journal, is stored, and answers a range it already holds without storing it again. After a rare store timeout, one batch can be stored twice.
• The agent follows every source by file identity and position, so a crash or restart resumes exactly where it stopped (see tailing behaviour).

How the store is kept private

• The log store is never exposed to the public internet and is not reachable from outside WatchTower15 by any route.
• All reads go through the WatchTower15 API, which enforces your organisation scope and the logs.view permission on every query. The browser never talks to the store.
• Agent ingest is authenticated per request with an HMAC signature keyed by a per-agent secret.

Current limits worth knowing

In the spirit of honest docs, a few things the pipeline does not do yet:

• There is no per-agent or per-organisation ingest rate cap beyond the per-batch line limit, and no volume cap unless your organisation has a log ingestion limit. Very high-volume sources should be scoped deliberately on the profile rather than firehosed.
• The searchable window is a fixed 30 days (see above).
• Anything that is not a server can push over the log ingest API with a per-source bearer token. Those lines share the same 30-day window, the same per-batch caps, and the same per-GB pricing as agent-shipped logs.

Related

• Logs overview & shipping — the pipeline and what the agent ships.
• Search & explore — querying within the retention window.
• Configuration reference — WT15_LOG_MAX_LINES and friends.
• Account limits & courtesy accounts — the log ingestion limit and the other account limits.