blog-post

Log Retention in Manticore Search: Handling Late Events and Replays

author image
View as markdown

A basic log retention policy deletes old logs. That part is easy. Making sure they stay deleted is harder. A log forwarder reconnects after an outage and resends events your cleanup job already removed. Without a check at ingestion, those expired logs can reappear in search.

This article shows how to avoid that problem with a seven-day event-time cutoff shared by ingestion and deletion in a self-hosted Manticore real-time (RT) table. The example is runnable: you can check the SQL responses and see what happens when an expired event is replayed.

Our Fluent Bit integration explains how to send logs to Manticore, while the OpenSearch Dashboards integration shows how to explore them. This article focuses on late arrivals and retries, rather than bulk loading historical logs.

Why Manticore Fits This Workflow

The same RT table handles full-text searches, timestamp filters, retries by document ID, and deletion. Manticore provides the SQL operations; the ingestion application decides what belongs in the retained window. You do not need a separate index just to query these example logs.

Part of the workflowWho handles it in this example
Search log messages and filter by timestampManticore SQL
Replace retries for the same document IDManticore REPLACE
Delete expired documentsManticore DELETE
Compact disk chunksManticore OPTIMIZE
Assign stable IDs and reject conflicting payloadsLog source / application
Enforce the event-time cutoff on every writeApplication
Schedule cleanup, coordinate writers, and handle quarantineApplication

That is the division of work shown in this article, not a built-in retention scheduler.

Event Time and Ingestion Time

event_ts comes from the log source; ingested_ts records when our application accepted the delivery. Both are integer UTC epoch seconds.

Retention by ingestion time gives a late log a new seven-day window on arrival. Retention by event time can reject it immediately if it is already too old. We use event time because searches should cover when events happened, not when they reached the server.

Each operation runs serially and computes cutoff = now - 604800. We keep events at the cutoff and delete only those older than it. Here is the admission policy:

InputActionReason
Integer event time below the cutoffDrop from the search pipelineAlready outside the chosen window
Integer event time at or above the cutoff, at most 300 seconds ahead of nowAcceptLate arrivals within the window remain useful
Missing or noninteger time, or more than 300 seconds aheadQuarantineCheck the timestamp or source clock

The 300-second allowance is an application setting. Accepted future events stay out of time-bounded searches until their timestamp arrives. In this lab, quarantined timestamps stay in memory until a JSON file is written at the end; a crash could lose them. A production ingestion service needs durable quarantine, as discussed below. Track dropped events separately from SQL errors.

Admission and cleanup share the same cutoff; expired replay never reaches the table.

Manticore performs the SQL writes, searches, and deletes. The application decides which events are allowed and when cleanup runs. The example does not configure automatic TTL or daily partitions.

A Small Searchable Schema

The lab runs Manticore 29.9.0, pinned to Docker digest sha256:461b419739a2bbbffb4be0b32db26c41d7b435bf7fd1fce95b414ca5ccbb3775. Its isolated container has a one-CPU limit, 1 GiB of memory, and a loopback-only HTTP port.

CREATE TABLE logs (
  message text,
  service string,
  event_ts timestamp,
  ingested_ts timestamp
);

message supports full-text search, and the timestamps allow numeric filtering. Each event gets a stable, globally unique positive integer ID within Manticore's supported range. Under the lab's immutable-event contract, every retry uses the same ID:

REPLACE INTO logs
  (id, message, service, event_ts, ingested_ts)
VALUES
  (4, 'gateway timeout', 'api', 1792450800, 1792454400);

REPLACE updates the same row for identical retries, leaving one searchable copy. This is not exactly-once delivery. An accepted retry refreshes ingested_ts, so it is not a first-seen timestamp. A reused ID with different content overwrites the old event: REPLACE does not compare payloads. Keep IDs unique across sources and reject conflicting payloads upstream.

Check admission before every REPLACE, including retries. Otherwise a deleted, expired event can be written again under the same ID. We confirmed this with a direct REPLACE against Manticore 29.9.0 outside the bundled lab.

Run the Boundary Checks

Download the runnable lab and its SHA-256 checksum . It needs Docker Compose v2 and Python 3, with no extra Python packages. The first run may download the pinned image.

sha256sum -c retention-lab.tar.gz.sha256
tar -xzf retention-lab.tar.gz
cd retention-lab
sha256sum -c SHA256SUMS
./run.sh

The runner checks port 19707, starts Compose project manticore-article07, runs the assertions, and removes its containers and data volume when finished. It recreates both lab tables, so use only the disposable engine. If the port is occupied, run ARTICLE07_PORT=19717 ./run.sh (or choose another free port). Run one copy at a time.

The script injects an application time of 2026-10-20 00:00:00 UTC (1792454400), giving a cutoff of 2026-10-13 00:00:00 UTC (1791849600). Manticore still runs on its real system clock; only the application's clock advances during the tests.

FixtureInitial resultAfter advancing the injected clock one second
ID 1, cutoff minus one secondDropped by event-time admissionStill absent
ID 2, exactly the cutoffAcceptedDeleted; replay dropped
ID 3, cutoff plus one secondAcceptedRetained at the new cutoff
ID 4, one hour old, replayedOne searchable documentRetained
ID 5, now plus 301 secondsQuarantinedNever written
ID 6, noninteger timestampQuarantinedNever written

At the boundary on October 20, 2026, the cutoff is October 13 at 00:00:00 UTC. A log from October 12 at 23:59:59 is rejected immediately, while one stamped exactly at the cutoff is accepted. One second later, that second event expires and cleanup removes it. Its next delivery is rejected too. A deleted log stays deleted, even after replay.

As a control, the ingestion_control table stores ID 1 with its old event time but a fresh ingestion time. A deletion based on ingested_ts < cutoff leaves it in place. That is the difference between the two retention policies.

Other tests accept an event exactly 300 seconds in the future but hide it until the query's upper bound reaches its timestamp. Missing, boolean, and fractional timestamps are quarantined. Retrying ID 4 moves ingested_ts forward without changing event_ts.

The fixture’s accepted IDs are initially [2, 3, 4]. After advancing one second, the application executes:

DELETE FROM logs WHERE event_ts < 1791849601;

The running engine returns [3, 4] after cleanup. Replaying ID 2 is rejected at admission, so the result stays [3, 4]; another cleanup also leaves it unchanged. The point is not just that the old event was deleted: replaying it did not bring it back.

A duplicate INSERT for ID 4 returns HTTP 500 with {"error":"duplicate id '4'"}. That's a duplicate-ID error, not a retention rejection. The runner checks the error body and stores it alongside successful SQL responses in evidence/sql.jsonl.

Queries That Respect the Window

Before advancing the clock, this query finds API timeouts in the seven-day event-time window, newest first:

SELECT id, message, event_ts, ingested_ts
FROM logs
WHERE MATCH('@message timeout')
  AND service = 'api'
  AND event_ts >= 1791849600
  AND event_ts <= 1792454400
ORDER BY event_ts DESC, id ASC
LIMIT 20;

The IDs come back as [4, 3, 2]. One second later, use 1791849601 and 1792454401 as the lower and upper bounds; ID 2 drops out. In a service, derive both values from one trusted clock reading and serialize query values safely. With a durable cutoff, use max(persisted_cutoff, now - 604800) for reads, admission, and cleanup. This lab uses fixed values, not arbitrary log text or user input.

Filtering by event time hides expired rows from search even when cleanup runs late, but it doesn't free disk space. Confirm a conditional DELETE by reading the rows back.

Deletion and Disk Space Are Different Checks

The lab flushes two disk chunks, using a replay of ID 4 to create the second. It records the file names and logical lengths inside the logs table directory before deletion, after deletion, and after:

OPTIMIZE TABLE logs OPTION sync=1, cutoff=1;

The table files totalled 266,745 bytes before deletion and 133,773 bytes after compaction. Deleting rows alone did not reduce those file sizes. The post-compaction readback still returned [3, 4].

These are file lengths for a tiny synthetic table, not allocated disk blocks, whole-volume usage, or a capacity estimate. Binlogs, backups, container logs, and quarantine take space elsewhere. The archive keeps the individual file captures so you can check the calculation.

The compaction manual explains the chunk cutoff and synchronous option. Compaction can use significant I/O and temporary disk space, so schedule it around workload and disk pressure rather than running it after every event.

Log Retention Best Practices for Production

The downloadable lab runs operations one at a time with an injected application clock. It demonstrates the retention rule, but a live ingestion service also needs to handle these cases:

  • Coordinate ingestion and cleanup. With multiple workers, one can validate an event, pause, and write it after another advances cleanup. Coordinate admission, the actual write, and cleanup as one protected sequence. A time filter can hide a wrongly reinserted row but cannot prevent wasted storage. Direct SQL writes that bypass admission pose the same risk.

  • Make the cutoff durable and monotonic. Persist a nondecreasing cutoff before deleting against it, and use it for reads and admission too. A backward clock jump must not reopen expired history. Detect large forward jumps before advancing the cutoff, or you could permanently delete valid data. Check source timestamps as well: plausible times are not necessarily truthful.

  • Keep quarantine and retry state durable. The example buffers quarantined timestamps in memory and writes JSON only at the end. For production, use a durable queue with access controls, an expiry policy, and a review or replay process. Define acknowledgement and crash-recovery behavior for the ingestion service.

  • Watch cleanup and capacity. Schedule cleanup, alert on failures and lag, and monitor accepted, dropped, and quarantined events. A seven-day window is not a disk quota: arrival rate, event size, merge overhead, and cleanup delays still determine disk use. This example does not test throughput or replication.

Before deploying, delete an expired event, replay it through the real ingestion service, and confirm it stays absent. Repeat after a worker restart. The bundled test demonstrates the sequential replay case; production restart safety also needs the durable cutoff and coordinated writes described above.

Takeaway

To keep expired logs from returning:

  1. Use the same event-time cutoff for ingestion, search, and cleanup.
  2. Check it before every write, including retries.
  3. Coordinate writes with cleanup so an expired row cannot be inserted again.

Manticore handles storage and search; the application enforces the retention policy.

Go from zero to Manticore in seconds

Install Manticore Search in one command on Linux or macOS:

curl https://manticoresearch.com | sh

For advanced installation options, see the full installation guide and the manual .