---
title: "Log Retention in Manticore Search: Handling Late Events and Replays"
description: "How to build an event-time log retention policy that keeps expired logs from reappearing, with a runnable Manticore example covering late arrivals, retries, and cleanup."
date: 2026-11-17
author: "Dmitrii Kuzmenkov"
image: https://manticoresearch.com/images/blog/log-retention-late-events.png
lang: en
url: https://manticoresearch.com/blog/log-retention-late-events/
translations:
  ru: https://manticoresearch.com/ru/blog/log-retention-late-events/
  zh: https://manticoresearch.com/zh/blog/log-retention-late-events/
---

# Log Retention in Manticore Search: Handling Late Events and Replays

How to build an event-time log retention policy that keeps expired logs from reappearing, with a runnable Manticore example covering late arrivals, retries, and cleanup.

A basic log retention policy deletes old logs. That part is easy. Making sure they stay deleted is harder. A log forwarder reconnects after an outage and resends events your cleanup job already removed. Without a check at ingestion, those expired logs can reappear in search.

This article shows how to avoid that problem with a seven-day event-time cutoff shared by ingestion and deletion in a self-hosted Manticore real-time (RT) table. The example is runnable: you can check the SQL responses and see what happens when an expired event is replayed.

Our [Fluent Bit integration](/blog/integration-of-manticore-with-fluentbit/) explains how to send logs to Manticore, while the [OpenSearch Dashboards integration](/blog/manticore-opensearch-dashboards-integration/) shows how to explore them. This article focuses on late arrivals and retries, rather than bulk loading historical logs.

## Why Manticore Fits This Workflow

The same RT table handles full-text searches, timestamp filters, retries by document ID, and deletion. Manticore provides the SQL operations; the ingestion application decides what belongs in the retained window. You do not need a separate index just to query these example logs.

| Part of the workflow | Who handles it in this example |
| --- | --- |
| Search log messages and filter by timestamp | Manticore SQL |
| Replace retries for the same document ID | Manticore `REPLACE` |
| Delete expired documents | Manticore `DELETE` |
| Compact disk chunks | Manticore `OPTIMIZE` |
| Assign stable IDs and reject conflicting payloads | Log source / application |
| Enforce the event-time cutoff on every write | Application |
| Schedule cleanup, coordinate writers, and handle quarantine | Application |

That is the division of work shown in this article, not a built-in retention scheduler.

## Event Time and Ingestion Time

`event_ts` comes from the log source; `ingested_ts` records when our application accepted the delivery. Both are integer UTC epoch seconds.

Retention by ingestion time gives a late log a new seven-day window on arrival. Retention by event time can reject it immediately if it is already too old. We use event time because searches should cover when events happened, not when they reached the server.

Each operation runs serially and computes `cutoff = now - 604800`. We keep events **at** the cutoff and delete only those older than it. Here is the admission policy:

| Input | Action | Reason |
| --- | --- | --- |
| Integer event time below the cutoff | Drop from the search pipeline | Already outside the chosen window |
| Integer event time at or above the cutoff, at most 300 seconds ahead of now | Accept | Late arrivals within the window remain useful |
| Missing or noninteger time, or more than 300 seconds ahead | Quarantine | Check the timestamp or source clock |

The 300-second allowance is an application setting. Accepted future events stay out of time-bounded searches until their timestamp arrives. In this lab, quarantined timestamps stay in memory until a JSON file is written at the end; a crash could lose them. A production ingestion service needs durable quarantine, as discussed below. Track dropped events separately from SQL errors.

![Admission and cleanup share the same cutoff; expired replay never reaches the table.](./log-retention-late-events/retention-policy.png)

Manticore performs the SQL writes, searches, and deletes. The application decides which events are allowed and when cleanup runs. The example does not configure automatic TTL or daily partitions.

## A Small Searchable Schema

The lab runs Manticore 29.9.0, pinned to Docker digest `sha256:461b419739a2bbbffb4be0b32db26c41d7b435bf7fd1fce95b414ca5ccbb3775`. Its isolated container has a one-CPU limit, 1 GiB of memory, and a loopback-only HTTP port.

```sql
CREATE TABLE logs (
  message text,
  service string,
  event_ts timestamp,
  ingested_ts timestamp
);
```

`message` supports full-text search, and the timestamps allow numeric filtering. Each event gets a stable, globally unique positive integer ID within Manticore's supported range. Under the lab's **immutable-event contract**, every retry uses the same ID:

```sql
REPLACE INTO logs
  (id, message, service, event_ts, ingested_ts)
VALUES
  (4, 'gateway timeout', 'api', 1792450800, 1792454400);
```

[REPLACE](https://manual.manticoresearch.com/Data_creation_and_modification/Updating_documents/REPLACE) updates the same row for identical retries, leaving one searchable copy. This is not exactly-once delivery. An accepted retry refreshes `ingested_ts`, so it is not a first-seen timestamp. A reused ID with different content overwrites the old event: REPLACE does not compare payloads. Keep IDs unique across sources and reject conflicting payloads upstream.

Check admission **before every REPLACE**, including retries. Otherwise a deleted, expired event can be written again under the same ID. We confirmed this with a direct `REPLACE` against Manticore 29.9.0 outside the bundled lab.

## Run the Boundary Checks

[Download the runnable lab](./log-retention-late-events/retention-lab.tar.gz) and its [SHA-256 checksum](./log-retention-late-events/retention-lab.tar.gz.sha256). It needs Docker Compose v2 and Python 3, with no extra Python packages. The first run may download the pinned image.

```sh
sha256sum -c retention-lab.tar.gz.sha256
tar -xzf retention-lab.tar.gz
cd retention-lab
sha256sum -c SHA256SUMS
./run.sh
```

The runner checks port 19707, starts Compose project `manticore-article07`, runs the assertions, and removes its containers and data volume when finished. It recreates both lab tables, so use only the disposable engine. If the port is occupied, run `ARTICLE07_PORT=19717 ./run.sh` (or choose another free port). Run one copy at a time.

The script injects an application time of `2026-10-20 00:00:00 UTC` (`1792454400`), giving a cutoff of `2026-10-13 00:00:00 UTC` (`1791849600`). Manticore still runs on its real system clock; only the application's clock advances during the tests.

| Fixture | Initial result | After advancing the injected clock one second |
| --- | --- | --- |
| ID 1, cutoff minus one second | Dropped by event-time admission | Still absent |
| ID 2, exactly the cutoff | Accepted | Deleted; replay dropped |
| ID 3, cutoff plus one second | Accepted | Retained at the new cutoff |
| ID 4, one hour old, replayed | One searchable document | Retained |
| ID 5, now plus 301 seconds | Quarantined | Never written |
| ID 6, noninteger timestamp | Quarantined | Never written |

At the boundary on **October 20, 2026**, the cutoff is **October 13 at 00:00:00 UTC**. A log from October 12 at 23:59:59 is rejected immediately, while one stamped exactly at the cutoff is accepted. One second later, that second event expires and cleanup removes it. Its next delivery is rejected too. **A deleted log stays deleted, even after replay.**

As a control, the `ingestion_control` table stores ID 1 with its old event time but a fresh ingestion time. A deletion based on `ingested_ts < cutoff` leaves it in place. That is the difference between the two retention policies.

Other tests accept an event exactly 300 seconds in the future but hide it until the query's upper bound reaches its timestamp. Missing, boolean, and fractional timestamps are quarantined. Retrying ID 4 moves `ingested_ts` forward without changing `event_ts`.

The fixture’s accepted IDs are initially `[2, 3, 4]`. After advancing one second, the application executes:

```sql
DELETE FROM logs WHERE event_ts < 1791849601;
```

The running engine returns `[3, 4]` after cleanup. Replaying ID 2 is rejected at admission, so the result stays `[3, 4]`; another cleanup also leaves it unchanged. The point is not just that the old event was deleted: replaying it did not bring it back.

A duplicate `INSERT` for ID 4 returns HTTP `500` with `{"error":"duplicate id '4'"}`. That's a duplicate-ID error, not a retention rejection. The runner checks the error body and stores it alongside successful SQL responses in `evidence/sql.jsonl`.

## Queries That Respect the Window

Before advancing the clock, this query finds API timeouts in the seven-day event-time window, newest first:

```sql
SELECT id, message, event_ts, ingested_ts
FROM logs
WHERE MATCH('@message timeout')
  AND service = 'api'
  AND event_ts >= 1791849600
  AND event_ts <= 1792454400
ORDER BY event_ts DESC, id ASC
LIMIT 20;
```

The IDs come back as `[4, 3, 2]`. One second later, use `1791849601` and `1792454401` as the lower and upper bounds; ID 2 drops out. In a service, derive both values from one trusted clock reading and serialize query values safely. With a durable cutoff, use `max(persisted_cutoff, now - 604800)` for reads, admission, and cleanup. This lab uses fixed values, not arbitrary log text or user input.

Filtering by event time hides expired rows from search even when cleanup runs late, but it doesn't free disk space. Confirm a [conditional DELETE](https://manual.manticoresearch.com/Data_creation_and_modification/Deleting_documents) by reading the rows back.

## Deletion and Disk Space Are Different Checks

The lab flushes two disk chunks, using a replay of ID 4 to create the second. It records the file names and logical lengths inside the `logs` table directory before deletion, after deletion, and after:

```sql
OPTIMIZE TABLE logs OPTION sync=1, cutoff=1;
```

The table files totalled 266,745 bytes before deletion and 133,773 bytes after compaction. Deleting rows alone did not reduce those file sizes. The post-compaction readback still returned `[3, 4]`.

These are file lengths for a tiny synthetic table, not allocated disk blocks, whole-volume usage, or a capacity estimate. Binlogs, backups, container logs, and quarantine take space elsewhere. The archive keeps the individual file captures so you can check the calculation.

The [compaction manual](https://manual.manticoresearch.com/Securing_and_compacting_a_table/Compacting_a_table) explains the chunk cutoff and synchronous option. Compaction can use significant I/O and temporary disk space, so schedule it around workload and disk pressure rather than running it after every event.

## Log Retention Best Practices for Production

The downloadable lab runs operations one at a time with an injected application clock. It demonstrates the retention rule, but a live ingestion service also needs to handle these cases:

- **Coordinate ingestion and cleanup.** With multiple workers, one can validate an event, pause, and write it after another advances cleanup. Coordinate admission, the actual write, and cleanup as one protected sequence. A time filter can hide a wrongly reinserted row but cannot prevent wasted storage. Direct SQL writes that bypass admission pose the same risk.

- **Make the cutoff durable and monotonic.** Persist a nondecreasing cutoff before deleting against it, and use it for reads and admission too. A backward clock jump must not reopen expired history. Detect large forward jumps before advancing the cutoff, or you could permanently delete valid data. Check source timestamps as well: plausible times are not necessarily truthful.

- **Keep quarantine and retry state durable.** The example buffers quarantined timestamps in memory and writes JSON only at the end. For production, use a durable queue with access controls, an expiry policy, and a review or replay process. Define acknowledgement and crash-recovery behavior for the ingestion service.

- **Watch cleanup and capacity.** Schedule cleanup, alert on failures and lag, and monitor accepted, dropped, and quarantined events. A seven-day window is not a disk quota: arrival rate, event size, merge overhead, and cleanup delays still determine disk use. This example does not test throughput or replication.

Before deploying, delete an expired event, replay it through the real ingestion service, and confirm it stays absent. Repeat after a worker restart. The bundled test demonstrates the sequential replay case; production restart safety also needs the durable cutoff and coordinated writes described above.

## Takeaway

To keep expired logs from returning:

1. Use the same event-time cutoff for ingestion, search, and cleanup.
2. Check it before every write, including retries.
3. Coordinate writes with cleanup so an expired row cannot be inserted again.

Manticore handles storage and search; the application enforces the retention policy.
