# Manticore Search under systemd: beyond fork, PID files, and guesswork

Run Manticore Search under systemd with accurate status reporting, cleaner reloads, journal-based logging, and safer shutdowns for RT workloads.

If you run Manticore Search on Linux, systemd should be the default way to manage it.

That sentence sounds obvious now, but for a long time it was only partly true. Manticore could run under systemd, yes, though the relationship was a little awkward. The daemon model came from an earlier Unix world; systemd came later and wanted different things from a service. So the setup worked, but never in a very satisfying way.

What changed is simple enough: Manticore now supports native systemd notifications.

Why care? Because several mildly annoying operational problems get better at once:

- `systemctl status` tells a more truthful story
- startup and reload are easier to follow
- logs fit naturally into `journalctl`
- shutdown is safer when real-time tables are flushing data
- PID files stop doing so much heavy lifting

That last point matters less than people think, until the day it matters a lot.

## Start with shutdown, because that's where things usually get real

The nicest part of this change is not the nicest-looking part. It is shutdown behavior.

If Manticore is flushing data from real-time tables, shutdown may take a while. Older setups often fell back to `searchd --stopwait` and a certain amount of hope. Sometimes that was enough. Sometimes it really was not.

The failure mode is boring and nasty: systemd decides the service is stuck, waits long enough, then sends `SIGKILL`. If Manticore is in the middle of a flush, that is about the worst possible moment to force the process down.

The newer behavior is cooperative. While Manticore is still flushing data, it tells systemd that progress is ongoing and that more time is needed. Concretely, it sends timeout-extension notifications every 15 seconds, each asking for another 30 seconds.

I would put this above almost every other benefit in the article. Better status output is nice. Cleaner reload is nice. Not getting your shutdown path mangled during an RT flush is nicer.

A slow stop is not automatically suspicious anymore. Sometimes it just means the server is finishing the part that actually matters.

## What the old setup looked like

Historically, Manticore used the normal Unix daemon pattern: detach from the terminal, double-fork into the background, write a PID file, continue on. That was not some strange design choice. If portability matters, you end up there pretty quickly.

The friction came from mixing that model with systemd.

Support existed, but it was limited. The unit had to trust that the daemon forked correctly and that the PID file existed and still pointed to the right process. When everything behaved, fine. When it drifted even a little, supervision got fuzzy.

The usual weak spots were predictable:

- process tracking depended on an external file
- status reporting was indirect
- startup progress was mostly invisible to systemd
- reloads and shutdowns were harder to monitor cleanly

None of this sounds dramatic when written out calmly. In practice it leads to exactly the kind of question operators hate: is this service actually healthy, or just currently alive.

One specific example is worth keeping in mind. If Manticore's internal watchdog restarted `searchd` after a crash, systemd could keep supervising the original process relationship instead of the newly resurrected daemon. That is where warnings like `Supervising process ... which is not our child` came from.

A smaller but common failure mode was config drift: change the `pid_file` path in Manticore, forget to update the unit, and `systemd` ends up tracking the wrong place. The daemon might be up while `systemctl status` tells a much less helpful story.

![systemd_searchd_pid](./systemd_integration/systemd_searchd_pid.png)

That screenshot captures the old setup fairly well. systemd could see a process tree, but not always the service lifecycle you cared about.

## The notify-based unit

The newer setup uses `Type=notify`, which lets Manticore report its own state directly instead of forcing systemd to infer too much from a PID file and guesswork.

The systemd unit now looks like this:

```systemd
[Unit]
Description=Manticore Search Engine
...

[Service]
Type=notify
...

ExecStart=/usr/bin/searchd --config /etc/manticoresearch/manticore.conf --nodetach $_ADDITIONAL_SEARCHD_PARAMS
...

Restart=on-failure
...
```

A few details matter here.

`Type=notify` means systemd expects status updates from Manticore itself.

`--nodetach` keeps `searchd` in the foreground. Under systemd, that is the right default. Anything else is extra ceremony. The flag itself is not new; what changed is that it is now part of the packaged systemd setup rather than a debugging-only kind of option.

`PIDFile` is no longer the main supervision mechanism, but it can still be useful and added manually if you have older tooling that expects it.

With this setup, Manticore can report states such as:

- starting
- loading tables
- reloading
- ready

That does not sound glamorous. Still, compared to the old "there is a process, probably fine" model, it is a real improvement.

## A small thing I like: reload becomes less awkward

Reloading Manticore has traditionally meant sending `SIGHUP` to `searchd`. That is still what happens under systemd too. The packaged unit uses:

```systemd
ExecReload=/bin/kill -HUP $MAINPID
```

So `systemctl reload manticore` does not restart the daemon. It sends `SIGHUP` to the running `searchd`, which initiates table rotation: Manticore reopens [plain tables](https://manual.manticoresearch.com/Read_this_first#Real-time-table-vs-plain-table) and switches to freshly built table files without doing a full daemon restart.

That distinction matters. `reload` is the low-risk way to trigger rotation without taking the service down first. Depending on the `seamless_rotate` setting, new queries may be briefly stalled and clients may see temporary errors during the rotation window. `restart` is a different operation entirely: it stops the service and then does a fresh start.

The nice part now is visibility. When rotation starts, Manticore reports `RELOADING=1` to systemd; when it finishes, it reports `READY=1` again. So `systemctl status manticore` can show that the daemon is actively rotating instead of just sitting there in a generic running state.

## `--nodetach` also fixes the logging story

This option is not new. The environment around it changed.

When Manticore stays attached to systemd instead of disappearing into the background, logs go to the journal and the main process is the one systemd actually knows about. There is less indirection. That usually pays off later, not immediately. Before `--nodetach`, startup messages could begin in `journalctl` and then the rest of the story would move into Manticore's own log files after detach.

So for many deployments, the normal tools are enough:

```bash
journalctl -u manticore
journalctl -u manticore -f
journalctl -u manticore --since "1 hour ago"
```

I like boring logging setups. One place to look first is underrated.

If you are happy using the systemd journal as the main log destination, you may not need the older logging setup in `manticore.conf`. Often that is one less thing to think about.

## About the internal watchdog

If Manticore runs under systemd, the simple setup is the one I would recommend: let systemd supervise the service, and do not enable Manticore's internal watchdog unless you actually need it.

Could you run both? Yes.
Would I choose that for a normal deployment? No.

If you explicitly enable the internal watchdog, the supervision model becomes less direct and you may see warnings like:

`Supervising process ... which is not our child`

That configuration is supported. It is also, for most people, just extra moving parts.

## A few loose notes

PID files matter much less to systemd itself with notification-based supervision. You may be able to remove `pid_file` from the configuration, but only if nothing else in your environment still depends on it. Old scripts tend to survive longer than anyone expects.

If you use executable configuration files, including shebang-based configs such as `#!/usr/bin/env python`, this setup tends to behave more naturally than some older service-management approaches did.

## Commands you will actually use

```bash
sudo systemctl start manticore
sudo systemctl stop manticore
sudo systemctl restart manticore
sudo systemctl reload manticore
sudo systemctl status manticore
sudo systemctl enable manticore
```

And for logs:

```bash
sudo journalctl -u manticore
sudo journalctl -u manticore -f
sudo journalctl -u manticore --since "1 hour ago"
```

That is basically it.

The practical result is not flashy: Manticore behaves more like a modern Linux service and less like a daemon from an older era that everyone learned to work around. But those are often the best infrastructure changes. They remove friction you had almost stopped noticing.