blog-post

Manticore Search authentication rollout checklist for production

In production, “turn it on and done” almost never works. Technically, it involves four steps: enable auth, restart Manticore, create an admin user, and update clients. In practice, it almost always breaks something non-obvious: a nightly worker, an admin's manual script, or a BI dashboard nobody has touched for six months.

Treat this not as a setting change, but as a small release: inventory clients first, then test, then cut over. That way, you can identify expected failures in advance instead of fixing everything during an emergency.

This checklist is for users who plan to enable authentication and want to roll it out as safely as possible in an existing system. Remember that authentication is disabled until you configure auth; after cutover, clients that still omit credentials will fail.

Phase 1: inventory before changing anything

Start by listing every client that connects to Manticore Search, including each application and independently deployed application component. Do this before editing the config.

Common things to check:

  • application search frontends
  • ingest workers
  • cron jobs
  • dashboards and BI tools
  • support or admin tools
  • schema change/update scripts
  • backup and maintenance scripts
  • local scripts run manually

For each client, record something like this:

ClientProtocolTables or clustersNeeded actionsNotes
product search appHTTPproductsreadwill use Bearer token
catalog ingest workerSQLproductswriteuses password auth
schema migration jobSQLproductsschemarun only during deploy
access administratorSQL*adminmanages users and permissions

Then confirm the deployment basics:

  • Are you running RT mode or plain mode?
  • In plain mode, where should the auth file live?
  • Is pid_file configured in the config? Bootstrap needs it.
  • Will SQL clients use SSL, and will HTTP clients use HTTPS when credentials cross a network?
  • Where will credentials, including temporary Bearer tokens, be stored safely?
  • Who is allowed to see the first admin password?

Do not skip the last two. CREATE USER returns a raw Bearer token; TOKEN generates a new token for the specified user. SHOW TOKEN later shows the stored token hash, not the raw token. If the raw token is lost, rotate it with TOKEN and update the token in your application.

Phase 2: set up least-privilege users

Create users based on specific workloads, rather than granting permissions based on what someone might need later.

Use a small matrix like this:

WorkloadUserPermissions
search frontendapp_readGRANT read ON 'products' TO 'app_read'
ingest workerapp_ingestGRANT write ON 'products' TO 'app_ingest'
schema migration jobschema_jobGRANT schema ON 'products' TO 'schema_job'
auth operatorsecurity_adminGRANT admin ON * TO 'security_admin'

Keep in mind that admin is a narrow permission. It only allows managing authentication and authorization state; it does not imply read, write, schema, or replication.

That matters because a person who manages credentials does not automatically need to read business data. A service that searches products does not need to write documents. A migration job does not need to manage users.

Plan negative permission tests as well. For every user you create, choose at least one thing it should be able to do and one thing it should not be able to do.

Examples:

  • app_read can search products.
  • app_read cannot insert into products.
  • app_ingest can write to products.
  • app_ingest cannot manage auth.
  • schema_job can change schema for products.
  • schema_job cannot read tables unless you grant read.

Phase 3: test in staging

Use a staging environment to rehearse the sequence of steps you plan to follow in production.

In RT mode:

searchd {
    data_dir = /var/lib/manticore
    auth = 1
    auth_log_level = info
    ...
}

To turn auth off explicitly in RT mode:

searchd {
    data_dir = /var/lib/manticore
    auth = 0
    ...
}

In plain mode:

searchd {
    auth = /var/lib/manticore/auth.json
    auth_log_level = info
    ...
}

Keep the auth file private. Before the first bootstrap, Manticore may create an empty auth file. After bootstrap, that file stores auth data and credential hashes.

Start searchd, then bootstrap the first administrator:

searchd --config /etc/manticoresearch/manticore.conf --auth

For scripted setup:

printf 'admin\nStrongPass#2026\nStrongPass#2026\n' | \
  searchd --config /etc/manticoresearch/manticore.conf --auth-non-interactive

⚠️ Warning: in this case, the password is passed in raw form on the command line and may end up in shell history and process listings. Use this only in a controlled environment.

This command creates the first administrator and grants all actions to that user. It does not return a bearer token. If that admin needs HTTP Bearer access, connect as the admin and run TOKEN, or use the HTTP POST /token endpoint.

Next, create staging users based on the notes from the previous phases. For example:

CREATE USER 'app_read' IDENTIFIED BY 'ReadPass#2026';
GRANT read ON 'products' TO 'app_read';

CREATE USER 'app_ingest' IDENTIFIED BY 'IngestPass#2026';
GRANT write ON 'products' TO 'app_ingest';

CREATE USER 'schema_job' IDENTIFIED BY 'SchemaPass#2026';
GRANT schema ON 'products' TO 'schema_job';

CREATE USER 'security_admin' IDENTIFIED BY 'AdminPass#2026';
GRANT admin ON * TO 'security_admin';

Store the returned Bearer tokens in secure storage. Remember: raw tokens must not be left in logs, shell history, or unprotected files. If you need to rotate one, use:

TOKEN 'app_read';

SHOW TOKEN is not a way to recover the raw token:

SHOW TOKEN FOR 'app_read';

Review users and permissions:

SHOW USERS;
SHOW PERMISSIONS;
SHOW PERMISSIONS FOR 'app_read';

Run one allow and one deny test for every user. For the read-only user:

curl -H "Authorization: Bearer <app_read_token>" \
  http://127.0.0.1:9308/sql?mode=raw \
  -d "SELECT * FROM products LIMIT 10"

Then verify that an unauthorized operation is denied:

curl -H "Authorization: Bearer <app_read_token>" \
  http://127.0.0.1:9308/sql?mode=raw \
  -d "INSERT INTO products(id,title) VALUES(1,'test')"

For this HTTP request, expect a 403 Forbidden response. Over SQL/MySQL, a permission denial returns ERROR 1045 with a permission-denied message.

SQL clients should connect with a Manticore user name and password. The SQL/MySQL protocol in Manticore supports mysql_native_password.

MYSQL_PWD=ReadPass#2026 \
  mysql -h127.0.0.1 -P9306 -uapp_read \
  -e "SELECT * FROM products LIMIT 10"

HTTP clients can use Basic authentication or Bearer tokens:

curl -u app_read:ReadPass#2026 \
  http://127.0.0.1:9308/sql?mode=raw \
  -d "SELECT * FROM products LIMIT 10"

HTTP authentication schemes (Basic, Bearer) are case-insensitive; user names are case-sensitive.

If you edit the auth file outside the daemon during maintenance, reload it:

RELOAD AUTH;

Phase 4: production rollout checklist

Use this as the cutover checklist. You can even print it and check items off.

  • Confirm you have a current config backup.
  • Confirm existing network protections stay in place.
  • Confirm pid_file is set in the config.
  • Confirm where the auth file will be created or loaded from.
  • Confirm password and token storage is ready.
  • Enable auth during a planned rollout window.
  • Expect unauthenticated clients to fail after auth is enabled.
  • Bootstrap the first administrator.
  • Create production users and permissions.
  • Store tokens in a protected secrets store immediately after they are issued. Do not store raw tokens in files, shell history, or logs.
  • Update SQL connection code to send user names and passwords.
  • Update HTTP connection code to use Basic authentication or Bearer tokens.
  • Run the allow and deny tests from staging.
  • Check the auth log.
  • Run application stress tests that cover search, ingest, dashboards, and maintenance scripts.
  • Rotate any temporary rollout credentials.
  • Keep the first admin credential out of normal application use.

Authentication events are written to a separate auth log when authentication is enabled. If the daemon log is /var/log/manticore/searchd.log, the auth log is /var/log/manticore/searchd.log.auth.

The auth_log_level values are:

  • disabled
  • error
  • warning
  • info
  • all
  • trace

The default is info. Start there unless you have a reason to reduce or increase the logging detail. Use trace only for diagnostics; it also includes all successful internal auth traffic.

Useful cleanup and maintenance commands:

SET PASSWORD 'NewReadPass#2026' FOR 'app_read';
REVOKE read ON 'products' FROM 'app_read';
DROP USER 'app_read';

SET PASSWORD changes the password used by SQL/MySQL and HTTP Basic auth. It does not revoke existing bearer tokens. To rotate Bearer access, create a new token with TOKEN or POST /token and update the client.

Phase 5: rollback and troubleshooting

If something goes wrong, the rollback should be straightforward:

  • restore the previous config
  • restart Manticore
  • restore the old access restrictions that existed before the auth rollout
  • be ready to roll back application configurations if clients were updated to send credentials

Do not delete the rollout notes. They are usually the fastest way to see which client was updated, which token was stored where, and which permissions were created.

SymptomLikely causeCheck
SQL access deniedWrong user, wrong password, or client auth mismatchCheck the configured user and confirm the client can use mysql_native_password.
HTTP 401Missing or invalid credentialsCheck the Authorization header and whether the client uses Basic auth or Bearer token auth.
HTTP 403User authenticated but lacks permissionCheck SHOW PERMISSIONS FOR '<user>'.
Bearer token does not workToken was lost, copied incorrectly, or already revokedRun TOKEN '<user>', store the returned token, and update the client.
User has fewer permissions than expectedMissing action grantCheck whether the operation needs read, write, schema, replication, or admin.
User has more permissions than expectedBroad target or missing explicit denyCheck wildcard grants, exact target grants, and any WITH ALLOW 0 rules.

Permission rules are determined by action type. When rules conflict, an explicit deny always takes precedence over an allow, even if the allow is more specific. If no matching allow exists, access is denied.

Authentication caveats in replication

If you use replication, plan a separate authentication rollout. The single-node checklist above is still useful, but clusters add failure modes that are easy to miss.

Distributed remote-agent queries authenticate as the current session user. The remote daemon needs matching auth material for that user and the required permission on the remote target table.

Replication operations use the replication action. Grant it only to the user that should run and own replication operations.

When auth is enabled, cluster joins can replace local auth data with auth data from the donor cluster. Make sure the replication user and auth data are consistent across nodes before joining. Treat searchd.log.auth as sensitive during cluster work because auth backups can include salts and credential hashes.

Keep the full cluster migration procedure in a separate document. Here, the goal is only to highlight authentication-related cluster risks.

Final check

Before calling the rollout done, confirm that:

  • Every isolated part of the system that uses Manticore Search has its own user.
  • Every user has only the actions it needs.
  • Bearer tokens are stored in a protected secrets store; raw tokens are not saved outside a controlled environment.
  • The operations team knows that SHOW TOKEN does not return the raw token, but shows its hash; use TOKEN or the HTTP endpoint to get a new token.
  • SQL and HTTP clients have been updated.
  • Expected denials were tested.
  • Auth logs are visible.
  • The rollback procedure is clear and documented.

We wish you a smooth authentication and authorization rollout!

Go from zero to Manticore in seconds

Install Manticore Search in one command on Linux or macOS:

curl https://manticoresearch.com | sh

For advanced installation options, see the full installation guide and the manual .