# How to search for long hashes and IDs with dict='keywords_32k'

A practical guide to searching long hashes, event IDs, message IDs, and email addresses in Manticore Search: limits, exact matching, wildcard search, tokenization, migration, and limitations.

Full-text search usually works with ordinary words: product names, titles, comments, and descriptions. Such tokens are rarely longer than a few dozen characters.

Logs and technical data are different. A SHA-256 hash is 64 characters long, while message IDs, correlation keys, event identifiers, and some email addresses can be even longer. The value is often meaningful only as a whole: if its tail is lost, one ID can easily be confused with another.

Manticore Search provides `dict='keywords_32k'` for these cases.

> `keywords_32k` is available starting with Manticore Search 27.1.1. Version 27.1.5 or newer is recommended when converting an existing table from `keywords`.

## The problem with the regular dictionary

By default, Manticore uses `dict='keywords'`. Its maximum token length is **42 bytes after normalization**.

The limit is measured in bytes, not characters. One ASCII character takes one byte, but a UTF-8 character can take several bytes.

If a token exceeds 42 bytes, Manticore truncates it:

- when indexing a document;
- when processing a search query.

As a result, querying the complete long value does not necessarily return zero results. Because the query is truncated too, the document may be found—but only by the first 42 bytes.

This creates a more serious problem: two different IDs with the same first 42 bytes become indistinguishable to full-text search. It is also impossible to find a value by a fragment located after the 42nd byte.

## What `keywords_32k` changes

`dict='keywords_32k'` increases the maximum normalized token length to **32768 bytes**, or 32 KB.

| Behavior | `dict='keywords'` | `dict='keywords_32k'` |
|---|---:|---:|
| Maximum token length | 42 bytes | 32768 bytes |
| Token exceeding the limit | Truncated | Skipped with a warning |
| Prefix and infix search | Supported | Supported |
| Morphology for tokens longer than 42 bytes | Token is already truncated | Not applied |
| RT tables | Supported | Supported |
| Plain tables | Supported | Supported |

The setting applies to the entire table:

```sql
CREATE TABLE events (
  message text,
  event_id text
)
dict='keywords_32k';
```

Regular short words in the same table continue to use the configured morphology. Tokens longer than 42 bytes are stored in their original normalized form, without stemming or lemmatization.

For machine identifiers, this is usually exactly what you need: a hash or event ID has no useful word stem.

## When to use `keywords_32k`

Use it when both conditions are true:

1. The value can exceed 42 bytes after tokenization.
2. It must be searchable through `MATCH()`, by prefix, or by substring.

Typical examples include:

- SHA-256 and other long hashes;
- event IDs and message IDs;
- request IDs, trace IDs, and other technical identifiers;
- long record keys;
- email addresses with a long local or domain part;
- technical values from logs;
- long identifiers containing separators.

However, `keywords_32k` is not required for every ID lookup.

### If you only need exact equality

If the application always receives the complete ID and only needs to check exact equality, a string attribute is sufficient:

```sql
CREATE TABLE events (
  message text,
  event_id string
);
```

Then use a regular filter:

```sql
SELECT id, message
FROM events
WHERE event_id =
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08';
```

### If you need both exact matching and full-text search

Use `string attribute indexed`:

```sql
CREATE TABLE events (
  message text,
  event_id string attribute indexed
)
dict='keywords_32k';
```

In this case, Manticore:

- stores the original value as a string attribute;
- lets you filter it with `WHERE`;
- also indexes it for `MATCH()` and wildcard search.

This is usually the most convenient schema for technical identifiers.

## Searching by the complete token

Create a table and add a 64-character SHA-256 hash:

```sql
DROP TABLE IF EXISTS events;

CREATE TABLE events (
  message text,
  event_id string attribute indexed
)
dict='keywords_32k';

INSERT INTO events (id, message, event_id) VALUES
(
  1,
  'delivery accepted',
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08'
);
```

Use the complete normalized token in a full-text search:

```sql
SELECT id, message
FROM events
WHERE MATCH(
  '@event_id 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08'
);
```

The `@event_id` operator restricts the search to the required field. Without it, Manticore searches for the value in all full-text fields of the table.

Quotes are not needed around a single simple alphanumeric token here. Quotes denote a phrase search; they do not turn `MATCH()` into a byte-for-byte comparison of the original string.

## A full-text match is not the same as exact equality

`MATCH()` operates on the result of tokenization and normalization. It can be affected by:

- `charset_table`;
- conversion to lowercase;
- `blend_chars`;
- `ignore_chars`;
- word forms and other text-processing settings.

For a strict comparison of the stored value, use a string attribute:

```sql
SET collation_connection='binary';

SELECT id, message
FROM events
WHERE event_id =
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08';
```

`binary` enables byte-for-byte string comparisons in the current SQL session. This setting does not affect full-text search behavior.

The practical rule is simple:

- `WHERE event_id = ...` — a strict comparison of the stored string;
- `MATCH('@event_id ...')` — a normalized-token search;
- `MATCH('@event_id prefix*')` — a prefix search;
- `MATCH('@event_id *fragment*')` — a substring search.

## How to inspect tokenization

Before loading a large volume of data, check that Manticore actually sees the value as a single token:

```sql
CALL KEYWORDS(
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08',
  'events'
);
```

The `normalized` column should contain the complete 64-character hash.

`CALL KEYWORDS` is especially useful for values containing:

- periods;
- hyphens;
- the `@` symbol;
- colons;
- slashes;
- characters from different writing systems.

This lets you inspect the actual token boundaries before indexing the data.

## Prefix and substring search

To search inside a token, enable `min_infix_len`:

```sql
DROP TABLE IF EXISTS events_infix;

CREATE TABLE events_infix (
  message text,
  event_id string attribute indexed
)
dict='keywords_32k'
min_infix_len='4';

INSERT INTO events_infix (id, message, event_id) VALUES
(
  1,
  'delivery accepted',
  '9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08'
);
```

Prefix search:

```sql
SELECT id, message
FROM events_infix
WHERE MATCH('@event_id 9f86d081*');
```

Substring search:

```sql
SELECT id, message
FROM events_infix
WHERE MATCH('@event_id *b2b0b822*');
```

A positive `min_infix_len` also enables prefix search. This example uses `4` to prevent excessively short and broad patterns.

### Why short patterns are dangerous

With `dict='keywords_32k'`, as with regular `dict='keywords'`, Manticore does not precompute every possible substring. Instead, at query time it expands a wildcard pattern into matching dictionary terms.

For example, the pattern `*ab*` may match a huge number of values. The more expansions it produces, and the more documents that contain each matching term, the more expensive the query becomes.

For production systems:

- do not allow users to search for excessively short fragments;
- choose `min_infix_len` based on real data;
- use `expansion_limit` to limit the number of expansions;
- test performance with a dictionary close to production size;
- restrict the search to a specific field with `@field`.

`index_exact_words='1'` is not required for wildcard search itself. You need it if you want to distinguish exact matches from wildcard matches when ranking, usually together with `expand_keywords`.

## Email addresses and other values with separators

`keywords_32k` changes only the maximum token length. It does not determine where a token begins and ends.

By default, a period, `@`, hyphen, and other characters may split a value into multiple parts. If an email address or message ID must also be indexed as a whole, you can use `blend_chars`:

```sql
DROP TABLE IF EXISTS mail_events;

CREATE TABLE mail_events (
  sender string attribute indexed,
  subject text
)
dict='keywords_32k'
blend_chars='., @, -'
min_infix_len='4';

INSERT INTO mail_events (id, sender, subject) VALUES
(
  1,
  'alessandro.verylonggeneratedlocalpart@example-corporate-domain.test',
  'delivery accepted'
);
```

Blended characters are indexed in two ways:

- as part of the complete token;
- as separators between regular parts of the value.

This makes it possible to search both the complete email address and individual words within it.

Searching for the complete value:

```sql
SELECT id, subject
FROM mail_events
WHERE MATCH(
  '@sender "alessandro.verylonggeneratedlocalpart@example-corporate-domain.test"'
);
```

The quotes matter here because `@` is also used in full-text query syntax. Inside a phrase, the parser can process it as a blended character.

Searching by a fragment:

```sql
SELECT id, subject
FROM mail_events
WHERE MATCH('@sender *generatedlocalpart*');
```

Inspect the tokenization result:

```sql
CALL KEYWORDS(
  '"alessandro.verylonggeneratedlocalpart@example-corporate-domain.test"',
  'mail_events'
);
```

In a real application, values passed to `MATCH()` must be escaped correctly. Simply appending user input to a query string can change the query's meaning because of `@`, `-`, `|`, `!`, `"`, `*`, and other operators.

## How to convert an existing table

For an RT table, you can change the setting with `ALTER TABLE`:

```sql
ALTER TABLE events dict='keywords_32k';
```

However, this affects only documents added or replaced after the setting is changed.

Existing documents are not automatically tokenized again. Their long tokens remain in the old truncated form until the documents are reindexed.

Follow these steps:

1. Update `dict`.
2. Verify the setting with `SHOW CREATE TABLE`.
3. Reindex or reload the existing documents.
4. Check several long values with `CALL KEYWORDS` and `MATCH()`.

For a plain table:

1. Change the configuration to `dict = keywords_32k`.
2. Apply the settings with `ALTER TABLE ... RECONFIGURE` if this fits your update workflow.
3. Rebuild the table completely from its data source.

Until the data is reindexed, the same table may contain both:

- old documents with truncated tokens;
- new documents with complete tokens.

This can produce different results for documents that appear identical.

## Why `dict='crc'` does not solve this problem

`dict='crc'` stores keyword checksums instead of their original text. However, it does not increase the allowed token length.

The exception to the regular 42-byte limit is implemented specifically by `dict='keywords_32k'`.

The `keywords` and `keywords_32k` dictionaries also store term text, which allows Manticore to expand prefix and infix wildcard queries against the dictionary.

If you need to search long machine identifiers, `crc` is not a replacement for `keywords_32k`.

## Current limitations

At the time of publication, `dict='keywords_32k'` has several limitations:

- `CALL SUGGEST` and `CALL QSUGGEST` are not supported;
- it cannot be used in percolate tables;
- tokens longer than 42 bytes are not highlighted in snippets or highlights;
- `indextool --dumpdict` cannot dump this type of dictionary;
- the full-text `REGEX` operator works with `dict='keywords'`, but not with `keywords_32k`.

The last point should not be confused with the `REGEX()` function for filtering string attributes. If the field is declared as `string attribute indexed`, attribute filtering and full-text search remain two separate mechanisms.

## Do not index secrets

Being able to search a long value does not mean that you should store it in a search index.

Do not index the following unless absolutely necessary:

- API keys;
- bearer tokens;
- session cookies;
- private keys;
- passwords and reset tokens;
- other data that grants access to the system.

`keywords_32k` solves the search problem, but does not protect the value from being read by users, backups, query logs, or system administrators.

If a secret must be matched by its exact value, it is safer to calculate a suitable hash in advance and store only the hash.

## Quick checklist

Before enabling `keywords_32k`, check:

1. Is the normalized token actually longer than 42 bytes?
2. Do you need full-text or wildcard search rather than only `WHERE value = ...`?
3. Does `CALL KEYWORDS` see the entire value as one token?
4. Do you need `blend_chars` for periods, hyphens, `@`, and other separators?
5. Is `min_infix_len` large enough?
6. Is the number of wildcard expansions limited?
7. Have the old documents been reindexed?
8. Is the field free of secret data?
9. Does the application avoid relying on highlighting, `SUGGEST`, percolate, or full-text `REGEX`?

## Summary

`dict='keywords_32k'` solves a specific problem: it allows a full-text index to store normalized tokens up to 32768 bytes long instead of the regular 42 bytes.

It is well suited to long hashes, event IDs, message IDs, email addresses, and other machine identifiers. Keep three points in mind:

- `keywords_32k` increases the token length but does not change tokenization rules;
- `MATCH()` on a complete token is not the same as a strict comparison of the original string;
- existing documents must be reindexed after the setting is changed.

If you only need exact equality, use a string attribute. If you need both exact matching and partial-value search, use `string attribute indexed` together with `dict='keywords_32k'`.

## Documentation

- [`dict` and `keywords_32k` limitations](https://manual.manticoresearch.com/Creating_a_table/NLP_and_tokenization/Low-level_tokenization#dict)
- [Token length limit](https://manual.manticoresearch.com/Creating_a_table/NLP_and_tokenization/Data_tokenization#Token-length-limit)
- [Wildcard search settings](https://manual.manticoresearch.com/Creating_a_table/NLP_and_tokenization/Wildcard_searching_settings)
- [`blend_chars`](https://manual.manticoresearch.com/Creating_a_table/NLP_and_tokenization/Low-level_tokenization#blend_chars)
- [`CALL KEYWORDS`](https://manual.manticoresearch.com/Searching/Autocomplete#CALL-KEYWORDS)
- [String attributes and indexed strings](https://manual.manticoresearch.com/Creating_a_table/Local_tables#String)
- [Updating full-text settings and reindexing](https://manual.manticoresearch.com/Updating_table_schema_and_settings)
- [Collations and string comparison](https://manual.manticoresearch.com/Searching/Collations)
- [Manticore Search 27.1.5](https://manticoresearch.com/blog/manticore-search-27-1-5/)