Manticore Search 3.0.0 is now available

Published: May 06, 2019

We are pleased to announce the release of Manticore Search 3.0.0, our long-awaited first major release. This is the first Manticore version which breaks index compatibility with old versions.

Downloads are available here and from the Manticore Github page. Docker image is available at Docker Hub.

Key changes

One of the big issues of Manticore Search 2 was the limitation of 4GB of strings/json attributes per index. To overcome this limit, the data had to be sharded across multiple indexes. Version 3 redesigns the index storage which lifts this limitation and makes possible to reduce number of shards (or simply use single index) and also fully optimize a RealTime index down to one chunk in cases affected by the limitation.

The index compatibility is broken as the new index storage has a different structure. Existing indexes created with Manticore 2 cannot be loaded by Manticore 3, but they can be easily upgraded with the conversion tool.

There are also no limitations regarding updating attributes dynamically (using UPDATE). Previously strings attributes could be updated only with REPLACE command and for JSON attributes only scalar key values could be updated.

Plain indexes kill-lists are applied now at index load/rotation rather per query. An index containing kill-lists will need to define a list of indexes for which the kill-lists will apply upon loading/rotation. This removes the need to define a specific order for searching in indexes as before. If you rotate multiple indexes at once Manticore Search will do smart rotation for your - it will first rotate indexes not targeted for kill-lists and at last the indexes marked on kill-list targets.

Document IDs are now positive signed 64-bit integers, previously they were unsigned 64-bit integers.

Other improvements

Added CONCAT() function - allows to concatenate two or more strings. If you want to concatenate non-string arguments you can additionally use TO_STRING().

Replication has received several improvements:

node_address directive - allows to specify a network address of the node
JOIN CLUSTER allows an easier syntax by providing only the address and port of one of the cluster’s nodes using ‘AT’ clause
ALTER CLUSTER cluster UPDATE nodes - for updating the list of nodes in a cluster
list of nodes is printed in SHOW STATUS

Performance notes

AND queries are generally faster in v3. The speedup depends on the keywords frequencies. Tests including top 1000 from a collection showed small gains in case of multiple top common keywords (~10%) to over 60% when some of the keywords were in the end of the top 1000. Rarer keywords should provide even bigger gains.

Wildcard searches are several percents faster in v3. OR searches in some cases may be several percents slower (at most 9% in performed tests), this issue will be addressed in future releases. RT indexes with lot of disk chunks are now significantly faster.

Thanks to early testers v3 has already been running for thousands of hours in production. Below are several improvements observed:

v3 is better in terms of RAM fragmentation:

Some of the users that are already testing v3 in production report better average performance compared to 2.8:

What to know before upgrading?

You will need to reindex all the indexes or use the converter tool.

If you are using kill-lists there are changes required in the index declarations. RT indexes should have no data in RAM chunk (run FLUSH RAMCHUNK) during conversion.

For distributed setups migration can start from data nodes (agents) to master node(s) as a master node running v2 can talk with agents running on v3.

Nothing changed in terms of search query syntax! All connecting protocols do not require any updates or changes to work with Manticore 3. As soon as indexes are loaded, existing clients can simply connect to the new version.

A full reference of changes and steps for migrating can be found in the migration guide.

While Manticore 3 was tested extensively in production by several early adopters there may still be some issues. We’ll appreciate if you report any issue on the github tracker.

For full changelog of 3.0.0 please consult the Release note.

Future notes

Concerning HTTP protocol, expect the following changes in next releases: /search endpoint will be deprecated and /sql response format will be the same as /json endpoint.