Manticore Search 3 brings for the first time a breakage in index compatibility and it cannot load existing indexes created with 2.x. This means the indexes needs to be ready before upgrading searchd to 3.x.

To have indexes in the new format, there are two options:

1. Fully rebuild the indexes. If your indexes are not very big and are not RealTime, this is the easy route.
2. Use the converter tool that can automatically any convert an index of any type (plain, percolate, RealTime) from Manticore 2 / Sphinx 2 format to the new format

There are several things to be aware before starting the migration:

• document ids are now positive signed big integers, previously being unsigned big integers
• the mechanism of indexes using kill-lists has significantly changed
• before starting to convert a RealTime index, the RAM chunk must be empty. If it has data, it should be written to disk using FLUSH RAMCHUNK. This operation must be made on existing 2.x, before performing the upgrade.
• v3 doesn’t limit string / json attributes to 4GB per index. In some cases in the past indexes had to be shared to many chunks only to allow fitting of the data. In v3 an index like this can be reduced to a single or reduced to a more limited number of chunks (in case it’s desired to retain multi-threaded searching). The 4GB limitation also applied to RT indexes. This meant OPTIMIZE couldn’t reduce the RT chunks to a single one if the 4GB threshold was reached. After converting a RT affected by this, for the new index OPTIMIZE can be applied to achieve one-chunk RT index.

Manticore Search 3 comes with a companion tool that can convert indexes from 2.x format to 3.x format. The tool is provided in a separate package called manticore-converter. This package doesn’t interfere or affects in any way the existing Manticore installation, so it can be installed side-by-side.

With the index_converter tool you can create 3.x versions of the indexes and when all are done you start upgrade the Manticore package.

If you have a single server:

• install manticore-converter package
• use index_converter to create new versions of the indexes in a different folder than the existing data folder ( using –output-dir option)
• stop existing Manticore, upgrade to 3.0, move the new indexes to data folder, start Manticore

To get a minimal downtime, you can copy 2.x indexes, config (you’ll need to edit paths here for indexes, logs and different ports) and binaries to a separate location and start this on a separate port and point your application to it.
After upgrade is made to 3.0 and the new daemon is started, you can point back the application to the normal ports. If all good, stop the 2.x copy and delete the files to free the space.

If you have a spare box (like a testing or staging server), you can do here first the index upgrade and even install Manticore 3 to perform several tests and if everything is ok copy the new index files to the production server.

If you have multiple servers which can be pulled out from production, do it one by one and perform the upgrade on each. For distributed setups, 2.x searchd can work as master with 3.x nodes, so you can do upgrading on the data nodes first and at the end the master node.

The converter tool can migrate one index at a time (using --index), all indexes from a given config file (using --all) or all indexes from a folder (defined by --path). As the converter will create a new version of existing index, the extra space required needs to be taken in account. If space is an issue, indexes should be converted one by one.

The most simple of using the converter (in case you don’t use kill-lists) is to just convert everything:

$ index_converter --config /home/myuser/sphinx.conf --all

By default v3 indexes are written in the same folder and old indexes are copied with ‘.old’ in their file names. New indexes can also be saved in a different folder using –output-dir option.

$ index_converter --config /home/myuser/sphinx.conf --index myrtindex --output-dir /new/location

Please note that hitlists file (.spp) is simply moved to new version (as it didn’t suffer any changes) and it’s not included in the backup of old version. Once testing of version 3 is complete and without issue, the backup of old index versions can be deleted.

In case of delta indexes, the new option killlist_target should be set in configuration before starting the conversion. The converter can also set the targets with –killlist-target:

$ index_converter --config /home/myuser/sphinx.conf --index deltaindex --killlist-target mainindex:kl

The third option is to convert the delta index, load it, use ALTER KILLLIST_TARGET to set targets and (re)load the targets to apply the kill-lists.

In case you have any issue, question or comments feel free to contact us:

• on twitter
• by sending an email to [email protected]
• posting on our Forum
• chatting with us in our Community Slack
• via our bug tracker on GitHub