Major new features

• Full support of Manticore Columnar Library . Previously Manticore Columnar Library was supported only for plain indexes. Now it's supported:
• in real-time indexes for INSERT, REPLACE, DELETE, OPTIMIZE
• in replication
• in ALTER
• in indextool --check
• Automatic indexes compaction (#478 ). Finally you don't have to call OPTIMIZE manually or via a crontask or other kind of automation. Manticore now does it on your own. You can set default compaction threshold via optimize_cutoff .
• Chunk snapshots and locks system revamp. These changes may be invisible from outside at first glance, but they improve the behaviour of many things happening in real-time indexes significantly. In a nutshell, previously most Manticore data manipulation operations relied on locks heavily, now we use disk chunk snapshots instead. In particular:
  • read operations (e.g. SELECTs, replication) are performed with snapshots
  • operations that just change internal index structure without modifying schema/documents (e.g. merging RAM segments, saving disk chunks, merging disk chunks) are performed with read-only snapshots and replace the existing chunks in the end
  • UPDATEs and DELETEs are performed against existing chunks, but for the case of merging that may be happening the writes are collected and are then applied against the new chunks
  • UPDATEs acquire an exclusive lock sequentially for every chunk. Merges acquire a shared lock when entering the stage of collecting attributes from the chunk. So at the same time only one (merge or update) operation has access to attributes of the chunk.
  • when merging gets to the phase it needs attributes it sets a special flag. When UPDATE finishes it checks the flag and if it's set, the whole update is stored in a special collection. Finally when the merge finishes, it applies the updates set to the newborn disk chunk
  • ALTER runs via an exclusive lock
  • replication runs as a usual read operation, but in addition saves the attributes before SST and forbids updates during the SST
• ALTER can add/remove a full-text field. Previously it could only add/remove an attribute.
• 🔬 Experimental: pseudo sharding for full-scan queries - allows to parallelize any non-full-text search query. Instead of preparing shards manually you can now just enable new option searchd.pseudo_sharding and expect up to CPU cores lower response time for non-full-text search queries. Note it can easily occupy all existing CPU cores, so if you care not only about latency, but throughput too - use it with caution.

Minor changes

• Linux Mint and Ubuntu Hirsute Hippo are supported via APT repository
• faster update by id via HTTP in big indexes in some cases (depends on the ids distribution)
• custom startup flags for systemd . Now you don't need to start searchd manually in case you need to run Manticore with some specific startup flag
• new function LEVENSHTEIN() which calculates Levenshtein distance
• added new searchd startup flags --replay-flags=ignore-trx-errors and --replay-flags=ignore-all-errors so one can still start searchd if the binlog is corrupted
• #621 - expose errors from RE2
• more accurate COUNT(DISTINCT) for distributed indexes consisting of local plain indexes
• FACET DISTINCT to remove duplicates when you do faceted search
• exact form modified doesn't require morphology now and works for indexes with infix/prefix search enabled

Breaking changes

• the new version can read older indexes, but the older versions can't read Manticore 4's indexes
• removed implicit sorting by id. Sort explicitly if required
• charset_table's default value changes from 0..9, A..Z->a..z, _, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F, U+401->U+451, U+451 to non_cjk
• OPTIMIZE happens automatically. If you don't need it make sure to set auto_optimize=0 in section searchd in the configuration file
• #616 ondisk_attrs_default were deprecated, now they are removed
• for contributors: we now use Clang compiler for Linux builds as according to our tests it can build a faster Manticore Search and Manticore Columnar Library
• if max_matches is not specified in a search query it gets updated implicitly with the lowest needed value for the sake of performance of the new columnar storage. It can affect metric total in SHOW META , but not total_found which is the actual number of found documents.

Migration from Manticore 3

• make sure you a stop Manticore 3 cleanly:
• no binlog files should be in /var/lib/manticore/binlog/ (only binlog.meta should be in the directory)
• otherwise the indexes Manticore 4 can't reply binlogs for won't be run
• the new version can read older indexes, but the older versions can't read Manticore 4's indexes, so make sure you make a backup if you want to be able to rollback the new version easily
• if you run a replication cluster make sure you:
• stop all your nodes first cleanly
• and then start the node which was stopped last with --new-cluster (run tool manticore_new_cluster in Linux).
• read about restarting a cluster for more details

Bugfixes

• Lots of replication issues have been fixed:
• 696f8649 - fixed crash during SST on joiner with active index; added sha1 verify at joiner node at writing file chunks to speed up index loading; added rotation of changed index files at joiner node on index load; added removal of index files at joiner node when active index gets replaced by a new index from donor node; added replication log points at donor node for sending files and chunks
• b296c55a - crash on JOIN CLUSTER in case the address is incorrect
• 418bf880 - while initial replication of a large index the joining node could fail with ERROR 1064 (42000): invalid GTID, (null), the donor could become unresponsive while another node was joining
• 6fd350d2 - hash could be calculated wrong for a big index which could result in replication failure
• #615 - replication failed on cluster restart
• #574 - indextool --help doesn't display parameter --rotate
• #578 - searchd high CPU usage while idle after ca. a day
• #587 - flush .meta immediately
• #617 - manticore.json gets emptied
• #618 - searchd --stopwait fails under root. It also fixes systemctl behaviour (previously it was showing failure for ExecStop and didn't wait long enough for searchd to stop properly)
• #619 - INSERT/REPLACE/DELETE vs SHOW STATUS. command_insert, command_replace and others were showing wrong metrics
• #620 - charset_table for a plain index had a wrong default value
• 8f753688 - new disk chunks don't get mlocked
• #607 - Manticore cluster node crashes when unable to resolve a node by name
• #623 - replication of updated index can lead to undefined state
• ca03d228 - indexer could hang on indexing a plain index source with a json attribute
• 53c75305 - fixed not equal expression filter at PQ index
• ccf94e02 - fixed select windows at list queries above 1000 matches. SELECT * FROM pq ORDER BY id desc LIMIT 1000 , 100 OPTION max_matches=1100 was not working previously
• a0483fe9 - HTTPS request to Manticore could cause warning like "max packet size(8388608) exceeded"