Manticore team is excited to announce Manticore Search version 4.2.0. What’s new in the release:

Major new features

• Pseudo-sharding support for real-time indexes and full-text queries. In previous release we added limited pseudo sharding support. Starting from this version you can get all benefits of the pseudo sharding and your multi-core processor by just enabling searchd.pseudo_sharding. The coolest thing is that you don’t need to do anything with your indexes or queries for that, just enable it and if you have free CPU it will be used to lower your response time. It supports plain and real-time indexes for full-text, filtering and analytical queries. For example, here is how enabling pseudo sharding can make most queries’ response time in average about 10x lower on Hacker news curated comments dataset multiplied 100 times (116 million docs in a plain index).
  
• Debian Bullseye is now supported.
• PQ transactions are now atomic and isolated. Previously PQ transactions support was limited. It enables much faster REPLACE into PQ, especially when you need to replace a lot of rules at once. Performance details:

Previous release 4.0.2

It takes 48 seconds to insert 1M PQ rules and 406 seconds to REPLACE just 40K in 10K batches.

root@perf3 ~ # mysql -P9306 -h0 -e "drop table if exists pq; create table pq (f text, f2 text, j json, s string) type='percolate';"; date; for m in `seq 1 1000`; do (echo -n "insert into pq (id,query,filters,tags) values "; for n in `seq 1 1000`; do echo -n "(0,'@f (cat | ( angry dog ) | (cute mouse)) @f2 def', 'j.json.language=\"en\"', '{\"tag1\":\"tag1\",\"tag2\":\"tag2\"}')"; [ $n != 1000 ] && echo -n ","; done; echo ";")|mysql -P9306 -h0; done; date; mysql -P9306 -h0 -e "select count(*) from pq"

Wed Dec 22 10:24:30 AM CET 2021
Wed Dec 22 10:25:18 AM CET 2021
+----------+
| count(*) |
+----------+
|  1000000 |
+----------+

root@perf3 ~ # date; (echo "begin;"; for offset in `seq 0 10000 30000`; do n=0; echo "replace into pq (id,query,filters,tags) values "; for id in `mysql -P9306 -h0 -NB -e "select id from pq limit $offset, 10000 option max_matches=1000000"`; do echo "($id,'@f (tiger | ( angry bear ) | (cute panda)) @f2 def', 'j.json.language=\"de\"', '{\"tag1\":\"tag1\",\"tag2\":\"tag2\"}')"; n=$((n+1)); [ $n != 10000 ] && echo -n ","; done; echo ";"; done; echo "commit;") > /tmp/replace.sql; date
Wed Dec 22 10:26:23 AM CET 2021
Wed Dec 22 10:26:27 AM CET 2021
root@perf3 ~ # time mysql -P9306 -h0 < /tmp/replace.sql

real    6m46.195s
user    0m0.035s
sys     0m0.008s

New release 4.2.0

It takes 34 seconds to insert 1M PQ rules and 23 seconds to REPLACE them in 10K batches.

root@perf3 ~ # mysql -P9306 -h0 -e "drop table if exists pq; create table pq (f text, f2 text, j json, s string) type='percolate';"; date; for m in `seq 1 1000`; do (echo -n "insert into pq (id,query,filters,tags) values "; for n in `seq 1 1000`; do echo -n "(0,'@f (cat | ( angry dog ) | (cute mouse)) @f2 def', 'j.json.language=\"en\"', '{\"tag1\":\"tag1\",\"tag2\":\"tag2\"}')"; [ $n != 1000 ] && echo -n ","; done; echo ";")|mysql -P9306 -h0; done; date; mysql -P9306 -h0 -e "select count(*) from pq"

Wed Dec 22 10:06:38 AM CET 2021
Wed Dec 22 10:07:12 AM CET 2021
+----------+
| count(*) |
+----------+
|  1000000 |
+----------+

root@perf3 ~ # date; (echo "begin;"; for offset in `seq 0 10000 990000`; do n=0; echo "replace into pq (id,query,filters,tags) values "; for id in `mysql -P9306 -h0 -NB -e "select id from pq limit $offset, 10000 option max_matches=1000000"`; do echo "($id,'@f (tiger | ( angry bear ) | (cute panda)) @f2 def', 'j.json.language=\"de\"', '{\"tag1\":\"tag1\",\"tag2\":\"tag2\"}')"; n=$((n+1)); [ $n != 10000 ] && echo -n ","; done; echo ";"; done; echo "commit;") > /tmp/replace.sql; date
Wed Dec 22 10:12:31 AM CET 2021
Wed Dec 22 10:14:00 AM CET 2021
root@perf3 ~ # time mysql -P9306 -h0 < /tmp/replace.sql

real    0m23.248s
user    0m0.891s
sys     0m0.047s

Minor changes

• optimize_cutoff is now available as a configuration option in section searchd. It’s useful when you want to limit the RT chunks count in all your indexes to a particular number globally.
• Commit 00874743 accurate count(distinct …) and FACET … distinct over several local physical indexes (real-time/plain) with identical fields set/order.
• PR #598 bigint support for YEAR() and other timestamp functions.
• Commit 8e85d4bc Adaptive rt_mem_limit. Previously Manticore Search was collecting exactly up to rt_mem_limit of data before saving a new disk chunk to disk, and while saving was still collecting up to 10% more (aka double-buffer) to minimize possible insert suspension. If that limit was also exhausted, adding new documents was blocked until the disk chunk was fully saved to disk. The new adaptive limit is built on the fact that we have auto-optimize now, so it’s not a big deal if disk chunks do not fully respect rt_mem_limit and start flushing a disk chunk earlier. So, now we collect up to 50% of rt_mem_limit and save that as a disk chunk. Upon saving we look at the statistics (how much we’ve saved, how many new documents have arrived while saving) and recalculate the initial rate which will be used next time. For example, if we saved 90 million documents, and another 10 million docs arrived while saving, the rate is 90%, so we know that next time we can collect up to 90% of rt_mem_limit before starting flushing another disk chunk. The rate value is calculated automatically from 33.3% to 95%.
• Issue #628 unpack_zlib for PostgreSQL source. Thank you, Dmitry Voronin for the contribution.
• Commit 6d54cf2b indexer -v and --version. Previously you could still see indexer’s version, but -v/--version were not supported.
• Issue #662 infinit mlock limit by default when Manticore is started via systemd.
• Commit 63c8cd05 spinlock -> op queue for coro rwlock.
• Commit 41130ce3 environment variable MANTICORE_TRACK_RT_ERRORS useful for debugging RT segments corruption.

Breaking changes

• Binlog version was increased, binlog from previous version won’t be replayed, so make sure you stop Manticore Search cleanly during upgrade: no binlog files should be in /var/lib/manticore/binlog/ except binlog.meta after stopping the previous instance.
• Commit 3f659f36 new column “chain” in show threads option format=all. It shows stack of some task info tickets, most useful for profiling needs, so if you are parsing show threads output be aware of the new column.
• searchd.workers was obsoleted since 3.5.0, now it’s deprecated, if you still have it in your configuration file it will trigger a warning on start. Manticore Search will start, but with a warning.

Bugfixes

• ❗Issue #650 Manticore 4.0.2 slower than Manticore 3.6.3. 4.0.2 was faster than previous versions in terms of bulk inserts, but significantly slower for single document inserts. It’s been fixed in 4.2.0.
• ❗Commit 22f4141b RT index could get corrupted under intensive REPLACE load, or it could crash
• Commit 03be91e4 fixed average at merging groupers and group N sorter; fixed merge of aggregates
• Commit 2ea575d3 indextool --check could crash
• Commit 7ec76d4a RAM exhaustion issue caused by UPDATEs
• Commit 658a727e daemon could hang on INSERT
• Commit 46e42b9b daemon could hang on shutdown
• Commit f8d7d517 daemon could crash on shutdown
• Commit 733accf1 daemon could hang on crash
• Commit f7f8bd8c daemon could crash on startup trying to rejoin cluster with invalid nodes list
• Commit 14015561 distributed index could get completely forgotten in RT mode in case it couldn’t resolve one of its agents on start
• Issue #683 attr bit(N) engine=‘columnar’ fails
• Issue #682 create table fails, but leaves dir
• Issue #663 Config fails with: unknown key name ‘attr_update_reserve’
• Issue #632 Manticore crash on batch queries
• Issue #679 Batch queries causing crashes again with v4.0.3
• Commit f7f8bd8c fixed daemon crash on startup trying to re-join cluster with invalid nodes list
• Issue #643 Manticore 4.0.2 does not accept connections after batch of inserts
• Issue #635 FACET query with ORDER BY JSON.field or string attribute could crash
• Issue #634 Crash SIGSEGV on query with packedfactors