We are pleased to announce the release of Manticore Search 3.3.0. Downloads are available here. Docker image is available at Docker Hub.

New features

Multi-threaded RT

One of 2 major features in 3.3.0 is multi-threading search for Real-Time indexes. In case of Real-Time index with multiple disk chunks (which often happens if you run OPTIMIZE seldom), using dist_threads > 1 will enable the search to use multiple threads for searching in parallel in the disk chunks. Until now a single thread was used by each RT index and if it had multiple disk chunks all of them were searched one by one making the search slow (or very slow). If your application is not CPU bound the new functionality can significantly increase your performance. The performance may be even higher than after OPTIMIZE.

CREATE TABLE^α and DROP TABLE^α

Another major feature available since 3.3.0 is support of CREATE and DROP TABLE. From now if you use Real-time indexes (which we highly recommend) you may not have a config for them at all. Here is how your config can look now:

searchd {
  listen = 9306:mysql41
  data_dir = data
  pid_file = 9306.pid
  binlog_path = data
}

And that’s it. After that to create an index just do

mysql> create table idx(f field, a int);
Query OK, 0 rows affected (0.00 sec)

and your index will be there:

mysql> desc idx;
+-------+--------+----------------+
| Field | Type | Properties       |
+-------+--------+----------------+
| id    | bigint |                |
| f     | field  | indexed stored |
| a     | uint   |                |
+-------+--------+----------------+
3 rows in set (0.00 sec)

Then just drop it when it’s not needed:

mysql> drop table idx;
Query OK, 0 rows affected (0.00 sec)

mysql> show tables;
Empty set (0.00 sec)

What it means is that now from your application it’s easy to control not only your data, but your schema too. Isn’t it awesome?
Caution: this feature is in alpha stage. It’s not documented yet, the syntax may change insignificantly and we don’t know anyone who uses it in production. If you want to be a beta-tester - let us know, we’ll work together to make sure it’s working properly.

In order to use this feature the configuration must not have any index/source definitions and data_dir needs to be set as in the example below. This mode doesn’t support plain indexes.

Existing setups work as before, but you cannot use the CREATE/DROP TABLE command there.

indexer --print-rt

Indexer now has a new option –print-rt which dumps the data from a plain index source as INSERT commands for a Real-Time index. Along with this, the schema of the data is provided as Real-Time fields/attributes.

EXPLAIN QUERY

EXPLAIN QUERY is a command to return full-text query execution tree just like SHOW PLAN, but unlike that it doesn’t perform actual search over the index and the index can be empty or even template type. Here is an example:

mysql> explain query idx 'abc "def ghi" "a b c d"~5 I'\G
*************************** 1. row ***************************
Variable: transformed_tree
Value: AND(
AND(KEYWORD(abc, querypos=1)),
PHRASE(
AND(KEYWORD(def, querypos=2)),
AND(KEYWORD(ghi, querypos=3))),
PROXIMITY(distance=5,
AND(KEYWORD(b, querypos=4)),
AND(KEYWORD(c, querypos=5)),
AND(KEYWORD(d, querypos=6))))
1 row in set (0.00 sec)

Other improvements

• Updated to Snowball 2.0 stemming library. This increases the number of stemmers to 25, all are included in our packages
• SHOW INDEX STATUS statement received LIKE for easier filtering
• Memory usage has been greatly improved for queries with high max_matches over distributed indexes
• For systemd packages we’ve changed LimitNOFILE from ‘infinity’ to 65536
• Percolate Queries are now lockless - in previous versions, a long CALL PQ or update to stored queries would interfere each other

A number of bugs have been fixed in this release, to name a few:

• HIGHLIGHT() issues with passage macro (493a5e9) and html stripping (ca81114)
• in some cases RT indexes created too many disk chunks (8bea0f6, a82d41c)
• docstore issues when used with ATTACH (6895374) and distributed indexes (d6f696e)

For full change log please consult the release notes.