Manticore Search 3.3.0: multi-threaded RT, CREATE TABLE, EXPLAIN QUERY and more

We are pleased to announce the release of Manticore Search 3.3.0. Downloads are available here. Docker image is available at Docker Hub.

New features

Multi-threaded RT

One of 2 major features in 3.3.0 is multi-threading search for Real-Time indexes. In case of Real-Time index with multiple disk chunks (which often happens if you run OPTIMIZE seldom), using dist_threads > 1 will enable the search to use multiple threads for searching in parallel in the disk chunks. Until now a single thread was used by each RT index and if it had multiple disk chunks all of them were searched one by one making the search slow (or very slow). If your application is not CPU bound the new functionality can significantly increase your performance. The performance may be even higher than after OPTIMIZE.

CREATE TABLEα and DROP TABLEα

Another major feature available since 3.3.0 is support of CREATE and DROP TABLE. From now if you use Real-time indexes (which we highly recommend) you may not have a config for them at all. Here is how your config can look now:

searchd {
  listen = 9306:mysql41
  data_dir = data
  pid_file = 9306.pid
  binlog_path = data
}

And that's it. After that to create an index just do

mysql> create table idx(f field, a int);
Query OK, 0 rows affected (0.00 sec)

and your index will be there:

mysql> desc idx;
+-------+--------+----------------+
| Field | Type | Properties       |
+-------+--------+----------------+
| id    | bigint |                |
| f     | field  | indexed stored |
| a     | uint   |                |
+-------+--------+----------------+
3 rows in set (0.00 sec)

 

Then just drop it when it's not needed:

mysql> drop table idx;
Query OK, 0 rows affected (0.00 sec)

mysql> show tables;
Empty set (0.00 sec)

What it means is that now from your application it's easy to control not only your data, but your schema too. Isn't it awesome?

Caution: this feature is in alpha stage. It's not documented yet, the syntax may change insignificantly and we don't know anyone who uses it in production. If you want to be a beta-tester - let us know, we'll work together to make sure it's working properly.

In order to use this feature the configuration must not have any index/source definitions and data_dir needs to be set as in the example below. This mode doesn't support plain indexes.

Existing setups work as before, but you cannot use the CREATE/DROP TABLE command there.

indexer --print-rt

Indexer now has a new option --print-rt which dumps the data from a plain index source as INSERT commands for a Real-Time index. Along with this, the schema of the data is provided as Real-Time fields/attributes.

EXPLAIN QUERY

EXPLAIN QUERY is a command to return full-text query execution tree just like SHOW PLAN, but unlike that it doesn't perform actual search over the index and the index can be empty or even template type. Here is an example:

mysql> explain query idx 'abc "def ghi" "a b c d"~5 I'\G
*************************** 1. row ***************************
Variable: transformed_tree
Value: AND(
AND(KEYWORD(abc, querypos=1)),
PHRASE(
AND(KEYWORD(def, querypos=2)),
AND(KEYWORD(ghi, querypos=3))),
PROXIMITY(distance=5,
AND(KEYWORD(b, querypos=4)),
AND(KEYWORD(c, querypos=5)),
AND(KEYWORD(d, querypos=6))))
1 row in set (0.00 sec)

Other improvements

  • Updated to Snowball 2.0 stemming library. This increases the number of stemmers to 25, all are included in our packages
  • SHOW INDEX STATUS statement received LIKE for easier filtering
  • Memory usage has been greatly improved for queries with high max_matches over distributed indexes
  • For systemd packages we've changed LimitNOFILE from 'infinity' to 65536
  • Percolate Queries are now lockless - in previous versions, a long CALL PQ or update to stored queries would interfere each other

A number of bugs have been fixed in this release, to name a few:

  • HIGHLIGHT() issues with passage macro (493a5e9) and html stripping (ca81114)
  • in some cases RT indexes created too many disk chunks (8bea0f6, a82d41c)
  • docstore issues when used with ATTACH (6895374) and distributed indexes (d6f696e)

 

For full change log please consult the release notes.

2 thoughts on “Manticore Search 3.3.0: multi-threaded RT, CREATE TABLE, EXPLAIN QUERY and more

  • Sorry ignore previous comment, I misread it as dumping an existing RT index. Rather than dumping (during indexing) a plain index.

Leave a Reply

Training

Personal and team training will maximize them performance. 

Custom development

Need cone custom or individual features?

Fill the form and don’t forget to make the description of what you need.

Free config review

There are often optimizations that can be made to a Sphinx / Manticore setup by changing some simple directives in the configuration or making quick changes to an index definition.

Some common mistakes and issues can include:

  • doing main+delta without kill-lists, even if the delta does include updated records found in the main
  • using wildcarding with very short prefix/infix which can hammer performance in some cases
  • disabled (unintentional) seamless rotates and getting stalls on index rotations
  • adding texts as string attributes even if they are not using for any kind of operation (filtering, grouping, sorting) or mandatory to be present in results
  • using deprecated settings 

Having a quick look on the configuration can show issues or potential issues, this is why we want to offer a gift to our growing community!

When uploading your configuration file, we recommend to remove any database credentials first.

We suggest also you give as many possible details about your setup: how big is the data you have, how typical queries look and what issues you experience.

Contact us