In this article we test how percolate queries perform in Manticore Search and Luwak.

Introduction

Recently we tested the performance of Percolate Queries in Manticore Search and Elasticsearch. Today we are looking at Luwak, a stored query engine based on Lucene search library.

When we developed Percolate Queries we looked at the existing engines that implement this feature. The benchmarks provided by Flax (authors of Luwak) done here made us test Manticore against Luwak as it was outperforming Elasticsearch. In the first tests, we found out Luwak was faster than our implementation, which helped us to identify some places that could be optimized and improve (a lot in some cases) the performance of Percolate Queries in Manticore Search.

Setup

The benchmarks were conducted by Stanislav Klinov, one of the Senior Core Developers of Manticore Search and also a long-time developer of Sphinx Search.

• latest Manticore Search

• Luwak 1.5.0 with the HTTP server branch - https://github.com/tomatolog/luwak/tree/luwak-server

• http://github.com/tomatolog/luwak_tests for data and query source and script to post query to daemon via HTTP

• http://github.com/Ivinco/stress-tester for running the benchmarks

We took the luwak-server branch and merged the luwak-1.5.0 branch to have both the HTTP server and the latest version of Luwak.

In the previous tests with Elasticsearch we only used 4 words queries (2 MUST and 2 NOT). Now we tried to reproduce the conditions of the benchmark made by Luwak (against Elasticsearch). We looked after 3 types of stored queries:

• queries of 7 words, of which 5 are MUST and 2 are NOT

• queries of 120 words, of which 100 are MUST and 20 are NOT

• queries of 20 wildcard words (each made of 4 char suffixes)

Our goal was to perform inverse searches using single and batched documents. The percolate queries are fired by either a single search worker or multiple workers (to test in concurency mode).

The percolates are executed using the HTTP APIs, for this reason we merged the Luwak branch containing the HTTP server with the latest stable code.

Tests

Luwak performs a bit better when we’re not using batching, but as soon as we start to batch documents, Manticore takes the lead.

5x MUST + 2x NOT over 100K stored queries

The average latencies also grow faster on Luwak as we start batching documents.

To put pressure on the engines, we added multiple concurent search workers that siege the engines. Luwak performs somehow better on 100 docs batch, as Manticore slows down, but it's still around 2x slower.

The average latencies follow the same pattern: Luwak has higher latencies when documents are batched.

100x MUST + 20x NOT over 100K stored queries

This case hit hard on Luwak.

20 wildcards of 4 chars length over 5k stored queries

When doing the tests we discovered a performance issue (which is not fixed yet) in Manticore when using stored queries with wildcards and batched documents. The throughput drops as we add several docs in a batch. Luwak, in contrast to previous tests, perform bad in non-batched mode, but as we batch documents it slightly outperforms Manticore.

When bombing the engines with concurent workers, there is no clear winner.

On avg latencies, Luwak is in some cases a littlte bit better when the number of search workers is lower, but the differences are not notable.

Conclusion

For stored queries with several non-wilcard terms, the only case when Luwak is on par with Manticore is when in a non-batched mode. As soon as we batch documents, Manticore highly outperforms Luwak in throughput and the average latencies are usual at half of Luwak’s.

The worst case scenario for Luwak is when stored queries have over hundred of non-wildcard terms. Even in non-batch mode, Manticore is over 3 times faster, reaching about 20x faster in case of 10 batched documents. Luwak also has extremely high latencies when documents get batched for matching.

For stored queries with wildcarded terms, there is no clear winner. Luwak performs slightly better in some cases. There’s an acknowledged bug on our side (Bloom reject filters don’t work as expected) which affects the performance. The best case for Luwak was on 20 docs batch with a single search worker running, when it’s almost 2x faster, but for the rest of the cases, the gains are much smaller or Manticore is a bit ahead.