Research on Jemalloc

As said on http://jemalloc.net/:

jemalloc is a general purpose malloc(3) implementation that emphasizes fragmentation avoidance and scalable concurrency support.

Some Sphinx and Manticore Search users prefer jemalloc over malloc and say it allows to save some RAM. I've tested how jemalloc affects resource consumption and response time for 2 weeks on 3 Manticore clusters (se03/03-2, se04/04-2, se05/05-2) each of which:

  • was getting the same amount of traffic
  • was hosting about the same amount of data
  • had 2 Manticore servers fully identical in terms of hardware, software and with 100% identical indexes and even load balancing between the servers
On se03 and se03-2 I enabled jemalloc and disabled THP (transparent huge pages) and restarted Manticore on se03-2
On se04-2 I just restarted Manticore, I didn't touch se04. se04/04-2 is a control group.
On se05-2 I enabled jemalloc, but didn't disable THP

 

Here's what I've got in 2 weeks:

RSS RAM consumption:

jemalloc_rss

Latency dynamics (I used moving average(response time)):

jemalloc_latency_1

Aggregated latency statistics:

jemalloc_latency_2

Conclusions:

1) RSS RAM: enabling jemalloc with or without disabling THP decreases RSS RAM consumption by searchd by 5-6%
2) enabling jemalloc without disabling THP (se05-2) makes latency worse significantly: avg(response time) - by 23%, 95/99p may be not affected since the differences are within the margin of error (see the control group of se04/04-2 where nothing was changed)
3) enabling jemalloc with disabling THP (se03-2) makes:
- avg latency 10% better
- 95% latency 8% worse
- 99% latency 3.6% better, but this is within the margin of error, so we can't count this
So in this case we need to decide what to prefer:
1) save 5% RAM and have 10% better avg latency, but higher latency spikes for 5% of heavier requests
2) or leave all as is: no RAM saving, worse avg latency, but better latency for heavier queries
The results may depend on many things: hardware, workload (amount of data, OPS etc.), but it looks like jemalloc is an interesting thing to try out and can save some RAM/money especially in large installation where saving 5% can mean not buying few new servers.

 
Let us know if you need our help with applying it on your installation. We can help you with other performance and resources optimizations as well.

P.S. Thanks https://www.sphinxconnector.net/ for the idea to try out jemalloc.

Leave a Reply

Training

Personal and team training will maximize them performance. 

Custom development

Need cone custom or individual features?

Fill the form and don’t forget to make the description of what you need.

Free config review

There are often optimizations that can be made to a Sphinx / Manticore setup by changing some simple directives in the configuration or making quick changes to an index definition.

Some common mistakes and issues can include:

  • doing main+delta without kill-lists, even if the delta does include updated records found in the main
  • using wildcarding with very short prefix/infix which can hammer performance in some cases
  • disabled (unintentional) seamless rotates and getting stalls on index rotations
  • adding texts as string attributes even if they are not using for any kind of operation (filtering, grouping, sorting) or mandatory to be present in results
  • using deprecated settings 

Having a quick look on the configuration can show issues or potential issues, this is why we want to offer a gift to our growing community!

When uploading your configuration file, we recommend to remove any database credentials first.

We suggest also you give as many possible details about your setup: how big is the data you have, how typical queries look and what issues you experience.

Contact us