In this article we talk about the new options for loading data introduced in 3.0.2

An index with data is composed of several components like dictionary, documents and hits lists , attributes. The whole index is not loaded entirely into RAM as it’s possible to not fit, rather it’s components are read and loaded in different ways.

Attributes

Attributes by default are loaded in RAM, but it was possible to choose whenever to load them all, load only scalar types (integer or floats) or keep them on disk.

Attributes are loaded as memory-mapped files. The advantage of loading attributes using mmap() is better performance than classic seek&read using pread(). It also allows to use large files using small amounts of RAM.

In 2.x attribute loading was handled via the ondisk_attrs. Even when we choose to load attributes in RAM, reading an entire file with mmap() doesn’t guarantee the file’s pages will remain in RAM, as the OS kernel can free them up (this is not a problem for the application as mmaped files are backed by the file itself on the disk). To be sure attributes stay in RAM and OS doesn’t swap them out to disk, mlock needed to be used.

In 3.0.2 we decided to make a change and instead of having a setting for which attribute types to get loaded and one for mlocking to have two settings, each for the attributes types that decide how they are loaded and if they should be locked.

The new settings are access_blob_attrs - which manages variable-length attributes ( string,JSON,MVA) - and access_plain_attrs - which manages the fixed-length attributes (uint,bigint,float,timestamp,bool). Possible values are mmap, mmap_preread and mlock.

Switching from the old directives to the new ones is pretty simple:

• ondisk_attrs = 0 means no attribute is left on disk and they are preread in RAM. This was the default value along with mlock=0. This is equivalent now to access_blob_attrs = mmap_preread and access_plain_attrs= mmap_preread. This mode should be used when all attributes can fit in RAM. to make sure they are not swapped to disk we can use instead access_blog_attrs = mlock and access_plain_attrs = mlock, equivalent of ondisk_attrs=0 and mlock=1.
• ondisk_attrs=1 means all attributes are left on disk. With the new options you have to specify access_blob_attrs=mmap and access_plain_attrs=mmap.
• ondisk_attrs=pool means to let strings,JSON and MVA attributes on disk. To get same effect now you need to do access_blob_attrs=mmap and access_plain_attrs=mmap_preread.

But now it’s also possible to keep the numeric attributes on disk and load instead the var-length ones ( by doing do access_blob_attrs=mmap_preread/access_plain_attrs=mmap) if for example you don’t have enough RAM for all and your queries perform more operations (filtering,grouping, sorting) on the variable-length ones.

It’s also possible to do mlocking only of some attributes. For example we want numeric attributes to be sure they stay in RAM, but the other attributes we can preread them in RAM, but we accept to not lock them to allow some memory space that can be freed.

Full-text index

The fulltext part of an index includes several components itself. The dictionary component (.spi) is always loaded into RAM as it’s critical for full-text searching performance and it’s size is not big. The doclists (.spd) holds the references of indexed words to documents and fields. This component is usually large and it was accessed until by using pread().

Another component of the full-text index is hitlists (.spp) which contain the positions of words in fields. Hitlists can also be large (but not as big as doclists) and were accessed in same way as doctlists. Starting with 3.0.2 both components can be loaded with mmap() instead of pread() using the access_doclists and access_hitlists. The setting accepts two values: file (which uses pread() and is a default) and mmap.

How to set the memory directives?

By default file is used for full-text index components and mmap_preread for all attributes which is the most balanced profile between memory usage and search performance.

To make sure attributes data loaded in memory is not dropped by OS mlock can be used. Locking should be on systems that have plenty of memory to fit the attributes but also leave enough memory for doc/hit lists. The best performance is achieved when indexes can be fully loaded/cached in RAM. Doc/hit lists using file method can be warmed up by sending their files to /dev/null.

For the cases where having the best performance from the first query, the --force-preread option of searchd can be used. This option should be used with care since the daemon doesn’t respond to incoming queries until the startup is done. For single setups this means no search queries are possible which may not be acceptable depending on the time required to finish the startup versus responding during this time with bigger times.

If the indexes are bigger than available memory or it’s acceptable to have longer warm up times, simple file/mmap should be used. If the indexes don’t fit well in RAM the performance will be influenced by the performance of storage. Choosing between more memory and faster storage depends on the cost of each. Faster storage (like NVMe) will reduce the shortcoming of not having enough memory. Having enough memory, but slow storage means startups or index reloads will be slower and performance will suffer until indexes are loaded into memory.