Hello!
In 3.3.0 we introduced a new daemon mode. We call it real-time mode (RT mode). It allows managing your indexes (tables) using CREATE TABLE and DROP TABLE. The plain mode which assumes defining index schemas in a configuration file remains supported as well.
Before that, you could insert and delete data from a real-time index, but there was no way to add and drop indexes via SQL/HTTP.
The new mode is enabled by specifying data_dir in the config file. Note that the configuration file is only used for daemon settings, no source or index sections are allowed in this mode.
ALTER TABLE can now be used to change index settings, for example
ALTER TABLE products ignore_chars='.'
However, these new settings only affect newly inserted documents; they don’t affect existing documents in the index.
data_dir
data_dir specifies the directory where all new indexes will be stored. Each index is stored in its individual folder inside the data_dir. You must not manually add indexes to this folder because indexes created by the daemon running in the new RT mode are different internally from the indexes created in the plain mode. Use IMPORT TABLE to import old indexes. It copies index files to a new folder inside the data_dir and performs the necessary index format changes.
See below for more details on index differences.
RT mode works only with RT or PQ indexes. You can’t create or import plain indexes to the daemon running in this mode. One possible solution is to use ATTACH to add a plain index to an RT index (in the daemon running in the old mode) and then use IMPORT TABLE to import that RT index to the daemon running in RT mode.
How to backup
As you are not supposed to manipulate the contents of data_dir manually, the only safe way (for now) to backup or move data_dir to another location is to shut down the daemon, copy/move data_dir and restart the daemon.
Replication
As replication creates/drops indexes on the nodes, it only works with the daemon running in the new RT mode.
External files
Only absolute paths for external files (word forms, stop words, exceptions) are allowed in CREATE TABLE and ALTER TABLE. This is to avoid confusion because relative paths are relative to the folder where the daemon was started and the user might not know it.
Implementation details
data_dir contains the indexes and a .json file that lists these indexes and some of their properties. This file is modified by the daemon, you are not supposed to change it manually.
All the paths inside the .json config are relative. The idea is that you can copy/move data_dir to a new node/location, modify data_dir in the configuration file, and run the daemon from the new location without any additional changes.
There are also index format differences (between indexes created in RT mode and plain mode) that serve the same purpose. Indexes created in the old mode store full paths to external files. Indexes created in RT mode store only file names, the path is generated dynamically based on index name and data_dir value in the config.
There is noticeable difference in the way the daemon works with external files (word forms, exception or stop words files) when running in RT mode. Before introduction of the RT mode indexes either embedded external files in the index header or used them from the location where they are stored (depending on the settings). Now, when Manticore is running in RT mode, embedding is disabled. For example, when doing a
CREATE TABLE (title text) wordforms ='/path/to/wordforms.txt'
word forms, stop words, and exceptions are copied to the index folder inside the data_dir.
IMPORT TABLE basically does the same. It copies external files to the index folder inside the data_dir. That might lead to some file duplication if the same external files are used when creating several indexes. However, that does not necessarily mean extra RAM usage by the daemon. For example, searchd loads only one instance of word forms file if there are several indexes that use copies of the same original word forms file.
If you have any questions about the new mode don’t hesitate to ask them in:
Our public forum
Or explore it for yourself from our “Manticoresearch - RT mode - index administration” interactive course.