In this article we talk about the new additions in the character set tables and stopwords which aim to simplify the usage of these options when configuring indexes.

When initially analyzing a document text, Manticore Search needs to know which symbols in the text are meaningful for further processing (breaking a full text into separate words, processing morphology and so on) and which are not. To define valid text characters, charset_table option is used. Through this option you can specify the set of symbols you want to work with.

Also, to provide better text search quality, Manticore Search performs so-called character folding. For example, when analyzing your search query uppercase symbols are mapped to lowercase ones, diacritic symbols (such as â, ê, î, ô etc.) are be mapped to their basic equivalents and so on. Charset table is the mechanism which defines all these transformations.

A special case are languages like Chinese which feature unsegmented text that cannot be easily split into separate words as they have no clear word separators. (‘CJK’ is a widely used term for these languages, an abbreviation from chinese-japanese-korean.) To provide effective search for these languages, ngram_chars option together with ngram_len should be used. It treats the text as a set of separate N-grams, each N-gram being a sequence of characters with a length equal to ngram_len value. (Currently ngram_len = 1 is supported only.) These characters need to be define in ngram_chars directive and cannot appear in the regular charset_table.

So if you want your search to support different languages you’ll need to define sets of valid characters and folding rules for all of them which can be quite a laborious task.

We’ve performed this task for you by preparing default charset tables that comprise non-cjk and cjk-languages respectively. These tables should be sufficient to use in most cases. They are based on ICU character foldings data that can be seen here and define the following foldings:

• ascii native digit folding

• case folding

• diacritic folding

• Han radical folding

• Hiragana folding

• Katakana folding

• letterform folding

• simplified Han folding

• subscript folding

• superscript folding

• width folding

Here you can see the source files used for default charsets : cjk.txt , non_cjk.txt

To be able to work with both cjk and non-cjk languages you should set the options in your configuration file as shown below:

charset_table     = non_cjk

...

ngram_len         = 1

ngram_chars     = cjk

In case you don’t need support for cjk-languages you can just omit ngram_len and ngram_chars options.

If you still need some specific rules to be enabled for certain languages you can create the appropriate characters tables manually and put them in your configuration file instead of default ones.

In addition to the above-mentioned options, you can use a stopwords option to exclude some of most frequent words which usually don’t have much relevance to the search queries from further processing and indexing . E.g., for English language those will probably include a, the, this, that and so on. To enable this option, create your stopwords file and set the path to it in your configuration file:

stopwords         = /path/to/your/stopwords/file.txt

The format of stopwords file is a plain text, containing the list of all words you would like to be regarded as stop words, with one word per line. To create a stopwords file one would either had to build it manually or by using indexer tool which can provide a list with the highest frequency words in the index.

Alternatively, just as in the case with charset_table and ngram_chars options, now you can use one of our default stopwords files. Currently stopwords for 50 languages are available. Here is the full list of aliases for them:

• af - Africaans

• ar - Arabic

• bg - Bulgarian

• bn - Bengali

• ca - Catalan

• ckb- Curdish

• cz - Czech

• da - Danish

• de - German

• el - Greek

• en - English

• eo - Esperanto

• es - Spain

• et - Estonian

• eu - Basque

• fa - Persian

• fi - Finnish

• fr - French

• ga - Irish

• gl - Galician

• hi - Hindi

• he - Hebrew

• hr - Croatian

• hu - Hungarian

• hy - Armenian

• id - Indonesian

• it - Italian

• ja - Japanese

• ko - Korean

• la - Latin

• lt - Lithuanian

• lv - Latvian

• mr - Marathi

• nl - Dutch

• no - Norwegian

• pl - Polish

• pt - Portuguese

• ro - Romanian

• ru - Russian

• sk - Slovak

• sl - Slovenian

• so - Somali

• st - Sotho

• sv - Swedish

• sw - Swahili

• th - Thai

• tr - Turkish

• yo - Yoruba

• zh - Chinese

• zu - Zulu

E.g., to use stopwords for Italian language, just put the following line in your config file:

stopwords         = it

If you need to use stopwords for multiple languages you should list all their aliases, separated with commas:

stopwords         = en,it,ru

All default stopwords files are stored in their separate folder. You can find them in /share/stopwords within your Manticore installation folder on Windows machines and in /usr/local/share/manticore/stopwords on Linux machines. The list of all languages which default stopwords are currently provided for can be found there in languages_list.txt as well.