TL;DR

We are pleased to introduce two new important features in Manticore Search: Fuzzy Search and Query Suggestions (or “Autocomplete”). These features improve search capabilities, offering a more user-friendly experience. You can see them in action in our open-source GitHub Issue Search Demo.

• Check out how it works in the GitHub demo
• Explore the open-source project: GitHub Demo Repository

Introduction

You may have read about the GitHub Issue Search demo and how we added Semantic Search to it to it. Today, we want to share two new features that further improve the search experience: Fuzzy Search and Query Suggestions(also known as “Autocomplete”):

• Fuzzy Search helps users find results even with small typos.
• Query Suggestions suggest relevant phrases while users type, speeding up the search process.

These features not only improve the usability of our search tool for GitHub issues but also demonstrate the advanced capabilities of Manticore Search in handling real-world search scenarios, enhancing the user interface and providing more relevant results.

Implementing Fuzzy Search

The Challenge of Typo Tolerance

One common frustration with search tools is when a simple typo leads to no results. Users expect search engines to understand their intent, even when they make small spelling mistakes. In some cases this can be tackled by using context understanding with semantic search, however, for traditional keyword searches, fuzzy search is the ideal solution.

Legacy Fuzzy Matching with CALL QSUGGEST

Historically, Manticore Search offered built-in support for fuzzy matching through its CALL QSUGGEST and CALL SUGGEST methods. The first allows you to find the last word in your given phrase with a Levenshtein distance that enables a kind of typo-tolerant search, but only for the last word. And the latter works for the first word in your phrase only. Example using the Manticore PHP client:

$params = [
    'index' => 'issue',
    'body' => [
        'query'=> 'fzzy',
        'options' => [
            'limit' => 10,
            'max_edits' => 2,
        ],
    ],
];
$response = $client->suggest($params);

In this code snippet the client uses the CALL SUGGEST function to find suggestions for the misspelled single word “fzzy”. The max_edits parameter controls the maximum Levenshtein distance allowed between the input word and potential suggestions. The Levenshtein distance is the minimum number of single-character edits (insertions, deletions, or substitutions) required to change one word into another. By setting max_edits to 2, we allow up to 2 character differences, making it possible to find typos and similar words within that edit distance.

While this approach is powerful, it can be somewhat complex because you need to handle all the suggestions returned by the function in your application. Additionally, CALL QSUGGEST doesn’t handle keyboard layout guessing (we’ll discuss it soon). To make fuzzy search more user-friendly and easier to integrate, we introduced a new option in Manticore Search that allows fuzzy search with specific parameters, without requiring extra code or custom logic.

New Simplified Fuzzy Matching

Now, we have a simpler option. By adding fuzzy=1 to your search query options, you can enable fuzzy search easily.

Here’s a simple example:

$client = new Client();
$index = $client->index('issues');

$query = 'fzzy serch';
$result = $index->search($query)->option('fuzzy', 1)->get();

foreach ($result as $hit) {
    echo $hit->getTitle() . "\n";
}

This code enables fuzzy search for the misspelled “fzzy serch,” allowing Manticore to find “fuzzy search.”. Here’s what it looks like in our Github Search Demo:

You can read more about fuzzy matching in the fuzzy search documentation.

Implementing Query Suggestions (Autocomplete)

The Power of Predictive Search

Query suggestions or autocomplete features make searching faster and help users discover relevant queries they might not have thought of:

Legacy Autocomplete with CALL KEYWORDS

Same as for CALL QSUGGEST, for fuzzy matching, Manticore Search historically has another method, CALL KEYWORDS, that can be used for autocomplete functionality. This method allows you to perform prefix and infix matching on keywords in your table, providing a simple yet effective way to implement autocomplete features. Unfortunately, it only works with strict prefix/suffix matching without typo tolerance, meaning if a user makes a mistake, it won’t find any keywords. Here’s how you can use it with the Manticore PHP client:

$index = 'myindex';
$query = 'pref*';

$response = $client->keywords($index, $query);

foreach ($response as $keyword) {
    echo $keyword['normalized'] . ' (docs: ' . $keyword['docs'] . ', hits: ' . $keyword['hits'] . ")\n";
}

In this example, we’re using the CALL KEYWORDS method to find all keywords in the myindex table that start with pref. The asterisk (*) at the end of pref* indicates a prefix match. You can also use infix matching by placing asterisks on both sides of your search term, like *some*.

While this method is quick and straightforward for implementing autocomplete, it lacks the flexibility of fuzzy matching. You need exact matches for the given prefix or infix, making it suitable for cases where precise keyword suggestions are required. This is where the new autocomplete method comes in to improve the search experience.

New Autocomplete Method

We’ve introduced a new method that builds suggestions from multiple input words and supports misspellings. This method combines the functions of CALL KEYWORDS and CALL QSUGGEST.

For example, here’s how to use the new autocomplete feature in the PHP Client:

$client = new Client();
$result = $client->autocomplete([
    'body' => [
        'table' => 'issues',
        'query' => 'hllo wor',
        'options' => [
            'fuzziness' => 1,
            'layouts' => ['us', 'ru'],
        ],
    ],
]);

foreach ($result[0]['data'] as $suggestion) {
    echo $suggestion . "\n";
}

This code suggests “hello world” for the input “hllo wor,” taking into account typos and keyboard layout issues.

Here’s how it’s done internally: we reused the fuzziness logic from Fuzzy Search to create relevant suggestions. The method uses the low-level CALL KEYWORDS and CALL QSUGGEST functionality to generate variants from your dataset.

To ensure accurate suggestions, we assess the distance based on word length and the number of documents available. For instance, if a user types “hllo wor” and your data includes “hello world,” the system will suggest “hello world” due to its fuzziness feature. This approach helps users receive accurate suggestions, even with misspellings.

You can find more information about Autocomplete in the autocomplete documentation.

Keyboard Layout Guessing

We also added keyboard layout guessing. If you’ve ever typed with the wrong layout, you’ll appreciate this feature. Manticore can now identify your intended typing even with layout switches.

We’ve mapped common keyboard layouts, including language-specific ones and QWERTY. Now, Manticore can determine your intended input, enhancing accuracy even when layout errors occur.

Benefits include:

• Improved search accuracy for multilingual users
• Eliminates frustration from accidental layout switches
• Seamless integration with our fuzzy search and autocomplete features

Applying New Features to the GitHub Issue Search Demo

With these new features in place, we were excited to enhance our GitHub Issue Search demo. Integrating fuzzy search and autocomplete was straightforward, thanks to Manticore’s user-friendly design.

To enable fuzzy search, we simply included these lines in our search query:

$search->option('fuzzy', 1);
$search->option('layouts', ['ru', 'us', 'ua']);

For query suggestions, we utilized the new autocomplete function:

$result = $client->autocomplete([
    'body' => [
        'table' => $table,
        'query' => $query,
        'options' => [
            'fuzziness' => 1,
            'layouts' => ['ru', 'ua', 'us'],
        ],
    ],
]);

These enhancements significantly improved the search functionality in our demo, delivering more precise results and useful suggestions to users.

How it looks on Github Issue Search demo

The integration of Autocomplete and Fuzzy Search with Keyboard Layout Guessing using Manticore Search improves how users interact with the search tool. By leveraging fuzziness for searches and offering suggestions based on typing patterns, users can quickly find relevant results.

Here’s a look at how the autocomplete feature operates, mimicking popular search suggestion interfaces:

Additionally, fuzzy search enhances user interaction by addressing common issues like misspellings. If someone searches for a product but makes a typo, fuzzy search ensures they still receive relevant results. Here’s how it looks in the GitHub Issue Search demo:

Conclusion

By integrating Fuzzy Search and Query Suggestions into Manticore Search, we have greatly enhanced its capabilities. These features make searches more intuitive, forgiving, and efficient.

The latest version is ready for you in the development preview release. Check out the documentation to install it and unlock these exciting features.

We encourage you to try these new features in our GitHub Issue Search demo and share your feedback. Your insights are invaluable as we continue to refine and expand Manticore Search’s capabilities.

Stay tuned for more updates, and please consider giving our GitHub repository a star if you find the project useful!