TL;DR
Manticore Load Emulator is an open-source benchmarking tool that helps you validate and optimize your Manticore Search deployments. Whether you’re planning a production deployment, debugging performance issues, or fine-tuning your configuration, this tool provides comprehensive testing capabilities with real-time monitoring and detailed analytics. Built for both simplicity and power, it supports everything from basic query testing to complex multi-process workload simulation.
Introduction
Here’s a fun fact: when people ask if Manticore can handle their workload, we take it as a challenge. And when performance matters as much as it does to us, you need to be prepared. Enter Manticore Load Emulator — an open-source tool designed to put Manticore (and your hardware) through its paces.
Whether you’re wondering if Manticore can handle your unique setup or just looking to squeeze every drop of performance out of it, this tool has you covered.
You can grab it on GitHub or by installing as a package from our repository (Manticore version > 6.3.8).
Why We Built It
We get asked a lot: “Can Manticore handle my workload?” Well, the short answer is usually “Yes.” But instead of making you take our word for it, we built a tool so you can test it yourself. Manticore Load Emulator is all about transparency and empowerment—plus, it’s a lot of fun to push your hardware to its limits.
Use Cases
1. Performance Validation for New Deployments
Before going live with Manticore Search, use the Load Emulator to simulate your expected workload. This ensures your infrastructure is properly scaled and configured for peak performance.
2. Debugging and Troubleshooting
Identify bottlenecks or performance degradation by running controlled tests. For example, analyze how specific query patterns or large datasets impact query latency and throughput.
3. Optimization of Existing Setups
Fine-tune configurations such as batch sizes, thread counts, or query caching to achieve optimal performance. Use the hyperparameter testing feature to automate comparisons.
4. Evaluating Infrastructure Changes
Planning to upgrade your hardware, adjust database sharding, or move to a different cloud provider? The Load Emulator allows you to benchmark and compare performance across different setups.
5. Stress Testing for Scalability
Simulate extreme workloads to evaluate how well your deployment scales under heavy traffic. This is especially useful for preparing for events like product launches or seasonal spikes.
Key Features
Here’s why Manticore Load Emulator is a standout tool for benchmarking and testing:
1. SQL-Powered Simplicity
The tool leverages Manticore’s SQL support, making it easy to emulate a wide variety of workloads. Writing your load or query scenarios is as simple as writing SQL commands.
2. High Concurrency Support
Simulate real-world, high-load scenarios using multiple threads and processes. If your server can handle it, this tool will push it to the brink.
3. Custom Query Generation
Generate dynamic queries with flexible patterns, from random text to precise ranges of integers, floats, or arrays:
valueExact value to use<increment>Auto-incrementing value starting from 1<increment/1000>Auto-incrementing value starting from 1000<string/3/10>Random string, length between 3 and 10<text/20/100>Random text with 20 to 100 words<text/{/path/to/file}/10/100>Random text using words from file, 10 to 100 words<int/1/100>Random integer between 1 and 100<float/1/1000>Random float between 1 and 1000<boolean>Random true or false<array/2/10/100/1000>Array of 2-10 elements, values 100-1000<array_float/256/512/0/1>Array of 256-512 random floats, values between 0 and 1
4. Batch Loading
Efficiently load millions of records at once with configurable batch sizes.
5. Real-Time Monitoring & Analytics
Track your test performance with comprehensive metrics including:
- Progress and throughput (QPS)
- Detailed latency percentiles
- Performance bottlenecks identification
- Real-time status updates
6. Flexible Configuration
Command-line arguments let you tweak every aspect of your workload. Need to simulate multiple types of queries? Use the --together option to run different workloads in parallel.
7. Hyperparameter Testing
Compare thread counts or batch sizes in a single run with comma-separated values. For example:
--threads=1,2,4,8 --batch-size=100,1000,10000
8. Quiet Mode for Analysis
In --quiet mode, the tool outputs semicolon-separated results that make it easy to visualize by copying directly into Google Sheets or Excel.
9. Multi-Process Support
Need to test sharded setups or multiple workloads simultaneously? This powerful flag allows you to run multiple test configurations in parallel within the same test session. Instead of running tests sequentially, which can be time-consuming, --together combines your test runs into a single, efficient execution.
For example, you can:
- Test multiple shards of your database simultaneously
- Run the same test suite against different database versions
- Validate your application across various configuration settings
- Compare performance metrics between different setups
Simply append --together to your test command, and specify multiple test configurations. The tool will handle the parallel execution while keeping the results organized and easily comparable.
10. Graceful Shutdown
If you need to stop mid-test (or accidentally hit Ctrl+C), the tool ensures a clean termination without leaving things messy.
11. Two Modes of Latency Tracking
- Histogram-Based: Memory-efficient and approximate for broad trends
- Precise: For exact latency percentiles, because sometimes precision is key
Getting Started
Here’s a quick example to get you started:
Example 1: Simple Insert
Insert 1 million integers in batches of 10K:
manticore-load \
--drop \
--init="create table t(a int)" \
--load="insert into t values(0, <int/1/1000000>)" \
--batch-size=10000 \
--total=1000000
Example 2: Simulating Concurrency
7-8% CPU load and 160K docs per second seems too low for your workload? Let’s load it up by increasing the thread count:
manticore-load \
--drop \
--init="create table t(a int)" \
--load="insert into t values(0, <int/1/1000000>)" \
--batch-size=10000 \
--threads=16 \
--total=10000000
OK, now we got 665K at 25% CPU load. Looks like we can squeeze more from the instance, but it requires writing to multiple tables at once. Let’s do that:
Example 3: Sharded Workload
Test a sharded environment by loading data into multiple tables simultaneously:
manticore-load --quiet \
--drop \
--init="create table t(a int)" \
--load="insert into t values(0, <int/1/1000000>)" \
--batch-size=10000 \
--threads=8 \
--total=5000000 \
--together \
--drop \
--init="create table t2(a int)" \
--load="insert into t2 values(0, <int/1/1000000>)" \
--batch-size=10000 \
--threads=8 \
--total=5000000 \
--together \
--drop \
--init="create table t3(a int)" \
--load="insert into t3 values(0, <int/1/1000000>)" \
--batch-size=10000 \
--threads=8 \
--total=5000000 \
--together \
--drop \
--init="create table t4(a int)" \
--load="insert into t4 values(0, <int/1/1000000>)" \
--batch-size=10000 \
--threads=8 \
--total=5000000
Too much data and you don’t need the progress? By adding --quiet flag, the tool will output only the final results:
Example 4: Combining Writes and Reads
Previous examples were focused on loading data. What if you want to test write/read performance? Let’s also use a more realistic workload than just integers.
manticore-load \
--total=1000000 \
--batch-size=10000 \
--drop \
--init="create table products(name text, price float, stock int)" \
--load="insert into products(name, price, stock) values('<text/3/10>', <float/1/100>, <int/1/100>)" \
--together \
--total=10000 \
--load="select * from products where match('<text/2/5>') and price < 10"
We can see how the read performance degrades as more data is loaded.
Example 5: Hyperparameter Testing
Let’s test how different thread counts affect the performance:
manticore-load \
--total=10000 \
--quiet \
--threads=1,4,8,16,32,64 \
--load="select * from products where match('<text/2/5>') and price < 10"
We can see that with the current schema and 1 million docs in the table the throughput can be up to 9858 selects per second at 4.3ms average latency with 6.5ms p95 latency.
Isn’t it great you can make such conclusions in just a couple of minutes?
Conclusion
Manticore Load Emulator is more than just a benchmarking tool—it’s your partner in ensuring optimal performance for your search deployments. Whether you’re running a small proof-of-concept or scaling to handle millions of queries, this tool provides the insights you need to make informed decisions about your infrastructure and configuration.
Remember: the best performance testing is iterative. Start small, gather data, make adjustments, and test again. With Manticore Load Emulator, you have all the tools you need to build and validate a high-performance search solution.