blog-post

Search Relevance Regression Testing in CI

author image
View as markdown

Your search endpoint is healthy. Someone searches for help with an E403 permission error and gets instructions for refreshing a session token. The query returns results, but the wrong procedure is ranked first.

Health checks will not catch that. Neither will a test that only counts results. To spot the problem before users do, you need sampl questions, a judgment of which documents actually answer them, and a way to check rankings in CI.

Here is a runnable Manticore example that does exactly that. It compares lexical, vector, and hybrid search, then removes the lexical branch. Search keeps returning results, but the ranking gets worse and the relevance check exits 1. The download includes the test corpus, judgments, evaluator, raw engine responses, and a CI workflow example.

Start With the Right Answer

Our earlier articles on hybrid search and auto-chunking cover ways to build retrieval. The question here is what happens after you change it: are people still finding the answers they need?

For this fictional API, E403 means the user needs workspace access; E401 means the session token has expired. A high similarity or fusion score cannot tell you which procedure solves the problem. That distinction has to come from the application.

The example uses 12 short documets about invented API errors and operational procedures. They are teaching fixtures, not Manticore documentation, and the error codes are fictional. Each query has a written intent and a grade for every document:

GradeMeaning in this example
2Gives the procedure that resolves the stated intent
1Related context, but does not resolve it
0Does not help with this intent

For E403 API rejects my request, the permission procedure gets grade 2. The token-refresh procedure gets grade 1: it is about a related failure, but it will not fix a permission problem. These labels were written from the fixtures before running retrieval. They have not been independently reviewed.

Grade 1 matters for nDCG, which gives some credit to related context. But a related procedure may still be the wrong advice. That is why we also measure how high the first grade-2 answer appears in the results.

For a real search application, collect queries from support cases and important user journeys. Include convincing but incorrect answers, ask the people who know the domain to review the grades, and keep the judgments in version control. A ranking test is only as useful as the answers it considers correct.

A Small Test Set With Pinned Inputs

The package has four tuning queries and eight held-out queries. The query path was checked on the tuning set, and the initial ranking settings were kept without a parameter sweep or changes to improve held-out scores. The archive records the final inputs and outputs, not a timestamped history of that sequence.

Both groups search the same 12 documents, and one error-code intent appears in both with different wording. These eight held-out queries help test the gate; they do not tell us how search will perform on new topics. For a real application, keep a larger evaluation set separate from tuning.

To make the run repeatable, the package pins:

  • Manticore 29.9.0, pinned to Docker digest sha256:461b419739a2bbbffb4be0b32db26c41d7b435bf7fd1fce95b414ca5ccbb3775;
  • the corpus, query split, complete judgment matrix, schema, query settings, and evaluator hashes;
  • Xenova/all-MiniLM-L6-v2 at revision 751bff37182d3f1213fa05d7196b954e230abad9, with SHA-256 checks for its configuration, tokenizer, and ONNX model;
  • five returned results, a fusion window of 12, RRF rank constant 10, and text/vector weights of 2:1.

Manticore generates document and query embeddings locally, using the pinned model and no hosted inference API. On first setup, the runner downloads the model files from revision-specific URLs and checks their hashes; it then mounts them read-only. The revision pin protects against changed files, not a download becoming unavailable. For long-lived CI, keep a verified copy of the model artifacts.

The runner also checks what Manticore actually stored: it reads the documents back and runs SHOW CREATE TABLE. That catches changes made to the index even when the fixture files on disk have not changed.

Each record in queries.json has the full query and a prepared lexical input. For error-code queries, lexical search uses just the identifier; for other queries, it joins the query words with OR (|). The runner does not extract identifiers or tokenize the text itself. Vector search receives the full query. Neither branch receives the expected document ID or the relevance grades.

The hybrid query combines both result lists with reciprocal rank fusion (RRF) . Here the text branch has twice the weight of the semantic branch:

SELECT id, title, hybrid_score() AS score
FROM article04_docs
WHERE MATCH('E403') AS text
  AND knn(vec, 'E403 API rejects my request') AS semantic
LIMIT 5
OPTION fusion_method='rrf', rank_constant=10,
       window_size=12, fusion_weights=(text=2, semantic=1);

For comparision, the lexical-only path uses MATCH() with ranker=bm25; the vector-only path uses model-backed knn() without MATCH(). All three use the same relevance labels. The 2:1 weighting is a choice for this example, not a general recommendation.

Here are the resuts for the eight held-out queries. MRR@5 measures the rank of the first answer that resolves the intent; nDCG@5 also gives credit for related documents. The next section explains both metrics.

Retrieval pathMRR@5nDCG@5
Lexical0.9380.825
Vector0.8440.922
Hybrid baseline0.9380.955

Hybrid search has the best nDCG in this run. Keeping results for all three paths is also useful for debugging: when a ranking changes, you can see whether the lexical branch, vector branch, or fusion is responsible.

Check Both the Answer and the Ranking

MRR@5 asks how soon the first grade-2 answer appears. It gives a query 1 / rank when that answer is in the top five, or zero when it is not. The mean is taken across queries. A grade-1 result does not count as a solution.

nDCG@5 looks at the whole top-five ranking. Grade-2 documents get more credit than grade-1 documents, and results farther down get less:

DCG@5  = sum((2^grade - 1) / log2(rank + 1)), ranks 1 through 5
nDCG@5 = DCG@5 / ideal DCG@5

To calculate the ideal DCG, the evaluator sorts all judged documents by grade and takes the top five. It does not build an ideal ranking from the results the engine happened to return. Missing positions earn no credit. An empty ranking has a metric value of zero, although this CI example also treats an empty engine response as an error.

The unit tests include a case where a grade-1 result comes first and the grade-2 answer comes second. Its reciprocal rank is 0.5, and its nDCG is (1 + 3/log2(3)) / (3 + 1/log2(3)). They also cover cutoffs, duplicate and unknown IDs, and related documents that aren't real answers. These cases help catch mistakes in the evaluator itself.

A Change That Returns Results but Fails the Gate

Now remove lexical search from the query builder. Manticore still accepts the queries, generates embeddings, and returns documents. In the example, --candidate switches from hybrid to vector-only retrieval; it does not change the evaluator or fake a failure.

The table includes every held-out query, even where vector-only search performs better. RR is the reciprocal rank of the first resolving answer; nDCG also accounts for related results.

Query / intentBaseline RR@5Candidate RR@5Baseline nDCG@5Candidate nDCG@5
h1: E401 request rejection1.0001.0000.9450.945
h2: E403 request rejection1.0000.2501.0000.631
h3: E507 snapshot failure1.0001.0001.0001.000
h4: E509 snapshot failure1.0001.0001.0001.000
h5: Recover a deleted document0.5001.0000.7971.000
h6: Keep erased content deleted after recovery1.0000.5000.9330.797
h7: Replace a schema while serving users1.0001.0000.9641.000
h8: Replace replication TLS credentials1.0001.0001.0001.000

Look at h2: the correct permission procedure drops from first to fourth, and token refresh takes the top spot. On h5, the candidate actually improves recovery instructions, moving the right answer above permanent deletion. Both movements matter. A better result for one query does not make the permission failure acceptable.

Observed MRR at five results: all queries fall from 0.938 to 0.844, identifiers from 1.000 to 0.813, while operational tasks remain at 0.875.

The overall MRR falls by just 0.09375, so a rule allowing a 0.10 drop would pass this candidate. But MRR for identifier queries falls by 0.1875. The operational-task average stays at 0.875 because the improvement on h5 cancels the loss on h6. Looking only at averages would miss both problems.

This example uses three gates: a query may not lose more than 0.20 nDCG, a query class may not lose more than 0.10 MRR, and every class must keep MRR at or above 0.60. These thresholds are demo settings; choose your own based on the failures you can tolerate. The vector-only candidate trips two of them:

REGRESSION h2: nDCG@5 delta -0.369
REGRESSION identifier: MRR@5 delta -0.188
FAIL

The runner exits 1 for a relevance regression and 2 for setup, validation, or engine errors. Both fail CI. Pointing ARTICLE04_PORT at a closed port, for example, produces ERROR: <urlopen error [Errno 111] Connection refused> and exits 2.

Run the Check Locally and in CI

Download the example and captured run , extract it, and open relevance-ci/. You need Python 3.10+, Docker Engine, and Compose v2. There are no pip dependencies. The default loopback port is 49418; set ARTICLE04_PORT if that port is already in use.

sha256sum -c SHA256SUMS
bash reproduce.sh

reproduce.sh verifies the model, starts the manticore-article04 Compose project with a 2-CPU and 3-GiB limit, and loads the example table. It checks the baseline, runs the vector-only candidate, and expects the candidate to exit 1. The demonstration succeeds only if the baseline passes and the bad change is caught. Both runs leave command output and reports behind.

For a blocking CI step, call the relevance check directly:

# Passes for the unchanged baseline:
python3 run.py check --out runs/baseline-check

# Intentionally fails with exit 1:
python3 run.py check --candidate --out runs/candidate-check

Each check run writes per-query IDs, metrics, and the unmodified engine responses (including retrieval scores) alongside numbered SQL files. Use a fresh output directory for each run; the tool refuses to overwrite existing evidence. The captured/ directory in the archive contains the published run and negative tests. To enable ci-example.yaml, copy it into your workflow directoy. Its gate intentionally runs the failing candidate, so replace that command when wiring the check into real CI.

With the model already cached, three check runs took 0.37–0.39 seconds each on a busy shared 32-core Linux server; the engine container was capped at 2 CPUs and used about 189 MiB. Initial setup downloads roughly 87 MB into model-cache/. Caching that directory saves the download on later CI runs.

You can also exercise the failure cases:

python3 integration_negatives.py

integration_negatives.py checks missing judgments, duplicate fixture IDs, changed inputs, empty results, engine errors, and direct changes to the corpus or schema in Manticore. The last two tests briefly modify the example table and then restore it; the baseline still passes afterwards. Use these tests only against the disposable example project.

To clean up:

docker compose down -v

This removes the example container, network, and data volume. The downloaded model (about 87 MB) and reports stay on disk.

Updating the Reference Ranking

Treat the reference ranking as an approved result for a specific corpus, model, and query setup. CI should compare against it, not silently regenerate it. If any input changes, the lock should prompt a review before the next baseline is accepted.

Read the per-query results before accepting a new baseline. If a worse ranking is intentional, document why. Then capture the replacement and update the lock using the process in the README. Replacing a reference just to make CI pass defeats the purpose of this test.

The example keeps the query builder and evaluator together in run.py. Its --candidate switch demonstrates one ranking change, but it is not meant for arbitrary edits to the query builder. Updating inputs.lock.json alone will fail the reference fingerprint check. For production, separate candidate query code from frozen evaluation inputs so you can test a change without recording a new baseline. The runner accepts only Manticore 29.9.0; moving to another version requires an updated image pin and lock, followed by a review of the rankings.

This is a test of the CI mechanics on 12 fictional documents, not a production relevance benchmark. It does not measure latency, scale, generated answers, or access-control behavior. Approximate retrieval and ties can vary by environment, so the gate allows small score differences instead of demanding identical rankings. Three runs on one host produced the same results; other hardware was not tested.

When you adapt the example, start with queries that have caused real support problems. Review the answer grades with domain experts, pick thresholds that reflect the cost of getting those answers wrong, and inspect individual failures before accepting a good-looking average.

Go from zero to Manticore in seconds

Install Manticore Search in one command on Linux or macOS:

curl https://manticoresearch.com | sh

For advanced installation options, see the full installation guide and the manual .