New Percentile and Median Absolute Deviation Aggregations in Manticore

When averages hide what is really happening, you need better distribution metrics. A single mean can look "healthy" while still masking long-tail slowdowns, skewed value distributions, or a small set of extreme outliers that heavily impact user experience and operational risk.
Manticore now supports percentile, percentile ranks and MAD (Median Absolute Deviation) aggregations, so you can analyze spread, tails, and outliers directly in search queries.
They are especially useful for latency monitoring, pricing analysis, user behavior, and any workload where "average" is not enough.

• percentiles(field[, {values='...',compression=N}])
  Returns estimated percentile values (for example p50, p95, p99)

• percentile_ranks(field, {values='...',compression=N})
  Returns the estimated percentage of documents with values less than or equal to each provided value

• median_absolute_deviation(field[, {compression=N}])
  Returns estimated MAD, a robust measure of spread around the median

The functions are approximate by design, which keeps memory usage bounded and performance predictable, and the optional compression controls the accuracy/memory trade-off: lower values are faster and lighter but can produce more approximation error; the default value is 200.

Usage

Manticore supports two syntax variants for these aggregations: SQL syntax and JSON API syntax.

SQL Example

SELECT
    percentiles(latency) AS p_default,
    percentiles(latency, {values='5,50,95',compression=200}) AS p_custom,
    percentile_ranks(latency, {values='10,150,1500',compression=200}) AS r_custom,
    median_absolute_deviation(latency, {compression=200}) AS mad
FROM agg_td\G

*************************** 1. row ***************************
p_default: {"1":10,"5":10,"25":20,"50":30,"75":40,"95":50,"99":50}
 p_custom: {"5":10,"50":30,"95":50}
 r_custom: {"10":20,"150":100,"1500":100}
      mad: {"value":10,"value_as_string":"10"}

How to read this

• p_custom gives direct percentile cutoffs (for example, p95 latency).
• r_custom tells you the share of docs below given thresholds.
• mad tells you how tightly values cluster around the median.

HTTP JSON Example

POST /json/search
{
  "table": "agg_td",
  "size": 0,
  "aggs": {
    "latency_percentiles": {
      "percentiles": {
        "field": "latency",
        "values": [5, 50, 95],
        "keyed": true
      }
    },
    "latency_ranks": {
      "percentile_ranks": {
        "field": "latency",
        "values": [10, 150, 1500],
        "keyed": true
      }
    },
    "latency_mad": {
      "median_absolute_deviation": {
        "field": "latency",
        "tdigest": {
          "compression": 200
        }
      }
    }
  }
}

{
  "took": 0,
  "timed_out": false,
  "aggregations": {
    "latency_percentiles": {
      "values": {
        "5.0": 10,
        "50.0": 30,
        "95.0": 50
      }
    },
    "latency_ranks": {
      "values": {
        "10.0": 20,
        "150.0": 100,
        "1500.0": 100
      }
    },
    "latency_mad": {
      "value": 10,
      "value_as_string": "10"
    }
  },
  "hits": {
    "total": 5,
    "hits": []
  }
}

Real-World Use Cases and Practical Tips

• Search/API performance: track p50/p95/p99 latency and MAD for jitter.

• E-commerce pricing: detect pricing distribution shifts and abnormal spread.

• Fraud and anomaly detection: compare median behavior vs MAD-based deviations.

• SLO reporting: combine percentile ranks with business thresholds (for example, "% under 200ms").

• Use percentiles when you need cutoffs (p95 = ?).

• Use percentile_ranks when you need percentages (<=200ms = ?).

• Use MAD when outliers are common and you need robust dispersion.

• Start with default compression=200; increase if you need higher precision.

Final Takeaway

With percentile and MAD aggregations, Manticore gives you stronger statistical visibility directly inside your search analytics workflow.
You can now monitor tails, evaluate thresholds, and measure robust spread in one place without shipping raw data to external systems first.