victoriametrics-cardinality-analysis

What this skill does

# VictoriaMetrics Cardinality Analysis

Systematic cardinality analysis for VictoriaMetrics. Collects TSDB status, metric usage stats, and label
value patterns, then produces a structured report with specific relabeling and stream aggregation configs
the user can apply directly.

The goal is to find the highest-impact optimization opportunities — metrics nobody queries, labels that
explode cardinality for no monitoring value, and patterns that indicate data hygiene problems (error
messages as labels, SQL text as labels, UUIDs as labels).

## Environment

Uses the same env vars as the `victoriametrics-query` skill:

```bash
# $VM_METRICS_URL - base URL
#   cluster: export VM_METRICS_URL="https://vmselect.example.com/select/0/prometheus"
#   single: export VM_METRICS_URL="http://localhost:8428"
# $VM_AUTH_HEADER - full HTTP header line (empty if no auth is required)
#   Prod:  export VM_AUTH_HEADER="Authorization: Bearer <token>"
#   Local: export VM_AUTH_HEADER=""
```

## Workflow

### Phase 1: Data Collection

Spawn 3 subagents in a **single response** to collect data in parallel. Each subagent prompt must
include the curl auth pattern and environment variable references above.

If the user specified a scope (job, namespace, metric prefix), pass it as `match[]` parameter to
TSDB status queries and as series selectors to label queries.

---

#### Subagent 1: TSDB Overview

**Agent name**: `cardinality-tsdb` | **Description**: "Collect TSDB cardinality stats"

**Query 1 — Yesterday's series (captures recently churned series):**

```bash
curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
  "$VM_METRICS_URL/api/v1/status/tsdb?topN=50&date=$(date -d 'yesterday' +%Y-%m-%d)" | jq '.data'
```

Queries yesterday's stats — broader than today (includes series that may have already churned) without scanning the entire TSDB.

**Query 2 — Today's active series:**

```bash
curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
  "$VM_METRICS_URL/api/v1/status/tsdb?topN=50" | jq '.data'
```

**Query 3 — Focus on known high-cardinality labels:**

```bash
for label in pod instance container path url user_id request_id session_id trace_id le name; do
  echo "=== focusLabel=$label ===" && \
  curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
    "$VM_METRICS_URL/api/v1/status/tsdb?topN=20&focusLabel=$label" | \
    jq --arg l "$label" '{label: $l, focus: .data.seriesCountByFocusLabelValue}'
done
```

**Return**: All raw JSON preserving structure. Include `totalSeries`, `totalLabelValuePairs`,
`seriesCountByMetricName`, `seriesCountByLabelName`, `seriesCountByLabelValuePair` from each query.

---

#### Subagent 2: Metric Usage Stats

**Agent name**: `cardinality-usage` | **Description**: "Find unused and rarely-queried metrics"

**Query 1 — Never-queried metrics:**

```bash
curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
  "$VM_METRICS_URL/api/v1/status/metric_names_stats?le=0&limit=500" | jq '.'
```

**Query 2 — Rarely-queried metrics (≤5 total queries):**

```bash
curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
  "$VM_METRICS_URL/api/v1/status/metric_names_stats?le=5&limit=500" | jq '.'
```

**Query 3 — Stats overview (tracking period):**

```bash
curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
  "$VM_METRICS_URL/api/v1/status/metric_names_stats?limit=1" | \
  jq '{statsCollectedSince: .statsCollectedSince, statsCollectedRecordsTotal: .statsCollectedRecordsTotal}'
```

If the endpoint returns an error, `storage.trackMetricNamesStats` may not be enabled on vmstorage.
Note this in the return and proceed — the analysis can still work with TSDB status data alone.

**Query 4 — Cross-check: are "unused" metrics referenced in alerting rules?**

```bash
curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
  "$VM_METRICS_URL/api/v1/rules" | jq '[.data.groups[].rules[].query]'
```

Extract metric names from rule queries. Any "unused" metric that appears in an alert/recording rule
is NOT safe to drop — it's queried indirectly.

**Return**: Unused metrics with cross-reference against alert rules. Flag each as:

- **safe to drop**: never queried AND not in any rule
- **used by rules only**: never queried by dashboards but referenced in rules — verify intent
- **rarely used**: low query count, may be accessed infrequently (e.g., monthly reports)

---

#### Subagent 3: Label Pattern Inspection

**Agent name**: `cardinality-labels` | **Description**: "Inspect label values for problematic patterns"

All data comes from the TSDB status endpoint — do NOT use `/api/v1/labels` or `/api/v1/label/.../values`.

**Query 1 — Label cardinality overview (unique value counts + series counts):**

```bash
curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
  "$VM_METRICS_URL/api/v1/status/tsdb?topN=50" | \
  jq '{labelValueCountByLabelName: .data.labelValueCountByLabelName, seriesCountByLabelName: .data.seriesCountByLabelName}'
```

`labelValueCountByLabelName` returns labels sorted by unique value count (replaces per-label `/values` counting).
`seriesCountByLabelName` shows how many series each label appears in.

**Query 2 — Sample values for high-cardinality labels via focusLabel:**
For each label with >100 unique values from Query 1, fetch sample values:

```bash
for label in <top labels from Query 1>; do
  echo "=== focusLabel=$label ===" && \
  curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
    "$VM_METRICS_URL/api/v1/status/tsdb?topN=20&focusLabel=$label" | \
    jq --arg l "$label" '{label: $l, topValues: .data.seriesCountByFocusLabelValue}'
done
```

`seriesCountByFocusLabelValue` returns label values sorted by series count — use the value names to detect problematic patterns.

**Query 3 — High-cardinality label-value pairs:**

```bash
curl -s ${VM_AUTH_HEADER:+-H} ${VM_AUTH_HEADER:+"$VM_AUTH_HEADER"} \
  "$VM_METRICS_URL/api/v1/status/tsdb?topN=50" | \
  jq '.data.seriesCountByLabelValuePair'
```

Shows which specific `label=value` pairs contribute the most series.

**Pattern detection** — classify label values from focusLabel samples:

| Pattern | Regex hint | Indicates |
|---------|-----------|-----------|
| UUIDs | `[0-9a-f]{8}-[0-9a-f]{4}-` | Request/session/trace IDs as labels |
| IP addresses | `\d+\.\d+\.\d+\.\d+` | Per-client or per-pod IP tracking |
| Long strings (>50 chars) | length check | Error messages, SQL, stack traces |
| SQL keywords | `SELECT\|INSERT\|UPDATE\|DELETE\|FROM\|WHERE` | Query text stored as label |
| URL paths with IDs | `/api/.*/[0-9a-f]+` | Unsanitized HTTP paths |
| Timestamps | epoch or ISO8601 | Time values as labels (unbounded) |
| Stack traces | `at .*\.(java\|go\|py):` | Error details as labels |

**Return**: Table of labels sorted by unique value count, with detected pattern, sample values from focusLabel, and series impact.

---

### Phase 2: Analysis

After all subagents return, compile and classify findings. This is the analytical core — apply
judgment, not mechanical filtering.

#### Category 1: Unused Metrics (Quick Wins)

Cross-reference metric usage stats with TSDB series counts:

- **Drop candidates**: `queryRequestsCount=0`, not in any alert/recording rule, >100 series
- **Verify candidates**: `queryRequestsCount=0` but referenced in rules — check if rule is still needed
- **Low-priority**: `queryRequestsCount≤5` with few series — not worth the config churn

Sort by series count descending — the biggest unused metrics are the biggest wins.

#### Category 2: High-Cardinality Labels

Labels with excessive unique values that drive series explosion:

| Label pattern | Assessment | Typical remedy |
|--------------|------------|----------------|
| `user_id`, `customer_id`, `account_id` | Should NEVER be metric labels — belongs in logs/traces | Drop label |
| `request_id`, `session_id`, `trace_id`, `span_id` | Correlation IDs — never metric labels | Drop label |
| `error`, `error_