P95 Labs — inference failure analysis for vLLM, Triton, and DCGM on Kubernetes

Read-only inference failure analysis · vLLM · Triton · DCGM Exporter

TTFT p95 — capacity-queued capture

warning

31,999.999999999993ms

target ≤ 1,000 ms — exceeded by 30,999.999999999993 ms

In plain terms: the server took 32 seconds to produce a first token against a one-second target, because requests were queueing behind a scheduler limited to two concurrent sequences.

Source: vllm:time_to_first_token_seconds
Runtime · Cluster: vLLM 0.28.0 · kind, Kubernetes 1.31
Rule: IFA-LAT-002

A signal you did not measure is not zero.

A runtime exposing no GPU metric is not a GPU idling at 0%. A server that has served nothing has no p95, not a p95 of zero. ifa is built to keep that distinction everywhere it reports a number — including on this page.

View the validation ledger Join the design-partner program View source

Try it

Three ways in, ordered by how little you need ready.

A default helm install now resolves — the image is public and a release is tagged. Start with the one that asks the least of you.

01 — The demo

A terminal. No cluster, no GPU, no inference server.

git clone https://github.com/p95labs/ifa && cd ifa
make demo

Seven simulated workloads serving vLLM-format Prometheus exposition over a real socket, scraped by the real collector, evaluated by the real rule engine. Nothing about the output is pre-canned — it produces roughly fifteen findings.

02 — Point it at a server you run

A vLLM server you already have. Nothing to install.

go build -o bin/ifa ./cmd/ifa
bin/ifa check http://your-vllm:8000/metrics -runtime vllm -model your/model

Reports which metrics ifa needs, which are present, and which are missing. It is the fastest way to find out whether this is useful to you, and it is read-only against an endpoint you already have.

03 — Install into a cluster

A Kubernetes cluster and Helm.

helm install ifa deploy/helm/ifa \
  --namespace ifa --create-namespace \
  --set config.kubernetes.namespace=inference

The chart is not published to a Helm repository yet, so this installs from a clone — there is no helm repo add. The published image is ghcr.io/p95labs/ifa, tagged 0.1.0-alpha.1 with no v prefix, even though the git tag has one.

The validation ledger

What is validated, and what is not.

Rows that say not validated carry the same visual weight as rows that say validated. That is the point — this is a record, not a pitch.

Component	Status	Detail
vLLM adapter	Validated	Validated against live vLLM 0.28.0, CPU backend — GPU-backed not yet validated. internal/runtime/vllm
Kubernetes path	Validated	Validated end to end in a real cluster in CI (Kubernetes 1.31). scripts/kind-integration.sh
Rule engine	Validated	Validated in-cluster at shipped defaults, including suppression. internal/recommender
Triton adapter	Validated	Validated against live Triton 25.12, CPU backend — GPU-backed not yet validated. internal/runtime/triton/testdata
DCGM adapter	Not validated	Implemented and unit-tested — not run on real hardware. internal/runtime/dcgm
High KV-cache pressure	Not validated	Highest live reading 2.86% — high-KV rules synthetic only. internal/runtime/vllm/testdata
Counter-rate and queue-growth rules	Not validated	Static fixture, no deltas — not exercised. internal/recommender
TimescaleDB history	Not validated	Implemented, bounded, non-blocking — no integration test. internal/storage/timescale

Both the vLLM and Triton adapters have been validated against live CPU-backed servers. DCGM has not. If you run it, ifa check <url> produces the report that would close it.

Stage

Solo-built, pre-users, alpha. No customers, no traction, no team. An alpha release, v0.1.0-alpha.1, is published, which makes the stage datable, not further along. The design-partner solicitation below is genuine — it is an invitation, not evidence of demand. No "trusted by," no logo wall, no invented metrics.

Rule taxonomy

19 rules across 7 families.

Nineteen rules is a real taxonomy, not a marketing number — presented here as one. Every rule reports evidence: the metric, the observed value, the threshold, and the comparison.

Family	Covers	Rules
IFA-CAP	Capacity and queueing internal/rules/capacity	4
IFA-ERR	Errors internal/rules/errors	2
IFA-KV	KV cache internal/rules/kv	2
IFA-SCL	Scaling internal/rules/scaling	2
IFA-LAT	Latency internal/rules/latency	4
IFA-OBS	Telemetry health internal/rules/observability	2
IFA-EFF	Efficiency internal/rules/efficiency	3
Total		19

The pair worth surfacing

A deep queue with a saturated GPU and a deep queue with an idle GPU have opposite fixes, and a dashboard cannot tell you which one you have. That distinction is the product.

IFA-CAP-001

Requests queueing while the GPU is saturated

internal/rules/capacity

IFA-CAP-002

Requests queueing while the GPU is idle

internal/rules/capacity

IFA-LAT-001

Time to first token dominated by queueing

internal/rules/latency

IFA-LAT-002

Time to first token dominated by prefill

internal/rules/latency

IFA-KV-001

KV cache exhausted: requests are being preempted

internal/rules/kv

Live validation

Two adapters, two real backends, real captures.

Both the vLLM and Triton adapters have been run against live servers and parsed real /metrics payloads. Six verbatim captures across the two runtimes are committed to the repository as regression fixtures.

vLLM 0.28.0 — CPU backend

Model facebook/opt-125m, official ARM64 CPU image. The CLI parsed the real /metrics endpoint: 51 KB payload, 122 metric families, 0 unparseable lines, 13 of 13 required metrics found. Three verbatim captures are committed to the repository as regression fixtures.

Capture	Running	Waiting	Waiting (capacity)	KV cache	TTFT p95
Idle	0	0	0	0%	39.9 ms
Under load	10	0	0	2.86%	—
Capacity-queued	2	12	12	0.46%	32,000 ms

vLLM reports KV utilisation as a 0–1 fraction — its own HELP text says "1 means 100 percent usage" — and the adapter converts it. That was checked against the live payload, because a fraction read as a percentage means the KV rules can never fire, silently, forever.

Triton 25.12 — CPU backend

Image nvcr.io/nvidia/tritonserver:25.12-pyt-python-py3, Python backend with a minimal echo model, KIND_CPU instance group. Three verbatim captures are committed to the repository: idle, loaded, and one with summary latencies enabled.

Capture	GPU utilization	Compute latency
Idle	—	—
Loaded	—	—
Summary latencies enabled	—	4.117 ms

Triton exposes a failure counter, nv_inference_request_failure, with a reason label — vLLM does not, which is why ifa deliberately reports no error rate for the vLLM adapter. That asymmetry is a real design decision driven by what each runtime actually publishes, not a template applied to both.

A CPU-only Triton server publishes no nv_gpu_* metrics at all, which makes these captures a real test of the site's central claim: an absent signal is reported as unmeasured, not as zero. The vLLM captures could not test that — vLLM on CPU still reports a KV cache.

Kubernetes validation

A real kind cluster, Kubernetes 1.31, node image pinned by tag and digest. The full path, end to end:

Helm install → ServiceAccount and RBAC → informer-based discovery → in-cluster Service networking → Prometheus scrape → vLLM parse → telemetry store → sustained rule evaluation → suppression → REST API response

Six assertions, zero semantic threshold overrides — the shipped defaults are active, including the 1,000 ms TTFT target and the 50% queue-share threshold. The only overrides are collection interval and sustain window, both purely to keep CI under four minutes.

Actual API response from the cluster run

{
  "code": "IFA-LAT-002",
  "severity": "warning",
  "window_seconds": 10,
  "evidence": [
    { "metric": "ttft_p95_ms", "source": "vllm:time_to_first_token_seconds",
      "observed": 31999.999999999993, "threshold": 1000, "comparison": ">", "unit": "ms" },
    { "metric": "queue_time_share_of_ttft",
      "observed": 44.531250000000014, "threshold": 50, "comparison": "<", "unit": "percent" }
  ]
}

Suppression, not duplication

The test also asserts that IFA-LAT-003 is absent — the specific diagnosis suppresses the generic symptom, which is behaviour a rule engine either has or fakes.

An unrounded float

31999.999999999993 is a real float from a real histogram interpolation, not a rounded marketing number. It is more convincing than any claim.

What broke, and why

A real model name broke the watcher. Here is the account.

The Kubernetes integration test was added to cover a path that had only ever been tested with a fake client. On its first run against a real API server, it failed — and the failure was not in the test.

internal/kubewatch

The watcher read the model name from a Kubernetes label, inference.io/model. Kubernetes label values cannot contain /. Every real model name is a Hugging Face repo id:

meta-llama/Llama-3.1-8B-Instruct
mistralai/Mistral-7B-v0.3
facebook/opt-125m

So the documented convention would have been rejected by the API server for every production model, and the project's own chart output demonstrated the broken form.

The unit tests passed because client-go's fake clientset does not validate label values. Only a real API server does. The fix moved model identity to an annotation, which accepts arbitrary values; runtime stayed a label, because vllm and triton are valid label values and selecting on runtime is useful. It shipped as its own change before the integration test merged, so the test covers the corrected convention rather than encoding a workaround.

Before — rejected by a real API server

# invalid: label values cannot contain '/'
metadata:
  labels:
    inference.io/model: meta-llama/Llama-3.1-8B-Instruct

After — model identity moved to an annotation

metadata:
  annotations:
    inference.io/model: meta-llama/Llama-3.1-8B-Instruct
  labels:
    inference.io/runtime: vllm

Not written as a humble-brag, and not as "we take quality seriously" — this is a short account of what broke, why the existing tests allowed it, and what changed. A project that publishes its own defects is more trustworthy than one that claims to have none.

Design partner program

Running vLLM or Triton on Kubernetes?

I'm looking for a small number of teams to run ifa read-only against real inference workloads. The next milestone is GPU-backed vLLM and live Triton — a design partner with either is the most useful conversation I can have right now.

vLLMTritonDCGM Exporter

Read the design-partner page Email hello@p95labs.com