P95 Labs is a read-only diagnostics layer for Kubernetes inference workloads. It correlates cluster state, Prometheus telemetry, queue depth, latency, and GPU signals to explain why p95 is breaking and what your team should inspect first.
p95
78ms
p50
19ms
GPU util
41%
Recommendation: queue depth rising while GPU utilisation remains low — inspect batching and concurrency limits before scaling.
The problem
Latency, queueing, autoscaling, and GPU behaviour are tightly coupled — but they're measured in separate tools that don't talk to each other.
Graphs tell you that p95 spiked. They rarely tell you which knob to inspect first, or why.
Idle-looking GPUs and slow responses at the same time point to a bottleneck somewhere other than raw compute.
By the time replicas spin up, requests have queued, timed out, and p95 has already broken for real users.
A small share of long-context requests can dominate tail latency without ever showing up in averages.
Without correlated signals, batch size and concurrency limits get set by intuition and rarely revisited.
The product
P95 Labs runs outside the inference request path. It reads operational signals and surfaces recommendations without handling user traffic.
system architecture
read-only boundaryYour Kubernetes cluster · data plane
Inference workloads
vLLM · Triton · TGI · KServe
Prometheus
latency · queue · GPU exporters
P95 Labs · control plane
Collector + TimescaleDB
high-cardinality time series
Correlation engine
transparent rule set
Ranked recommendations
explained + scoped
Recommendations are surfaced to your team. P95 Labs never writes to the cluster — a human reviews and applies every change.
Automatically maps inference deployments, replicas, and runtimes across scoped namespaces.
Pulls the latency, queue depth, and GPU signals you already export — no new agents in the request path.
Time-series storage built for high-cardinality inference metrics and historical correlation.
Transparent, inspectable rules — every recommendation explains the signals behind it.
Install with a Helm chart, drive everything from a CLI that fits existing platform workflows.
P95 Labs never changes your cluster on its own. It recommends; your team approves and acts.
Current alpha
P95 Labs now has a working Kubernetes-native alpha with workload discovery, Prometheus collection, TimescaleDB telemetry storage, workload-aware recommendations, CLI access, Docker packaging, Helm deployment, documentation, network policy support, and air-gapped deployment preparation. The next validation target is real inference runtime testing, starting with vLLM.
client-go pod watcher
metrics ingestion
telemetry storage
dual backend
workload-aware rules
telemetry · recs · workloads
cluster deployment
containerised
restricted networking
export + load
model + RBAC
evaluation-ready
next milestone — real inference runtime validation, starting with vLLM.
Phase 2.5
P95 Labs is being prepared for cautious design partner evaluation. The alpha now includes architecture notes, security model documentation, RBAC documentation, data collection boundaries, an operations runbook, a design partner pilot guide, air-gapped deployment notes, NetworkPolicy support, and bundle scripts for restricted environments.
Documented trust boundaries and posture.
What is and isn't collected, in writing.
Exact read-only verbs and scopes.
Install, operate, and uninstall steps.
Bundle scripts and NetworkPolicy template.
A structured path for evaluation.
Recommendations
Each recommendation includes severity, the related metric, a plain-language explanation, and a concrete next step to inspect.
// Examples shown are illustrative and depend on available telemetry.
P95-QUEUE-001severity: highquery
queue_depth > 0 and gpu_util < 0.5Requests are backing up before they reach the GPU. Compute is available, so the limit is in batching, concurrency, or admission — not capacity.
inspect max concurrent requests and batch settings before adding replicas or GPUs.
P95-TAIL-004severity: highquery
histogram_quantile(0.95, …) ↑ with rate(requests[1m])Tail latency rose in lockstep with a traffic burst while p50 stayed flat — a sign that a small fraction of requests are absorbing the spike.
inspect burst handling and per-request token limits; check for long-context outliers.
P95-HPA-007severity: mediumquery
hpa_desired_replicas - hpa_ready_replicas > 0Desired replicas climbed several intervals before ready replicas caught up. The queue was already draining slowly during the gap.
inspect scale-up thresholds and warm-pool / preStop timing to shorten reaction time.
P95-KV-012severity: mediumquery
kv_cache_utilization > 0.9 and seq_len_p99 ↑KV cache utilisation is approaching limits during long-context windows, which can trigger eviction and recompute under load.
inspect max sequence length and KV cache allocation relative to model footprint.
P95-POD-019severity: highquery
increase(kube_pod_container_status_restarts_total[30m]) > 0 and p95 ↑The workload restarted during the same window where tail latency rose. Requests may have queued while replicas became unavailable.
inspect container restarts, readiness probes, rollout events, and replica availability.
Security
Production inference is sensitive. The alpha role is scoped to read the workload and telemetry resources it needs. Write, update, patch, and delete permissions are not included for workload mutation.
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: p95-readonly rules: - apiGroups: ["", "apps"] resources: [pods, deployments, nodes] verbs: [get, list, watch] # no create / update / patch / delete
P95 Labs observes. It does not mutate workloads or configuration on its own.
No prompts, request bodies, or model outputs are collected by default — only operational telemetry.
Ships with tightly scoped RBAC requesting only the read permissions it needs.
Deploy into the namespaces you choose. Nothing outside that scope is touched.
Recommendations are surfaced for your team. A human always approves before action.
Remove the Helm release and RBAC cleanly, with no residual controllers left behind.
Data boundaries
The alpha focuses on operational telemetry, Kubernetes workload metadata, runtime metrics, and recommendation outputs.
// Not collected by default in the current alpha.
Design partner program
We're looking for a small number of teams to validate P95 Labs against real inference workloads in read-only mode.
A good fit is a team that has seen p95 latency spikes, queue buildup, unclear GPU utilisation, autoscaling lag, or manual batching and concurrency tuning.
Contact: hello@p95labs.com
What you get
What we look for