Performance Engineering on Eva Winterschon https://winterschon.gitlab.io/tags/performance-engineering/ Recent content in Performance Engineering on Eva Winterschon Hugo -- gohugo.io en Some copyright notice - [my license](https://example.com/license) Thu, 18 Jun 2026 23:39:50 +0000https://winterschon.gitlab.io/logo.svg Inference Sector KPI SLO https://winterschon.gitlab.io/en/posts/inference-sector-kpi-slo/ Thu, 18 Jun 2026 23:39:50 +0000 https://winterschon.gitlab.io/en/posts/inference-sector-kpi-slo/ Why should anyone care about KPIs and SLOs?

Generally, unless one gets paid to deal with OBS/Tel and SLA conformance.. ehh. Except that the modern world operates within the bounds of a diverse and often not-fascinating array of performance indicators and services’ operating metrics.

So.. Here are two for today.

  • KPI == Key Performance Indicator
  • SLO == Service Level Objective

Minor Backstory - Today

While aggregating directory content from several workstations into a unified NFS remote mount,(can’t simply have one workstation, right? right), another encounter with SLO defs appears upon the terminal.

This time it’s for LLM Inference customers and their typical requirements. Perhaps our standards repository would be helpful as an adjunct resource:

Inference Service KPIs by Sector

An extension of distributed cloud services architecture with heavy focus on baremetal with VMs and Containers throughout. Many ways to solve the problems, scale the infra, etc.

SLO Additions to the Classifier Repo

  • This file is a compact companion to the observability + telemetry glossary.
  • Each threshold is a starting point and should be tightened to real-world workload SLOs after baselining standard operations.
Sector High-signal KPIs Default alert style
LLM Inference (GP-GPU Service Infra) TTFT, ITL/TPOT, E2E, QTS SLO-derived or baseline-relative; baseline-relative
LLM Inference (API Service Infra) P95 LAT, ERR, 429R, UST SLO-derived
LLM Inference (Network Hardware + Protocol Infra) OWD, PDV, LOSS, ECN baseline-relative; budget-relative
LLM Inference (Prompt Caching, Compute + Re-Compute) PCHR, CTR, TTFTR, RCR absolute for cache-eligible traffic
LLM Inference (Prompt Caching, Storage Infra) CHR, CLAT, COCC, EVR absolute; absolute for cache-eligible traffic
LLM Inference (Prompt Caching API + Load-Balancers) RCHR, LQ, URT, RTR absolute
LLM Training (bulk initial datasets) ITPS, DHR, UDR, CONT absolute; capacity-envelope
LLM Training (pre-training MoE) MFU, STEP, A2AS, EIR absolute starting point; baseline-relative
LLM Training (post-training MoE) TPS, MFU, RACC, RMAR absolute starting point; baseline-relative
HPC - High Frequency Trading - VM Clusters RDY, CSTP, NUMA MISS, T2D absolute starting point
HPC - High Frequency Trading - Baremetal W2W, JITR, CPM, MPKI baseline-relative; budget-relative
HPC - High Frequency Trading - CDN CHR, OOR, TTFB, OTTFB absolute
HPC - High Frequency Trading - Low-Latency Exec OWD, JIT, FLAT, OLAT baseline-relative; budget-relative
HPC - Dark Fiber Regional Network + Infra AVAIL, OWD, Pre-FEC BER, OSNRM SLO-derived; budget-relative
HPC - Quantitative Research + Machine Learning BTT, FLAT, TSS, FFL absolute starting point; baseline-relative
HPC - Big-Data & Multivariate Pattern Analysis JDUR, SKR, SPILL, CLAG SLA-derived; absolute
SLA + SLO Monitoring, Telemetry, Alerting Infra SLI-AV, SLI-LAT, EBR, UP SLO-derived

Global Policy Adjustments

  • baseline_window:
    • 7d median or p50/p95 baseline unless otherwise stated
  • slo_policy:
    • page: error budget burn rate > 14.4 over 1h and 5m, or > 6 over 6h and 30m
    • ticket: error budget burn rate > 1 over 3d and 6h
  • capacity_policy:
    • warn: sustained > 80% of validated steady-state capacity
    • critical: sustained > 90% of validated steady-state capacity or latency inflects above SLO
  • regression_policy:
    • warn: > 1.10x to 1.25x baseline depending on sector sensitivity
    • critical: > 1.20x to 1.50x baseline depending on sector sensitivity

Reference Considerations

  • Thresholds are starting operational thresholds, not universal laws.
  • Where a standards or vendor reference provides an acceptable range, that range is used.
  • Where no universal value exists, thresholds are either SLO-derived or baseline-relative (typically against a 7d median or a validated capacity envelope).
  • For latency-sensitive sectors, alert on percentile regressions and budget exhaustion, not averages.
]]> Performance Engineering - Evaluative Metrics https://winterschon.gitlab.io/en/posts/performance-engineering-evaluative-metrics/ Sat, 13 Jun 2026 15:43:01 +0000 https://winterschon.gitlab.io/en/posts/performance-engineering-evaluative-metrics/ LLM Inference
Sector Common metrics Less-known / specialist metrics
GP-GPU Service Infrastructure TTFT, ITL/TPOT, E2E request latency, TPS, RPS/QPS, Queue-Fill, KV/C metrics request_queue_time, request_prefill_time, request_decode_time, inflight time, SM occupancy, HBM/DRAM memory throughput
API Service Infrastructure p50/p95/p99 latency, streaming TTFB/TTFT, RPS/QPS, error rate, 429 rate, active downstream connections, backend latency x-envoy-upstream-service-time, upstream_rq_time, TR/Tw/Tc/Tr/TA timer split in HAProxy, retry rate, queue-slot saturation, circuit-breaker usage
Network Hardware + Protocol Infra RTT (round-trip time), one-way latency, jitter/PDV (packet delay variation), PPS/BPS, packet loss, retransmits ECN mark rate, PFC (priority flow control) pause events, microburst depth, CQE (completion queue entry) compression state, SymbolErrors, LinkRecoveries, PTP offset/path delay
Prompt Caching, Compute + Re-Compute cache hit rate, cached_tokens, prompt_tokens, generation_tokens, TTFT reduction, cache read/write counts partial-hit ratio, cache-miss ratio, prefix-cache block hit rate, predicted KV hit rate, per-request routing overhead
Prompt Caching, Storage Infra cache hit ratio, eviction rate, cache latency, TTL, GPU/CPU cache usage CPU-vs-GPU cache-hit split, priority-based eviction effect, TTL refresh behavior, KV-block fragmentation / block reuse
Prompt Caching, API + Load-Balancers per-route/model hit rate, active and pending requests, backend latency, request rate, retries, error rate cache-aware routing predicted KV hit rate, routing overhead, queue time before backend selection, circuit-breaker saturation, connection-slot pressure

Acronyms

  • TTFT == Time to First Token
  • ITL == Inter-Token Latency
  • TPOT == Time per Output Token
  • E2E == end-to-end
  • TKS == tokens/sec
  • RPS == requests/sec
  • QPS == queries/sec
  • KV/C == Key-Value Cache

LLM Dataset Training

Operations Common metrics Less-known / specialist metrics
Pre-loading datasets ingest throughput (docs/s, rows/s, bytes/s, tokens/s), tokenization throughput, DataLoader throughput, dedup ratio fuzzy-duplicate rate, semantic-duplicate rate, contamination rate/AUC, consumer lag, shuffle spill bytes
Pre-training + MoE tokens/s, step time, MFU (model FLOPs utilization), training loss/perplexity, all-to-all/dispatch time, GPU memory auxiliary load-balancing loss, router z-loss, expert capacity factor, token-drop rate, per-layer expert imbalance
Post-training + MoE step time, samples/s or tokens/s, train/val loss, reward/preference accuracy, KL divergence, MFU chosen vs rejected reward margin, chosen/rejected rewards, router aux/z loss during alignment, expert saturation under small-batch tuning

HPC + HFT Analytics Infrastructure

Sector Common metrics Less-known / specialist metrics
Virtual Machine Clusters p50/p99 service latency, jitter, CPU ready time, throughput, packet loss CPU co-stop, NUMA locality (numa_hit/numa_miss/local_node/other_node), vCPU scheduling contention, latency-sensitivity effectiveness
Baremetal Systems wire-to-wire latency, p99/p999 jitter, PPS/Mpps, cycles, instructions, LLC-load-misses, branch-misses, RX/TX drops NUMA locality, interrupt coalescence, CQE compression state, packet pacing, driver extended stats / ring stress
Content Delivery Networking cache hit ratio, TTFB, origin offload, request rate, error rate, egress bandwidth shield-layer hit ratio, origin_ttfb, child/parent cache status, regional cache-performance variance
Low-Latency Trade-Execution Networks one-way latency, RTT, jitter, packet loss, order latency, feed latency microburst depth, queue/buffer pressure, ECN mark rate, PFC pause events, path asymmetry, PTP offset
Dark Fiber Regional Network Infra one-way latency, RTT, availability, BER, pre-FEC BER, post-FEC BER OSNR/ESNR, Q-factor, CD (chromatic dispersion), PMD (polarization mode dispersion), FEC degrade indicators
Quantitative Research + Machine Learning backtest wall-clock runtime, feed latency, order latency, feature-serving latency, feature-ingestion throughput training-serving skew, feature inflight vs write-to-store success metrics, feature health/correctness monitoring
Data Analytics + Multivariate Analysis job duration, stage/task duration, throughput, end-to-end delay, records-consumed-rate, consumer lag, shuffle read/write spill bytes, skew via task-duration discrepancy or shuffle-read imbalance, straggler share, input-pipeline prefetch effectiveness

SLA + SLO Reliability, Telemetry, Alerting

  • Pre-Defined latency/error/throughput SLIs and error budgets require burn-rate alerting
  • Prometheus and Alertmanager define scrape and notification timing controls
  • OpenTelemetry defines histograms and exemplars
  • Apdex is a standard user-satisfaction score
  • Elastic APM measures application performance traces
Sector Common metrics Less-known / specialist metrics
SLA + SLO Monitoring, Telemetry, Alerting Infra availability SLI, latency SLI, error rate, throughput, Apdex, error-budget burn rate multi-window multi-burn-rate alerts, scrape_duration / scrape_timeout ratio, group_wait / group_interval / repeat_interval, histogram bucket design, exemplars

Shared Analytics of Interest

Metric Signal Typical domains
Queue time Separates saturation from raw compute/network slowness LLM serving, API gateways, load balancers, HFT
Prefill vs decode split Distinguishes prompt-processing bottlenecks from token-generation bottlenecks LLM GPU serving
Prefix/KV cache hit rate Direct proxy for avoidable recompute and TTFT improvement LLM serving, agentic systems
Auxiliary MoE loss and router z-loss Early warning for expert imbalance and routing instability MoE training
CPU ready / co-stop / NUMA miss Often the real cause of inconsistent latency in virtualized clusters VM-based HFT / HPC
Microburst depth and ECN/PFC behavior Reveals congestion that average bandwidth hides Low-latency Ethernet / RoCE fabrics
OSNR / CD / PMD / pre-FEC BER Core optical-health indicators long before full link failure Dark fiber / coherent optics
Spill bytes and consumer lag Early warning for data-path backpressure and skew Big-data pipelines
Burn rate and exemplars Better operational signal than raw alert count or average latency SLO / observability stacks
]]>