推理服务 / NAVIN RESEARCH
Benchmark an LLM Inference Server Correctly
Report workload, hardware, concurrency, prompt and output lengths, latency percentiles, throughput, and quality together.
Report workload, hardware, concurrency, prompt and output lengths, latency percentiles, throughput, and quality together.
简要回答: A useful benchmark reproduces the intended workload. Warm the server, control request distributions, report time to first token and inter-token latency, and include p95 or p99 under sustained concurrency. Throughput without quality and workload details is not comparable.
实施步骤
- Pin model artifact, runtime version, flags, drivers, and hardware.
- Generate a request distribution matching production.
- Warm caches, then run long enough to reach steady state.
- Report errors, queueing, latency percentiles, and tokens per second.
验证清单
- Repeat runs and publish variance.
- Verify outputs are valid and quality has not collapsed.
- Test overload and admission-control behavior.
常见错误
- Comparing different prompt or output lengths.
- Using only one request at a time.
- Omitting failed and rejected requests.
实际决策
先交付能够通过上述检查的最小设计。记录测试中使用的模型、提示词、工具、数据集与运行时版本，让后续变化可以被比较而不是被猜测。依赖供应商的限制、模型名称、价格或预览功能之前，请重新检查链接文档；这些细节变化往往快于本文的工程原则。
