推理服务 / NAVIN RESEARCH
KV Cache Planning for LLM Inference
Estimate memory from context, concurrency, model architecture, precision, and reuse before promising capacity.
Estimate memory from context, concurrency, model architecture, precision, and reuse before promising capacity.
简要回答: Weights are only part of serving memory. The key-value cache grows with active sequence length and concurrent requests, so long contexts can sharply reduce capacity. Measure with the exact runtime and scheduling policy rather than relying on a simple model-size estimate.
实施步骤
- Define input and output length distributions.
- Measure per-request cache use for the selected model and precision.
- Reserve memory for weights, runtime workspace, fragmentation, and safety margin.
- Test prefix reuse, paging, or eviction behavior if supported.
验证清单
- Load-test mixed short and long requests.
- Observe rejection, preemption, and tail latency.
- Verify isolation when cache reuse spans requests.
常见错误
- Sizing only for model weights.
- Advertising maximum context at maximum concurrency.
- Ignoring memory fragmentation and runtime overhead.
实际决策
先交付能够通过上述检查的最小设计。记录测试中使用的模型、提示词、工具、数据集与运行时版本，让后续变化可以被比较而不是被猜测。依赖供应商的限制、模型名称、价格或预览功能之前，请重新检查链接文档；这些细节变化往往快于本文的工程原则。
