安全评估 / NAVIN RESEARCH
Evaluate RAG Before Tuning the Generator
Separate retrieval recall, ranking, context quality, answer support, and citation correctness.
Separate retrieval recall, ranking, context quality, answer support, and citation correctness.
简要回答: Diagnose retrieval-augmented generation in stages. First confirm the needed evidence exists and is retrieved. Then test ranking and context assembly. Only after that should you evaluate whether the model answers faithfully and cites the supporting passage.
实施步骤
- Create questions with labeled source passages and unanswerable cases.
- Measure retrieval recall at several cutoffs.
- Inspect reranking and duplicate chunks.
- Score answer correctness, support, and citation precision separately.
验证清单
- Run ablations with oracle evidence.
- Test stale and conflicting documents.
- Verify access control before retrieval results enter context.
常见错误
- Changing embeddings and prompts simultaneously.
- Using answer similarity as a retrieval metric.
- Forcing an answer when evidence is absent.
实际决策
先交付能够通过上述检查的最小设计。记录测试中使用的模型、提示词、工具、数据集与运行时版本，让后续变化可以被比较而不是被猜测。依赖供应商的限制、模型名称、价格或预览功能之前，请重新检查链接文档；这些细节变化往往快于本文的工程原则。
