安全评估 / NAVIN RESEARCH
Use LLM-as-a-Judge Carefully
Design rubrics, randomize ordering, calibrate against humans, and keep deterministic checks first.
Design rubrics, randomize ordering, calibrate against humans, and keep deterministic checks first.
简要回答: Model judges are useful for nuanced qualities but can be biased by style, length, position, or self-preference. Give a concrete rubric and evidence, hide irrelevant identity, randomize pair order, and validate judge agreement against expert labels.
实施步骤
- Define dimensions and anchored score examples.
- Use deterministic graders before subjective judging.
- Run pairwise order swaps and repeated judgments.
- Escalate low-agreement or high-risk cases to humans.
验证清单
- Measure agreement and bias by answer length and model source.
- Keep a private calibration set.
- Version judge model, prompt, and rubric.
常见错误
- One vague 'which is better?' prompt.
- Using the candidate model as its only judge.
- Reporting judge scores as objective truth.
实际决策
先交付能够通过上述检查的最小设计。记录测试中使用的模型、提示词、工具、数据集与运行时版本，让后续变化可以被比较而不是被猜测。依赖供应商的限制、模型名称、价格或预览功能之前，请重新检查链接文档；这些细节变化往往快于本文的工程原则。
