推理 / NAVIN RESEARCH
Build a Small Reasoning Evaluation Dataset That Matters
Create a compact, versioned eval set from real failures before chasing broad benchmark scores.
Create a compact, versioned eval set from real failures before chasing broad benchmark scores.
简要回答: Begin with 30-100 representative cases tied to actual user value. Include normal, boundary, ambiguous, and adversarial inputs. Store expected properties and graders, not only one brittle reference sentence.
实施步骤
- Sample production-like tasks after removing sensitive data.
- Label task type, difficulty, required evidence, and failure severity.
- Use deterministic graders wherever possible and rubric graders for subjective qualities.
- Freeze a regression set while rotating a private holdout set.
验证清单
- Review grader disagreement manually.
- Track pass rate by task slice, not only one average.
- Run the baseline and candidate under identical settings.
常见错误
- Writing evals from model capabilities instead of user tasks.
- Leaking all test cases into prompts or examples.
- Changing prompts, models, and graders simultaneously.
实际决策
先交付能够通过上述检查的最小设计。记录测试中使用的模型、提示词、工具、数据集与运行时版本，让后续变化可以被比较而不是被猜测。依赖供应商的限制、模型名称、价格或预览功能之前，请重新检查链接文档；这些细节变化往往快于本文的工程原则。
