应该记录那些并非模型错误的难题
Q: Should I record problems that aren’t the model’s fault?
应记录任何让产品变得不实用的因素，包括与模型无关的缺失或损坏功能。优先处理问题，而非分析错误原因。例如，客服未提供订单追踪链接，即使AI无此功能也应记录。识别和优先处理问题是构建评估的基础，某些问题可能是工程或设计缺陷。专注于问题而非根本原因分析，有助于在查看更多数据时编写更高质量的标注。
Yes. When reviewing interactions, write down anything that makes the product less useful. This includes missing or broken features that have nothing to do with the model. Additionally, don’t focus on why the error occurred, as that should only come after you prioritize which issues to fix.
For example, a support agent might tell a customer that an order has shipped without providing a tracking link. Even if your AI doesn’t have the ability to fetch a tracking link, record that problem. A prerequisite to building evals is to identify and prioritize which issues to fix through error analysis. Some of these issues may end up being engineering or design issues that don’t need an automated evaluator, but they are still important to fix!
Lastly, we’ve found that deferring root-cause analysis and focusing on problems allows you to write higher-quality annotations while looking at more data.
This article is part of our AI Evals FAQ, a collection of common questions (and answers) about LLM evaluation. View all FAQs or return to the homepage.
来源: Hamel Husain · hamel.dev