本文共 2300 字，大约阅读时间需要 7 分钟。

回顾与总结

• 选择一个能够反馈未来期望的开发集和测试集。而它们可能不同于训练集
• Choose dev and test sets from a distribution that reflects what data you expect to get in the future and want to do well on. This may not be the same as your training data’s distribution. 
• 尽量在同一份数据中选择开发集和测试集。
• Choose dev and test sets from the same distribution if possible. 
• 选择一个单一的评价指标指导你的团队优化算法。如果你关心大多数的性能，那么考虑把它们“结合”成一个指标（如用平均值、方差等）或者定义一个优化指标和多个满足指标（参见第9章）
• Choose a single-number evaluation metric for your team to optimize. If there’re multiple goals that you care about, consider combining them into a single formula (such as averaging multiple error metrics) or defining satisficing and optimizing metrics. 
• 机器学习是一个高度循环的过程：尝试不同的点子，找到最适合的她。
• Machine learning is a highly iterative process: You may try many dozens of ideas before finding one that you’re satisfied with. 
• 拥有开发/测试集和一个单一数值指标可以使你更加快速的评估各种算法，提高开发效率。
• Having dev/test sets and a single-number evaluation metric helps you quickly evaluate algorithms, and therefore iterate faster. 
• 开启一个项目的时候，首先应该做的，就是先建立开发/测试集和评价指标，至少在一周内就得完成，当然，如果项目成熟的话，花多点时间也不成问题。
• When starting out on a brand new application, try to establish dev/test sets and a metric quickly, say in less than a week. It might be okay to take longer on mature applications. 
• 在你有很多数据的情况下，传统的方法：即将数据按7：3分为开发集和测试集就不再试用了。这时候开发/测试集可以低于30%。
• The old heuristic of a 70%/30% train/test split does not apply for problems where you have a lot of data; the dev and test sets can be much less than 30% of the data. 
• 你的开发集必须要大到足够发现微小的细节的程度，但未必要非常大，这样才能指导你的团队按正确的方向前进。你的测试集要大到能够准确的评估系统在实际中的表现。
• Your dev set should be large enough to detect meaningful changes in the accuracy of your algorithm, but not necessarily much larger. Your test set should be big enough to give you a confident estimate of the final performance of your system. 
• 如果你的开发集和评价指标不在起作用——即无法再继续指引你的团队了，那么建议你马上修改它们：
  • 如果你过度拟合了开发集，那么就添加更多数据给开发集
  • 若果你实际关心的数据类型与测试集和开发集的数据类型有所不同，建议你更新它们
  • 若果说你的评价指标所评价的不在是你关注的重点，建议你修改指标
• If ever your dev set and metric are no longer pointing your team in the right direction, quickly change them: (i) If you had overfit the dev set, get more dev set data. (ii) If the actual distribution you care about is different from the dev/test set distribution, get new dev/test set data. (iii) If your metric is no longer measuring what is most important to you, change the metric. 

转载地址：https://jacobzhuo.blog.csdn.net/article/details/81556995 如侵犯您的版权，请留言回复原文章的地址，我们会给您删除此文章，给您带来不便请您谅解！