Q: How can I efficiently sample production traces for review?
Q: How can I efficiently sample production traces for review?
Claude Code uses clustering methods to select examples from each cluster for review, combining active learning techniques to update fault classification and find similar cases or different faults by monitoring the annotations. file. The review method needs to be experimentally optimized based on the target and retain random samples to discover fault patterns not described by the current signal.
There are many ways to sample production traces for review. Here are some common methods.
| Method | What it does | Main limitation |
|---|---|---|
| Random | Selects traces with equal probability. | A small batch can miss rare cases. |
| Clustering | Groups traces by similar content and selects examples from each group. | The result depends on the features and clustering choices. |
| Data analysis | Reviews extreme values such as latency or tool count. | An extreme value may have nothing to do with quality. |
| Classification | Uses an evaluator or another model to flag likely failures. | It favors problems the classifier already knows how to find. |
| Feedback | Selects traces with negative user feedback. | It misses problems that users do not report. |
The table above orders sampling methods from the most exploratory to the most targeted. When you’re starting out, you should optimize for exploration of the data. As you learn more, you can start to lean more heavily on signals to select traces. The proper mix of methods depends on your goals and requires experimentation.
Keep some random traces in every batch. This gives you a chance to find failure modes that your current signals do not describe.
How do I measure rare failure modes?
Use targeted sampling to find rare failures. Search for signals that correlate with the failure, such as a specific tool sequence, unusually long traces, retries, or a known input pattern. Review the targeted batch to collect examples and improve the failure definition.
This flashcard from our evals flashcards series visualizes these methods.
Use labels to choose the next traces
We can borrow a technique from machine learning called active learning to sample production traces. In active learning, a system asks a person to label the data points that would be most useful for its next update.
In Shreya Shankar’s walkthrough, Claude Code clusters traces and chooses examples from each cluster for review. A monitor command watches annotations.json for new labels. When a label arrives, the agent updates a failure taxonomy and looks for similar cases or different failures.
In the above video, active learning is used in the context of error analysis to find new cases to review. However, this approach can be used anywhere in the workflow where you are annotating data.
This article is part of our AI Evals FAQ, a collection of common questions (and answers) about LLM evaluation. View all FAQs or return to the homepage.
Source: Hamel Husain · hamel.dev