Label a small set before AI sorts the rest

If you’re asking AI to sort incoming work, don’t start by handing it the whole queue. First, make a small labeled sample: a set of examples where a person has already chosen the right category using the rules your team follows.

That sample gives you something concrete to compare. NIST says accuracy checks should use realistic test sets that represent expected conditions, and it cautions that performance can vary outside the conditions tested. A few examples won’t establish broad reliability, but they can expose a mismatch before you build a workflow around it. NIST’s guidance on validity and reliability explains why test sets should fit the intended use.

Make a small reference set

Choose one recurring, low-consequence sorting task, such as routing internal requests to “billing,” “access,” or “other.” Use examples you’re authorized to share, and remove personal or confidential details the task doesn’t need.

Before asking AI, label a varied handful yourself. Include straightforward cases and a few that are easy to confuse. Use the team’s existing definitions. If no one can agree on what a category means, pause there: the sorting rule needs clarification before AI can apply it consistently.

Then give the same examples and definitions to AI. Ask for one category per item, a short reason tied to the text, and an “unclear” label when the information doesn’t support a choice. Don’t show it your labels first if you want to see how it applies the rules independently.

Illustrative example: A team labels 12 anonymized requests. It marks one as “access” because the employee cannot sign in, and another as “billing” because an invoice amount is wrong. AI labels both “billing.” The disagreement points to a useful question: does the team’s definition distinguish account access from charges? This example is hypothetical, not a tested result.

Compare the disagreements, not just the total

Check each AI category against the human label. Record which cases match, which don’t, and whether the mistakes share a pattern. A single overall match count can hide the fact that one important category is repeatedly missed.

If a mismatch comes from an unclear rule, fix the rule and try a fresh set. If AI invents a reason that isn’t in the request, or keeps forcing uncertain cases into a category, keep those cases with a person. Don’t quietly change the human labels to make the AI look better.

The sample should resemble the work you expect it to handle. If all your examples are clean and routine, the comparison says little about short, incomplete, or unusual requests. NIST also emphasizes that testing conditions should reflect the context where a system will be used. Its measurement guidance discusses representativeness and the risks of evaluating on data that doesn’t match the real setting.

A small sample is a first check, not a guarantee. It cannot prove future accuracy, cover every edge case, or justify automatic routing when a wrong category could affect a customer, payment, access, or staff decision. Keep a person responsible for reviewing consequential cases, and repeat the check when the request mix or category rules change.

The practical decision is simple: if the sample reveals a pattern you can correct and verify, try a limited, reviewable next step. If the labels are still inconsistent, keep the sorting human until the rules are clearer.