When AI gives a different answer, check what changed

Suppose you ask AI to draft a short update from the same approved notes. One version says a project is “nearly ready.” Another says it is “waiting for approval.” The phrasing is different, but so is the status a leader might take away.

The first question is not which paragraph sounds better. It is whether the answers differ on something that changes the work: a fact, a commitment, a decision, or who needs to act next.

AI can give different answers to the same question. OpenAI says that the same question may produce different responses across separate queries, and that generated answers can be incorrect or misleading. That is useful context, but it does not tell you how much a particular difference matters. For that, start with the job the answer is meant to do. OpenAI’s guidance on whether ChatGPT tells the truth recommends checking important information against reliable sources.

Separate wording changes from work changes

Some variation is just presentation. “The draft is ready for review” and “The draft can go to review” may mean the same thing if the notes support both. A shorter sentence, a different heading, or a more formal tone can be a harmless change.

Other differences can change what someone does. “The draft is ready for review” is not the same as “The draft was approved.” A request due Friday is not the same as one due next Friday. “Jordan will send it” names an owner; “it will be sent” does not. A proposed next step is not a recorded commitment.

Before comparing answers, name the details that carry meaning for this task. Depending on the work, those might be:

• What happened, and when
• The current status and the evidence for it
• A number, unit, or time period
• Whether a statement is confirmed, proposed, or still unknown
• The person responsible for the next step
• Any promise, deadline, or approval

This list is illustrative. A weekly progress update needs different checks from a customer response or a summary of a policy. Use details that your team has already agreed matter. If the team has not agreed what a status means, an AI comparison will not settle that definition.

Compare the answers against the same notes

Here is a hypothetical example. A manager provides these approved notes:

The draft is complete. Priya asked for a legal review on Tuesday. Legal has not replied. The customer delivery date remains unconfirmed.

One AI answer says:

The draft is complete and awaiting legal review. The customer delivery date is still unconfirmed.

A second says:

Legal approved the draft Tuesday, and the team expects to deliver to the customer next week.

The second answer is fluent, but it adds an approval and a delivery expectation that are absent from the notes. The useful review is not a vote on which answer feels more polished. It is a comparison to the supplied evidence.

Try a small table for the details that could affect a decision:

| Detail to check | What the notes support | Answer A | Answer B | | --------------- | ------------------------------------ | --------------- | ---------------- | | Draft status | Complete | Complete | Complete | | Legal review | Requested Tuesday; no reply recorded | Awaiting review | Approved Tuesday | | Customer date | Unconfirmed | Unconfirmed | Next week |

The table makes the consequential differences visible. It does not prove the notes are complete or current. If the approved notes are out of date, both answers can faithfully reflect old information and still mislead the reader.

For a simple task, you do not need to generate several versions every time. This comparison is most useful when you have a reason to inspect variation: a regenerated answer changed its conclusion, the same kind of work will recur, or the output may guide a customer-facing or operational decision. If you are checking just one answer, compare it with the source records and your criteria. A second AI answer is another draft, not an independent witness.

Make the review specific enough to repeat

A vague instruction such as “check whether these are good” leaves the reviewer to invent the standard each time. Write down a few checks that match the task instead.

For a project update, a review list might say:

1. Dates and quantities match the source notes.
2. The status does not claim approval or completion unless the notes say so.
3. A future action is labeled as proposed unless an owner committed to it.
4. Unconfirmed items remain unconfirmed.
5. The next step and its owner are included only when the notes identify them.

You can ask AI to help organize a comparison, but keep the criteria visible to the person reviewing it. For example:

Compare these two drafts against the notes. List any differences that change a date, number, status, approval, commitment, owner, or next action. For each difference, quote the relevant wording from both drafts and the supporting note. Mark anything the notes do not settle as “unverified.” Do not choose a preferred answer or add missing facts.

That request can produce a useful review aid. It can also miss a difference or report a false one. Check the quoted wording and the original notes yourself, particularly for dates, amounts, customer promises, and approvals. If the answer depends on a system of record, check that system rather than relying on a pasted summary alone.

Decide what deserves follow-up

When you find a difference, classify it before editing:

• Presentation: The meaning is unchanged. Keep whichever wording is clearer for the reader.
• Source mismatch: One version conflicts with the notes or record. Correct the draft to match the verified source, or update the source if it is wrong and you have authority to do so.
• Missing evidence: Neither answer is supported enough to settle the question. Preserve the uncertainty and ask the process owner for the missing fact.
• Changed judgment: The answers interpret the same facts differently. Decide whether the task needs a clear rule, a named reviewer, or a human decision before anything is sent or recorded.

A difference in judgment may be the important one. If two answers classify the same case differently, the team may need to clarify its criteria. That does not mean the model should decide which criteria to adopt. It means the difference has revealed a question for the person who owns the work.

There is a fair reason to avoid this extra comparison on routine, low-stakes work. It takes time, and checking two drafts can cost more than writing one yourself. If the output is only a private brainstorm, small wording changes may not matter. Use the review where the cost of a changed fact or action justifies it, and keep the method proportionate to the task.

Test the check on one real task

Choose a task with approved information and a reviewer who knows what the answer is supposed to preserve. Do not start with private or sensitive material unless your organization has approved that tool and use. A made-up example is enough to practice the method.

1. Write down three to five details that would change the next decision if they were wrong.
2. Provide the same task and source material for each answer you compare. If the model, instructions, tools, or source documents differ, record that too; the comparison is no longer isolating answer variation.
3. Put the answers side by side and mark only differences that affect those details.
4. Check each marked difference against the original source, not against whichever answer sounds more confident.
5. Decide whether to use, revise, verify, or hold the draft. Record what remains unknown.

For recurring work, keep the checklist with the procedure and revisit it if the task, inputs, or AI system changes. NIST’s AI Risk Management Framework says accuracy measures should use realistic test sets that represent expected conditions, and that performance should be evaluated in conditions similar to the intended setting. It also notes that ongoing testing or monitoring can help assess whether a deployed system is performing as intended. That guidance is for managing AI systems, not a guarantee that a brief comparison will validate a workflow. NIST’s discussion of AI risk and trustworthiness supports the broader point: test against the work and conditions where people will rely on the output.

A handful of examples can show whether your review list catches obvious changes. It cannot establish that every future answer will be accurate. Keep the test conditions and limitations in view, and watch for errors in actual use if the task becomes recurring.

Let the evidence, not the smoother answer, decide

A confident sentence is not stronger evidence than a cautious one. Nor does a changed answer automatically mean the AI failed. The key is whether a decision-critical detail changed, and what the underlying record supports.

If both drafts preserve the facts and commitments, choose the clearer one or keep your existing version. If they differ on an important detail, verify it at the source and leave it unresolved when the source does not answer it. The person responsible for the work still owns the decision.

Sources: OpenAI’s guidance on whether ChatGPT tells the truth and NIST’s AI Risks and Trustworthiness.