Evaluating AI responses.

What does an evaluator check?

An evaluator checks more than whether a response feels good. A reviewer may need to check whether the answer follows explicit instructions, stays factually grounded, meets the user's need and communicates clearly. The rubric defines the task, and your preference counts only where the rubric allows it.

For instance, a fluent answer can still fail if it ignores a required format, invents a source or skips a key constraint. A plain answer can outperform it when it gives the user exactly what they requested.

A repeatable evaluation process

Use the same sequence every time. Consistency helps prevent irrelevant writing style from overriding important criteria.

• Read the task twice. Extract what the user explicitly requested, including formatting rules and prohibited content.
• Identify the critical conditions. Which requirement would make an otherwise polished answer unusable?
• Check each response independently. Do not pick a winner based on a first impression.
• Compare on rubric criteria. Distinguish major problems from stylistic preferences.
• Write concise evidence. Quote or paraphrase the exact failure and explain its impact.
• Record what works. Note the parts the response gets right, with the same kind of evidence you use for errors.

Example reasoning

The prompt says “Give exactly three safety tips in simple English.” Response A gives three accurate, clear tips. Response B gives five well-written tips. B may look richer, but it breaks the requested limit. The rubric decides how much that costs.

Comments another reviewer can verify

An evaluation comment should make the decision auditable. “A is better” provides no evidence. “A follows the three-item limit, and B lists five items despite the explicit instruction” explains the result.

A good comment names the criterion, the observed behavior and the user impact. For example, “Instruction following. Response B adds two extra items, so it misses the required length.” Avoid inflating every minor difference into a critical failure.

Strengths count too

Many beginners hunt only for flaws. Training teams use ratings, comparisons and comments to decide which answers a model should give more often, so what a response does well matters as much as what it gets wrong. If reviewers record only errors, the data shows what to avoid and says little about what a good answer looks like. Strong behaviour then gets less reinforcement.

Name the strengths as precisely as the errors. “Response A follows the three-item limit, uses plain words for a beginner and cites the source it was given” helps more than “A is good”. If your project asks you to report only errors, follow its rules.

A practice question

I wrote these questions for practice, and none come from a platform assessment. Think through your answer before you open the suggested approach.

Your preparation checklist

Use this checklist before a skills assessment. Your browser stores your progress, and nothing reaches this site.