Quality
review.

The difference between reviewing and rating

In some workflows, an evaluator gives a score to an AI output. A quality reviewer checks whether the evaluator did that correctly. The reviewer may inspect the evidence, the scoring rubric, the severity assigned and the evaluator's explanation.

A reviewer should distinguish disagreement in taste from a demonstrable scoring error. If two reasonable interpretations fit the guidance, escalation or calibration may be better than pretending the rule is unambiguous.

Severity is about impact

Not every imperfection deserves the same label. When your workflow uses severity levels, evaluate how strongly an issue affects the user's goal and whether it breaks a core requirement.

• Minor. A noticeable imperfection with little effect on usefulness.
• Moderate. A real problem that reduces reliability or needs correction.
• Major. A core task failure, a significant inconsistency or a serious factual defect.
• Critical. The most serious failures under the governing rubric.

These are general principles. Each project defines its own scale, and its definitions come first.

From opinion to evidence

A useful quality-review note should include where the problem occurs, what is wrong and why the rubric treats it that way. For an image evaluation, identify the relevant region and visual inconsistency. For text, quote the sentence or instruction that matters. For audio, a timestamp may be essential.

For example, “The evaluator marked no issues, but the generated object has a duplicated handle in the upper-right area. This visible structural defect falls under the artifact-consistency criterion.”

A quality-review scenario

I wrote these exercises based on common evaluation principles. They reveal no proprietary tasks.

Your preparation checklist

Review these points before handling QA and reviewer assessments. Progress stays on your device.