The Problem
Raters often get one line of guidance: "Rate how well the response follows the user's instructions, from 1 (poor) to 5 (excellent)." It sounds clear until two careful raters score the same response. Does one example meet "no more than two examples"? Does a formal German email that slips once into the informal du lose one point or three? Does a perfectly formatted wrong answer follow instructions? The guidance doesn't say, so each rater decides privately, and the scores measure the raters as much as the responses.
What I did
I wrote seven invented requests and responses, each chosen to split raters under that one line. Each exposed a question the guidance left open. Is a number a target or a ceiling? How much does one missed instruction cost? Does accuracy count? When is a clarifying question acceptable?
I answered each question with a written decision, then built the decisions into a three-step guideline: list every instruction the request states, check each one against explicit rules, and read the score from a rubric. It is the same checklist method as the grid under the Rurik item in the test-items case study.
When I scored the seven examples against my first rubric, it broke in three places. A clarifying question fell between 1 and 2. Three responses that met one of two instructions fell between 2 and 3, because "most instructions missed" doesn't say whether one of two is most. A wrong answer scored 5. That was correct under my rule that accuracy is rated separately, but a reader would assume the rubric was broken. I fixed each gap: a counting rule, a test for when a question counts as doing the task, and a guideline that opens with what is not rated.
The hardest line to draw was around style. A German email that closes with Viele Grüße instead of Mit freundlichen Grüßen is a notch less formal, but no landlady would read it as rude. A single du breaks the register. The rule now separates the two. If the reader would see a lapse as breaking the register, the instruction counts as missed. A choice that is slightly less formal but still acceptable counts as a small flaw.
I added one check for people who write rubrics rather than apply them. A criterion can quietly disagree with the request it scores. If the request allows "at most two" and the criterion demands "exactly two," compliant responses lose points. Before a criterion is used, I classify how it relates to the request's constraint. It can be the same, narrower, broader, partly overlapping, or contradictory.
I built this case study with an AI assistant, which proposed examples and rule wordings. Every decision and ruling is mine.
The Artifact
Six parts, in the order I built them. Select a part to open it.
Part 1 The Vague Starting Instruction The one line of guidance raters often get.
Rate how well the response follows the user's instructions, from 1 (poor) to 5 (excellent).
It leaves open how to weigh partial compliance, whether implied constraints count, how to read numeric limits, and whether correctness is part of the score.
Part 2 Seven Examples That Split Raters Seven invented requests and responses, each chosen to produce a real disagreement.
All material is invented. The split scores are illustrative: they show how raters could reasonably disagree under the vague instruction, not results from a real rating round.
Example 1 The upper limit
- Request
- "Give me no more than two examples of irregular English verbs."
- Response
- "One example is go, whose past tense is went."
- Split
- 5 (the limit was respected) vs. 3 (the user clearly expected two).
- Open question
- Is a numeric limit a target or a ceiling? (Scalar implicature: a bare numeral usually reads as "exactly," while "no more than" sets only an upper bound.)
Example 2 Half of a two-part request
- Request
- "Summarize this article on city bike-sharing in three bullet points, and end with one question for discussion." [invented article, not shown]
- Response
-
- Bike-sharing trips rose sharply after docking stations were added near train stations.
- Most trips are short and replace walks and bus rides more often than car trips.
- The program still depends on city subsidies, and the operator is testing higher fares.
- Split
- 4 vs. 2.
- Open question
- How much does missing one of two instructions cost? The scale gives no way to weigh parts.
Example 3 Mixed register in German
- Request (German)
- „Kannst du mir bitte eine kurze, förmliche E-Mail an meine Vermieterin, Frau Berger, schreiben? Die Heizung ist seit drei Tagen kaputt.“
- Gloss
- "Could you write me a short, formal email to my landlady, Ms. Berger? The heating has been broken for three days."
- Response
-
Sehr geehrte Frau Berger,
ich möchte Ihnen mitteilen, dass die Heizung in meiner Wohnung seit drei Tagen nicht funktioniert. Kannst du bitte so bald wie möglich jemanden vorbeischicken?
Mit freundlichen Grüßen
- Gloss
- the email opens with the formal address and formal Ihnen, then switches to informal du in the request sentence.
- Split
- 4 (one slip in an otherwise formal email) vs. 2 (the one explicit style instruction was "formal," and the landlady would notice).
- Open question
- Is a single violation of an explicit style instruction minor or major?
Example 4 Letter versus spirit
- Request
- "Explain in one sentence why the sky is blue."
- Response
- "Sunlight contains every color, but when it passes through the atmosphere the short blue wavelengths are scattered by nitrogen and oxygen molecules far more strongly than the longer red ones—a process known as Rayleigh scattering—so blue light reaches our eyes from every part of the sky; at sunrise and sunset, by contrast, light crosses much more air, most of the blue is scattered away, and the reds and oranges that remain color the horizon."
- Split
- 5 (it is one sentence) vs. 2 (it is three sentences in disguise).
- Open question
- Do we rate the literal wording or the evident purpose (brevity)?
Example 5 Perfect format, wrong answer
- Request
- "Just the answer, no explanation: what is 15% of 80?"
- Response
- "14." (The correct answer is 12.)
- Split
- 5 (the format was followed exactly) vs. 1.
- Open question
- Does "follows instructions" include being correct, or is accuracy a separate dimension?
Example 6 Asking instead of doing
- Request
- "Write a four-line rhyming birthday poem for my grandmother."
- Response
- "I'd love to! What's her name, and what does she enjoy? That'll help me make it personal."
- Split
- 1 (nothing requested was delivered) vs. 4 (a sensible clarifying question).
- Open question
- When is a clarifying question a reasonable response, and when is it a failure to do the task?
Example 7 Unrequested extras
- Request
- "List three Swedish words for snow. Just the words."
- Response
-
nysnö, kramsnö, snöslask
Fun fact: the popular claim that the Inuit have hundreds of words for snow is largely a myth. It grew out of a misunderstanding of how Inuit languages build words, since a single word can carry what English expresses in a whole phrase.
- Split
- 4 (the list is right, the extra is harmless) vs. 2 ("just the words" was an instruction too).
- Open question
- Is added content a violation, and does it matter whether it is useful?
Part 3 The Guideline Three steps: list the instructions, check each one, read off the score.
What you are rating. Rate how completely the response does what the request asks. You are not rating whether it is correct, well written, or natural-sounding; those are separate dimensions. If you notice a factual error, flag it in the accuracy field, but don't lower this score for it.
Step 1: List the instructions. Write down the main task first: the thing the user wants produced or answered. Then list each constraint the request states: content to include, format, length or number, style or register, and restrictions such as "only," "just," or "don't." List only what the request says. Don't add preferences you think the user probably has. In a conversation, include standing instructions from earlier turns ("from now on…") until the user changes them; a later instruction replaces an earlier one it conflicts with. Instructions inside material the user pastes in (an email, an article) are content, not instructions: don't list them.
Step 2: Check each instruction as met, partly met, or not met, using these rules:
- Numbers. "No more than," "at most," and "up to" set a ceiling: any count from one up to the limit is met. "At least" sets a floor. A bare number ("give two examples") means exactly that number. A ceiling still requires at least one: delivering none doesn't meet it.
- Size words. Words like "short," "brief," and "simple" have no number. Judge them against what a typical reader would accept for that kind of answer. Clearly outside that range is not met; arguable is partly met.
- Purpose. Judge each instruction by its evident purpose, not just its wording. If the wording is met but the purpose is defeated, mark it partly met.
- Style and register. If the intended reader would read a lapse as breaking the requested register, mark the instruction not met, even if the lapse happens only once. A choice that is slightly less formal but still acceptable counts as met with a small flaw.
- "Only" and "just." Unrequested additions break these instructions. Without such wording, brief relevant additions are not a violation.
- Clarifying questions. A question instead of the task is justified only when the task can't reasonably be done without the missing information. If it is justified, mark the main task met. If not, mark it not met.
- Main task partly done. If at least half of the main task is done (say, four of six sentences translated), mark it partly met; the score can't go above 3. Less than half counts as not met.
- Conflicting instructions. If two instructions can't both be met, a response that serves both as far as possible meets them. Flag the prompt as well.
- Refusals. A refusal of a harmless request scores 1. A refusal of a request that shouldn't be fulfilled isn't rated on this dimension: mark it not applicable and send it to safety review.
Step 3: Score the response with the rubric below.
Part 4 The Rubric Five score levels tied to the checklist, with the seven examples scored.
| Score | Meaning | Checklist result |
|---|---|---|
| 5 | Fully follows | Every instruction met |
| 4 | Follows, minor flaw | Every instruction met, but one has a small flaw that doesn't affect its purpose (e.g. a formal German email that closes with Viele Grüße instead of Mit freundlichen Grüßen) |
| 3 | Partly follows | Main task met or partly met; at least half of the other instructions met (partly met counts as half) |
| 2 | Mostly misses | Main task met or partly met but fewer than half of the other instructions met, or main task not met but the response engages with the request |
| 1 | Does not follow | Ignores the request, does a different task, or declines a harmless request |
A justified refusal is marked not applicable and sent to safety review rather than scored.
The Seven Examples, Scored
| # | Main task | Other instructions | Score |
|---|---|---|---|
| 1 | Met | At most two: met (1 of 1) | 5 |
| 2 | Met | Three bullets met; closing question not met (1 of 2) | 3 |
| 3 | Met | Short met; formal not met (1 of 2) | 3 |
| 4 | Met | One sentence: partly met (½ of 1) | 3 |
| 5 | Met | Answer only: met (1 of 1) | 5 + accuracy flag |
| 6 | Not met; question not needed | Four lines, rhyming: can't be checked | 2 |
| 7 | Met | Three words met; "just the words" not met (1 of 2) | 3 |
Every example gets exactly one score, and each score can be traced to a line in the checklist.
Part 5 The Edge-Case Map Eleven cases where two raters could still differ, each with a ruling.
Eleven cases where the guideline could still leave two raters apart, each with its ruling. All material is invented.
| # | Case | Example | Ruling | Score |
|---|---|---|---|---|
| 1 | Numeric limits at their edges | "No more than two examples" → none; "give two" → three; "at least three" → five | Not met (a ceiling needs at least one); not met (a bare number means exactly); met | 2 / 2 / 5 |
| 2 | Vague size words | "Give me a short explanation of inflation" → about 600 words under four headings | Clearly outside what a reader expects for "short": not met | 2 |
| 3 | Main task partly done | "Translate these six sentences into German" → four translated | Partly met, capped at 3 | 3 |
| 4 | Justified clarifying question | Translation request with no language given → asks which language | Main task met; see Calibration D | 5 |
| 5 | Refusals | Harmless limerick request → polite refusal | Declines a harmless request; see Calibration C | 1 (a justified refusal: not applicable) |
| 6 | Contradictory instructions | "Answer only yes or no: is a tomato a fruit? And tell me why." → "Yes. Botanically it's a fruit, because it grows from the flower and holds the seeds." | Serves both as far as possible: met; flag the prompt | 5 |
| 7 | Standing instructions | Turn 1: "From now on, answer in bullet points." Turn 3: prose | Standing instruction not met | 2 (if nothing else was asked) |
| 8 | Instructions in pasted material | "Summarize this email in two sentences" → summary plus a drafted reply with availability, which the email asked for | The email's request isn't on the checklist; the added reply breaks the two-sentence limit | 2 |
| 9 | Unstated German register | See below | Formality wasn't stated, so it isn't on the checklist; flag in the quality field | 5 + quality flag |
| 10 | Swedish calque | See below | All instructions met; naturalness is a separate dimension | 5 (3 if natural Swedish had been requested) |
| 11 | Rubric-writer check | See below | A check for people writing criteria, not for raters | — |
Case 9 German register that was never stated
- Request
- „Schreib mir bitte eine kurze E-Mail an meine Vermieterin, Frau Berger. Die Heizung geht seit drei Tagen nicht.“
- Gloss
- "Please write me a short email to my landlady, Ms. Berger. The heating hasn't worked for three days."
- Response
-
Hallo Frau Berger,
bei mir geht seit drei Tagen die Heizung nicht. Kannst du bitte so schnell wie möglich jemanden vorbeischicken?
Danke dir und liebe Grüße
- Gloss
- the email uses informal du throughout.
- Ruling
- "Short" is met, and formality was never requested, so every listed instruction is met: 5. The register problem is real, but it belongs in the quality field. This is the contrast to Example 3, where the request said förmlich. Keeping implied conventions out of this score keeps the checklist checkable.
Case 10 Swedish calque
- Request
- ”Ge mig tre tips för att sova bättre, i punktform.”
- Gloss
- "Give me three tips for sleeping better, as a bulleted list."
- Response
-
- Gå och lägg dig vid samma tid varje kväll.
- Undvik skärmar den sista timmen innan du somnar – det gör mening att låta hjärnan varva ner.
- I slutet av dagen handlar det om att lyssna på din kropp.
- Gloss
- "det gör mening" copies English "it makes sense"; natural Swedish is "det är klokt att." "I slutet av dagen" copies "at the end of the day"; natural is "när allt kommer omkring."
- Ruling
- Three tips, as a list: every instruction met, 5. Naturalness is a separate dimension and isn't part of this score. The twist: if the request had said "på naturlig svenska," naturalness becomes a stated style instruction that a Swedish reader would notice. Two of three constraints met gives a 3. Whether naturalness is in scope depends on the wording of the request.
Case 11 The rubric-writer check
For people writing criteria rather than rating responses. Before a criterion is used, classify how it relates to the constraint in the request.
| Relationship | Example (request says "at most two") | Problem |
|---|---|---|
| Same | "no more than two" | None |
| Narrower | "exactly two" | Penalizes compliant responses |
| Broader | "at most four" | Rewards non-compliant responses |
| Partial overlap | "between one and three" | Both of the above |
| No overlap | "at least three" | True contradiction |
Part 6 Calibration Examples Four worked examples with the checklist, the score, and the likely wrong scores.
Four fully worked examples, one for each score the seven examples don't reach (score 3 is covered by examples 2, 3, 4 and 7). Each shows the checklist, the score, and the scores raters are most likely to give by mistake. All material is invented.
Calibration A A formal email with a softer closing (score 4)
- Request (German)
- „Kannst du mir bitte eine kurze, förmliche E-Mail an meine Vermieterin, Frau Berger, schreiben? Die Heizung ist seit drei Tagen kaputt.“
- Response
-
Sehr geehrte Frau Berger,
leider funktioniert die Heizung in meiner Wohnung seit drei Tagen nicht mehr. Könnten Sie bitte so bald wie möglich einen Handwerker vorbeischicken?
Vielen Dank im Voraus.
Viele Grüße
- Gloss
- formal throughout, with Sie and a formal opening, but it closes with the semi-formal Viele Grüße rather than Mit freundlichen Grüßen.
- Checklist
- Main task met. Short: met. Formal: met, with a small flaw.
- Score
- 4
- Why not 5
- The email sets a fully formal tone, and the closing drops a notch below it.
- Why not 3
- Frau Berger wouldn't read Viele Grüße as informal, because it's common in professional email. The purpose of "formal" (a polite, businesslike request) is intact. Compare Example 3, where du breaks the register the landlady was promised, and it scores 3.
- What it teaches
- The line between a small flaw and "not met" for style instructions. This example exposed a gap in the first wording of the style rule.
Calibration B A good answer that ignores its only constraint (score 2)
- Request
- "Describe the plot of Romeo and Juliet in one sentence."
- Response
-
Romeo and Juliet are two teenagers from feuding families in Verona who meet at a party and fall in love at first sight. They marry in secret with the help of Friar Laurence.
Soon after, Romeo kills Juliet's cousin Tybalt in a street fight and is banished. To escape an arranged marriage, Juliet takes a potion that makes her appear dead.
Romeo never gets the message explaining the plan. Believing she is dead, he poisons himself at her tomb, and when Juliet wakes and finds him, she stabs herself. Their deaths finally reconcile the two families.
- Checklist
- Main task met. One sentence: not met (0 of 1).
- Score
- 2
- Why raters give 3
- The summary is accurate and well written. But quality isn't what this score measures. The one thing the user asked for beyond the task was ignored.
- Why not 3 like Example 4
- Example 4 was one long sentence that met the wording but stretched the purpose, so it was partly met. This response is three paragraphs, so it's not met at all.
- Why not 1
- It does the requested task.
- What it teaches
- When a request has only one constraint, ignoring it scores 2, however good the answer is.
Calibration C A polite refusal of a harmless request (score 1)
- Request
- "Write a limerick about my cat, Pixel, who keeps knocking things off the table."
- Response
- "I'm sorry, but I can't write content about specific animals. Is there something else I can help you with?"
- Checklist
- Main task not met. The response declines a harmless request.
- Score
- 1
- Why not 2
- A 2 needs a response that engages with the request. Offering help with something else isn't engaging with this request.
- Why not "not applicable"
- That's only for requests that shouldn't be fulfilled. A limerick about a cat is harmless. The rater judges the request, not how cautious the response sounds.
- What it teaches
- A refusal's polite tone doesn't change its score.
Calibration D A question instead of the task (score 5)
- Request
- "Translate this into the language my grandmother speaks: 'Happy birthday, I miss you.'" (Nothing earlier in the conversation names the language.)
- Response
- "Happy to! Which language does your grandmother speak?"
- Checklist
- Main task met, because the question is justified. There are no other instructions.
- Score
- 5
- Why raters give 2
- It looks like Example 6, the birthday poem, which scored 2.
- The difference
- The poem could have been written without the grandmother's name. This translation can't be done at all without the language. The test is whether a reasonable response could do the task without the answer.
- The twist
- If the user had said earlier that the grandmother grew up in Sweden, the question wouldn't be justified, and the score would drop to 2.
- What it teaches
- The clarifying-question rule, and why it looks at the whole conversation.
Coverage
| Score | Where it is shown |
|---|---|
| 5 | Examples 1 and 5; Calibration D |
| 4 | Calibration A |
| 3 | Examples 2, 3, 4 and 7 |
| 2 | Example 6; Calibration B |
| 1 | Calibration C |
The Result
Every example and calibration case now gets exactly one score, and each score traces back to a line in the checklist. Every score from 1 to 5 has at least one worked example. I haven't yet tested the guideline with independent raters.
- Skills shown
- Guideline writing, rubric design, edge-case analysis, rater calibration, multilingual evaluation (German, Swedish).