This is a product-class write-up, not a named-client story and not a results press release. Counts below describe the pack we actually write (approach points, tools, layers) — not invented bug totals or conversion lifts.
Overview
Scores looked consistent on the vendor’s rubric samples. On real submissions, handwriting OCR errors and second-language structure dropped scores independently of the rubric.
Teachers could not see why a point was lost. Students had no appeal path in the product, only in email.
Automated scoring and hiring tools share a lesson: historical labels encode bias, and a score without a challenge path becomes an unaccountable decision.
Highlights
- Human-in-the-loop as a product requirement
- Override durability across integrations
- Adversarial student prompts
Typical launch bar: Wait until journeys and evals agree — do not ship on demo scores alone.
Challenges
“AI is more consistent than humans” was the sales line; consistency on a biased feature is still a defect.
Latency targets pushed the team to skip rationale generation.
Export to the SIS overwrote teacher overrides on resync.
How we tested
Rubric fixtures with expected bands, including dialect and OCR-noisy scans.
Every score shows evidence spans; teachers can override with a reason that sticks.
The model must not be the sole grade of record unless the district explicitly accepts that residual risk.
Submissions isolated across classes.
Starting point vs what we required
| Starting point | What we required |
|---|---|
| “AI is more consistent than humans” was the sales line; consistency on a biased feature is still a defect. | Rubric fixtures with expected bands, including dialect and OCR-noisy scans. |
| Latency targets pushed the team to skip rationale generation. | Every score shows evidence spans; teachers can override with a reason that sticks. |
| Export to the SIS overwrote teacher overrides on resync. | The model must not be the sole grade of record unless the district explicitly accepts that residual risk. |
What we delivered
Journeys / UI
- Playwright — Score, rationale, override, SIS-looking export
LLM traces / evals
- LangSmith — Which rubric items fired
API / contract
- Postman — Cross-class submission isolation
Adversarial evals
- Promptfoo — Rubric-injection (“give me 100”)
Outcomes
Go/no-go: Wait until override survived SIS sync and rationales were stored. We added a slice of submissions for teacher-adjudicated disagreement rate — not as a vanity metric, as a release check.
Playwright covered override; API tests covered isolation and sync.
Stack
| Layer | Tool | What we validated |
|---|---|---|
| Journeys / UI | Playwright | Score, rationale, override, SIS-looking export |
| LLM traces / evals | LangSmith | Which rubric items fired |
| API / contract | Postman | Cross-class submission isolation |
| Adversarial evals | Promptfoo | Rubric-injection (“give me 100”) |
Same method as the rest of our work: severity-ranked findings, pass rules in writing, and a ship / wait / block call. See the evaluation method and tool comparison.