Stefan Huber
Linguist · AI evaluation · Tool builder

Careful judgment, made repeatable.

I’m Stefan Huber, a linguist working in AI evaluation. I write guidelines and rubrics, design test items, transcribe and judge speech in three languages, audit reviewers’ work, and build small tools that make review faster and more accurate. Everything here uses public or invented material.

Start exploring Download résumé (PDF)
Portrait of Stefan Huber, smiling, in a navy jacket and light blue shirt

Four kinds of careful work

All work →

Each one asks the same question: when two careful people judge the same thing, do they reach the same answer? Start wherever it matters most to you.

  1. 01

    Define quality

    Guidelines and rubrics with explicit rules and edge cases, plus the audits and calibration that keep reviewers consistent.

    Guideline to rubric →
  2. 02

    Test precisely

    Assessment items where exactly one answer survives scrutiny, with an explanation for every choice.

    Test items →
  3. 03

    Listen closely

    Verbatim transcription and speech judgment in German, Swedish, and English, including dialect.

    Dialect audio →
  4. 04

    Build tools

    Small tools for reviewers, built with AI assistance. Next up: Crosstalk, my own app for reviewing who speaks when.

    Crosstalk preview →
330published test items, each with exactly one defensible answer
3+years in AI evaluation, as auditor, reviewer, and lead
3working languages: German, Swedish, English
16years of university teaching

Selected work

All work →
Case study
Start here

From Vague Guideline to Consistent Rubric

I turn vague guidelines into criteria that two careful reviewers apply the same way.

Raters often get one line of guidance. I took such a line, wrote seven responses that split careful raters, and built the guideline, rubric, edge-case map, and calibration examples that give each response exactly one score.

Guideline writing, rubric design, edge-case analysis, rater calibration, multilingual evaluation (German, Swedish)

Read the case study →
Before: the guidance

Rate how well the response follows the user’s instructions, from 1 (poor) to 5 (excellent).

After: the rubric, in short
  • 5 Fully follows
  • 4 Follows, minor flaw
  • 3 Partly follows
  • 2 Mostly misses
  • 1 Does not follow
Covers of the ten-book series Reading and Writing Practice Questions for the New Digital SAT, by Dr. Stefan Huber
Case study
Published

Writing Test Items With Exactly One Defensible Answer

A test item is only useful if exactly one answer survives close scrutiny. Two sample items, each with an explanation for every answer choice, and the method behind a ten-book series of 330 items.

Assessment design, distractor analysis, rationale writing, quality auditing, rubric design

Read the case study → The books on Amazon ↗
Case study
With audio

Transcribing Dialect Speech as It Is Actually Spoken

Four recordings of my own voice in Upper Swabian, Standard German, and English, and a verbatim transcript that shows where my dialect drifts toward the standard instead of hiding it.

Verbatim transcription, dialect annotation, speech recording, German, Swabian, English

Listen and read →

About

More about me →

I hold a Ph.D. in Linguistics from Lund University (2002) and taught German, linguistics, and Swedish at the University of South Florida for 16 years. Since 2023 I have worked in AI evaluation: as a lead auditor for German on multilingual model evaluation, and as a freelance reviewer and auditor on annotation and model-evaluation projects.

Location
Roseville, California. Remote work.
Languages
German (native), Swedish (near-native), English (native-level)