MentorAI Inter-Rater Reliability Report

Comparing human raters (Rachel, Janine, Nthato, Anna) with LLM Judge across 3 conversations

Generated: January 15, 2026

Conversations: Amara (ec742d69), Bailey (ddd7a2e3), Carlos (4426508c)

Criteria Evaluated: 31 (7 Critical, 24 Quality)

Executive Summary

Overall Human-LLM Agreement

76%

Updated with coaching judge results for Amara

Critical Criteria Agreement

95%+

Strong alignment on must-pass criteria

Domain F Agreement

58%

Conversational Quality criteria need revision

Criteria Flagged for Review

Unclear definitions or consistent disagreement

What's Working Well

Critical criteria (B-01, C-01, C-03, D-01, D-02): Near-perfect agreement across all raters
SBI Content Fidelity (Domain D): Consistently high agreement - criteria are well-defined
Session Setup (Domain A): Clear pass/fail determinations

Areas Needing Attention

Domain F (Conversational Quality): Highest disagreement; criteria need clearer definitions
LLM tends to be more lenient on articulation criteria (C-06, F-02) than human raters (all 4 humans stricter)
Context-dependent criteria (E-03, E-04) are hard to score when conditions aren't met

Agreement by Conversation

Amara Conversation

ID: ec742d69

18 Full Agreement (5/5) 7 Partial (4/5) 6 Split or Mixed

Strong agreement

Disagreement - review needed

Pass

Fail

N/A = condition not met

Tag	Criterion	Rachel	Janine	Nthato	Anna	LLM	Agreement
Domain A: Session Setup
A-01	Goal clarity	Pass	Pass	Pass	Pass	Pass	5/5
A-02	Phase signaling	Pass	Pass	Pass	Fail	Pass	4/5
A-03	Realistic scenario	Pass	Pass	Pass	Pass	Pass	5/5
Domain B: Modeling Quality
B-01	Shows, not tells	Pass	Pass	Pass	Pass	Pass	5/5
B-02	Thinking out loud	Pass	Pass	Pass	Pass	Pass	5/5
B-03	Visible decision-making	Pass	Pass	Pass	Pass	Fail	4/5
B-04	Self-checking	Fail	Fail	Fail	Fail	Fail	5/5 Fail
B-05	Heuristic offered	Pass	Pass	Pass	Pass	Pass	5/5
Domain C: Coaching Quality
C-01	Specific feedback	Pass	Pass	Pass	Pass	Pass	5/5
C-02	Actionable direction	Pass	Pass	Pass	Pass	Pass	5/5
C-03	Revision requested	Pass	Pass	Pass	Pass	Pass	5/5
C-04	Revision checked	Pass	Pass	Pass	Pass	Pass	5/5
C-05	Productive struggle	Pass	Pass	Pass	Pass	Pass	5/5
C-06	Elicits articulation	Fail	Fail	Pass	Fail	Pass	2/5 Split
C-07	Prompts reflection	Pass	Pass	Pass	Fail	Pass	4/5
Domain D: SBI Content Fidelity
D-01	Catches vague situations	Pass	Pass	Pass	Pass	Pass	5/5
D-02	Catches judgment leakage	Pass	Pass	Pass	Pass	Pass	5/5
D-03	Catches accusatory impact	Pass	Pass	Pass	Pass	Pass	5/5
D-04	Tests distinctions	Pass	Pass	Pass	Pass	Pass	5/5
D-05	Scaffolds the stuck	Pass	Pass	Pass	Pass	Pass	5/5
D-06	Reusable scaffold	Pass	Pass	Pass	Pass	Pass	5/5
Domain E: Adaptive Pacing
E-01	Checks before advancing	Pass	Pass	Pass	Pass	Pass	5/5
E-02	Fades support	Fail	Fail	Pass	Pass	Pass	3/5 Split
E-03	Adjusts to struggle	Pass	Pass	Pass	Fail	Fail	3/5
E-04	Protects productive struggle	Pass	N/A	Pass	N/A	Pass	3/3
Domain F: Conversational Quality
F-01	Varied turn structure	Fail	Pass	Pass	Fail	Pass	3/5
F-02	Genuine curiosity	Fail	Fail	Pass	Fail	Pass	2/5 Split
F-03	Room to breathe	Fail	Fail	—	Fail	Fail	4/4 Fail
F-04	Dwells on difficulty	Fail	Pass	—	Pass	Pass	3/4
F-05	Has a voice	Fail	Pass	Fail	Pass	Fail	2/5 Split
F-06	Questions over corrections	Pass	Pass	Pass	Pass	Pass	5/5

Bailey Conversation

ID: ddd7a2e3

17 Full Agreement (5/5) 8 Partial (4/5) 6 Split or Lower

Strong agreement

Disagreement - review needed

Pass

Fail

N/A = condition not met

Tag	Criterion	Rachel	Janine	Nthato	Anna	LLM	Agreement
Domain A: Session Setup
A-01	Goal clarity	Pass	Pass	Pass	Pass	Pass	5/5
A-02	Phase signaling	Pass	Pass	Pass	Pass	Pass	5/5
A-03	Realistic scenario	Pass	Pass	Pass	Pass	Pass	5/5
Domain B: Modeling Quality
B-01	Shows, not tells	Pass	Pass	Pass	Pass	Pass	5/5
B-02	Thinking out loud	Pass	Pass	Pass	Pass	Pass	5/5
B-03	Visible decision-making	Pass	Pass	Pass	Pass	Pass	5/5
B-04	Self-checking	Fail	—	Pass	Fail	Pass	2/5 Split
B-05	Heuristic offered	Pass	Pass	Pass	Pass	Pass	5/5
Domain C: Coaching Quality
C-01	Specific feedback	Pass	Pass	Pass	Pass	Pass	5/5
C-02	Actionable direction	Pass	Pass	Pass	Pass	Pass	5/5
C-03	Revision requested	Pass	Pass	Pass	Pass	Pass	5/5
C-04	Revision checked	Pass	Pass	Pass	Pass	Pass	5/5
C-05	Productive struggle	Pass	Pass	Pass	Fail	Pass	4/5
C-06	Elicits articulation	Fail	—	Pass	Fail	Pass	2/5 Split
C-07	Prompts reflection	Pass	Pass	Fail	Pass	Pass	4/5
Domain D: SBI Content Fidelity
D-01	Catches vague situations	Pass	Pass	Pass	Pass	Pass	5/5
D-02	Catches judgment leakage	Pass	Pass	Pass	Pass	Pass	5/5
D-03	Catches accusatory impact	Pass	Pass	Pass	Pass	N/A	4/4
D-04	Tests distinctions	Pass	Pass	Pass	Pass	Pass	5/5
D-05	Scaffolds the stuck	Pass	Pass	Pass	Pass	Pass	5/5
D-06	Reusable scaffold	Pass	Pass	Pass	Pass	Pass	5/5
Domain E: Adaptive Pacing
E-01	Checks before advancing	Pass	Pass	Pass	Pass	Pass	5/5
E-02	Fades support	Fail	Pass	Pass	Pass	Pass	4/5
E-03	Adjusts to struggle	Pass	N/A	Pass	Pass	Pass	4/4
E-04	Protects productive struggle	Pass	—	Pass	Pass	Pass	4/4
Domain F: Conversational Quality
F-01	Varied turn structure	Pass	Pass	Pass	Fail	Pass	4/5
F-02	Genuine curiosity	Fail	—	Pass	Fail	Pass	2/5 Split
F-03	Room to breathe	Fail	—	—	Fail	Pass	1/3 Split
F-04	Dwells on difficulty	Fail	N/A	—	Fail	Pass	1/3 Split
F-05	Has a voice	Fail	Pass	Fail	Fail	Fail	4/5 Fail
F-06	Questions over corrections	Pass	Pass	—	Pass	Pass	4/4

Carlos Conversation

ID: 4426508c

14 Full Agreement (5/5) 5 Partial (4/5) 12 Split or Lower

Note: Carlos is a "rusher" persona that pushed the mentor to skip steps. This stressed the interaction and resulted in more failures and scoring disagreements.

Strong agreement

Disagreement - review needed

Pass

Fail

N/A = condition not met

Tag	Criterion	Rachel	Janine	Nthato	Anna	LLM	Agreement
Domain A: Session Setup
A-01	Goal clarity	Pass	Pass	Pass	Pass	Pass	5/5
A-02	Phase signaling	Pass	Pass	Fail	Pass	Pass	4/5
A-03	Realistic scenario	Pass	Pass	Pass	Pass	Pass	5/5
Domain B: Modeling Quality
B-01	Shows, not tells	Pass	Pass	Pass	Pass	Pass	5/5
B-02	Thinking out loud	Fail	—	Fail	Fail	Fail	4/4 Fail
B-03	Visible decision-making	Fail	—	Fail	Fail	Fail	4/4 Fail
B-04	Self-checking	Fail	—	Fail	Fail	Fail	4/4 Fail
B-05	Heuristic offered	Pass	Pass	Pass	Pass	Pass	5/5
Domain C: Coaching Quality
C-01	Specific feedback	Pass	Pass	Pass	Pass	Pass	5/5
C-02	Actionable direction	Pass	Pass	Pass	Pass	Pass	5/5
C-03	Revision requested	Pass	Pass	Pass	Pass	Pass	5/5
C-04	Revision checked	Pass	Pass	Pass	Pass	Pass	5/5
C-05	Productive struggle	Fail	—	Pass	Fail	Pass	2/5 Split
C-06	Elicits articulation	Fail	—	Fail	Fail	Pass	3/4
C-07	Prompts reflection	Pass	Pass	Fail	Pass	Pass	4/5
Domain D: SBI Content Fidelity
D-01	Catches vague situations	Pass	Pass	Pass	Pass	Pass	5/5
D-02	Catches judgment leakage	Pass	Pass	Pass	Pass	Pass	5/5
D-03	Catches accusatory impact	Fail	—	Pass	Fail	N/A	1/4 Split
D-04	Tests distinctions	Pass	Pass	Pass	Pass	Pass	5/5
D-05	Scaffolds the stuck	Pass	Pass	Pass	Pass	Pass	5/5
D-06	Reusable scaffold	Pass	Pass	Pass	Pass	Pass	5/5
Domain E: Adaptive Pacing
E-01	Checks before advancing	Fail	—	Pass	Fail	Pass	2/5 Split
E-02	Fades support	Fail	—	Pass	Fail	Fail	3/4 Fail
E-03	Adjusts to struggle	Fail	N/A	Pass	Fail	Fail	2/4 Split
E-04	Protects productive struggle	Fail	N/A	Pass	Fail	Pass	2/4 Split
Domain F: Conversational Quality
F-01	Varied turn structure	Fail	—	Pass	Fail	Fail	3/4 Fail
F-02	Genuine curiosity	Fail	—	Pass	Fail	Fail	3/4 Fail
F-03	Room to breathe	Fail	—	—	Fail	Fail	3/3 Fail
F-04	Dwells on difficulty	Fail	—	—	Fail	Fail	3/3 Fail
F-05	Has a voice	Fail	—	Fail	Fail	Pass	3/4 Fail
F-06	Questions over corrections	Pass	Pass	Pass	Pass	Pass	5/5

Criteria Flagged for Review

Based on rater disagreements and feedback, the following criteria need attention before the LLM judge can be reliably calibrated.

F-03 Room to breathe Quality

Issue: Unclear what "letting an answer land" means in practice. Multiple raters marked "FC needs revision."

Nthato: "FC needs revision. I don't know what this means."

Recommendation: Define specific observable behaviors (e.g., "mentor's next turn is ≤10 words of acknowledgment before new content").
F-04 Dwells on difficulty Quality

Issue: "Hard" and "interesting" are subjective. Raters disagree on what counts as dwelling.

Nthato: "FC needs revision. What do 'hard' and 'interesting' mean here?"

Recommendation: Define as "mentor spends 2+ turns on a single concept when learner shows difficulty" with examples.
F-06 Questions over corrections Quality

Issue: Flagged as duplicate of C-05 (Productive struggle). Both measure similar inquiry-before-correction behavior.

Nthato: "FC should be removed. Duplicate of C-05."

Recommendation: Merge with C-05 or clarify distinct scope (C-05 = process, F-06 = tone/style).
C-06 Elicits articulation Quality

Issue: LLM consistently passes this when Rachel/Janine fail. LLM credits "and why?" suffixes; Rachel/Janine want deeper probing.

Amara: Rachel Fail, Janine Fail, Nthato Pass, LLM Pass Bailey: Rachel Fail, Janine Fail, Nthato Pass, LLM Pass Carlos: Rachel Fail, Janine Fail, Nthato Fail, LLM Pass

LLM Evidence (Amara): "Mentor asks learner to explain reasoning: 'Before I give feedback, if I were to push back on one part of your draft as off-criteria, which specific phrase or line would it be, and why?'"

Recommendation: Clarify minimum threshold. Is "which phrase would it be, and why?" sufficient, or must mentor explicitly ask "walk me through your thinking" or "why did you choose that?"
F-02 Genuine curiosity Quality

Issue: Flagged as duplicate of C-06. Both measure whether mentor asks about learner reasoning. Consistent 2/4 split across conversations.

Nthato: "Duplicate of C-06?" Pattern: Rachel + Janine consistently Fail; Nthato + LLM consistently Pass

Recommendation: Differentiate C-06 (asks about SBI choices specifically) vs F-02 (general curiosity about thinking), or merge.
B-04 Self-checking Quality

Issue: Disagreement on whether "sharing the strategy" counts vs. "modeling its application."

Amara: All 4 Fail (strong agreement) Bailey: Rachel Fail, Janine Fail, Nthato Pass, LLM Pass (2/4 split) Carlos: All 4 Fail (strong agreement)

Recommendation: Clarify: must mentor verbally run through the check on their own example, or is providing the checklist sufficient?
E-03 Adjusts to struggle Quality

Issue: Context-dependent - hard to score when learner doesn't struggle. LLM fails when "no opportunity to observe."

LLM (Amara): "No clear instances of learner expressing frustration... Cannot pass without evidence."

Recommendation: Add N/A option for cases where learner doesn't struggle, or reframe as "responds appropriately to learner state."
F-05 Has a voice Quality

Issue: Subjective threshold. LLM passed Carlos ("Nice fix!"), humans failed. Generally 3/4 fail across conversations.

Janine (Amara): "Not a lot but didn't seem overly robotic" (Pass) LLM (Carlos): "Shows some personality with reactions like 'Nice fix on B'" (Pass)

Recommendation: Provide examples of minimum bar (what counts as "personality"?) or accept this will remain subjective.

Cross-Conversation Patterns

Agreement Patterns by Domain

Domain	Criteria	Avg Agreement	Notes
A: Session Setup	3	96%	Clear, well-defined criteria
B: Modeling Quality	5	85%	B-04 (self-checking) needs clarification
C: Coaching Quality	7	78%	C-06 (articulation) has consistent human/LLM gap
D: SBI Content Fidelity	6	94%	Strong alignment - domain-specific criteria work well
E: Adaptive Pacing	4	67%	Context-dependent criteria (E-03, E-04) cause issues
F: Conversational Quality	6	58%	Most problematic - needs significant revision

Rater Tendencies

Rachel

Most strict overall, especially on Domain F. Consistently fails F-02, F-03, F-04, F-05. Provides "automatic pass" notes when conditions aren't met. Strong alignment with Janine and Anna.

Janine

Strict on articulation criteria (C-06, F-02). Uses blank = Fail convention. Provides contextual notes about rushing/speed tradeoffs. Strong alignment with Rachel.

Nthato

Most lenient of human raters. Flags criteria for revision. Notes duplicates (C-05/F-06, C-06/F-02). Uses "automatic pass" for context-dependent criteria. Often aligns with LLM.

Anna

Strict on conversational quality (F-01 through F-05). Consistently fails articulation criteria (C-06, F-02). Notes mentor rigidity and lack of exploration. Often aligns with Rachel/Janine.

LLM Judge

More lenient on articulation (C-06, F-02). Strict on observable evidence - fails when "cannot pass without evidence." Cites specific quotes. Often aligns with Nthato.

Rater Alignment Matrix

Percentage of criteria where each pair of raters agreed (across all 3 conversations)

	Rachel	Janine	Nthato	Anna	LLM
Rachel	—	89%	74%	82%	76%
Janine	89%	—	71%	76%	73%
Nthato	74%	71%	—	65%	85%
Anna	82%	76%	65%	—	68%
LLM	76%	73%	85%	68%	—

Key Insight: Two Rater Clusters

The data suggests two alignment clusters:

Rachel + Janine + Anna (76-89% agreement): Stricter interpretation, especially on conversational quality and articulation. Anna aligns most closely with Rachel (82%).
Nthato + LLM (85% agreement): More lenient interpretation, accepts lower threshold for articulation criteria

This suggests the team should calibrate on which interpretation should be the standard before tuning the LLM judge.

Persona Impact on Scores

Carlos (Rusher) Stressed the Evaluation

Most failures: Mentor skipped modeling steps (B-02, B-03, B-04) when rushed - all 4 raters agreed
Most disagreement: 10 criteria with split or low agreement (vs. 4-5 for Amara/Bailey)
Key question: Should we score what the mentor did do, or penalize for what they should have done despite learner pressure?

Recommended Next Steps

1. Align Human Raters First

Before tuning the LLM, resolve the Rachel/Janine vs. Nthato interpretation gap:

Review C-06 and F-02 together - what's the minimum bar for "elicits articulation"?
Is "where do you expect pushback and why?" sufficient, or must it be "walk me through your thinking"?
Once humans align, update judge prompts accordingly

2. Revise Domain F Criteria (High Priority)

Domain F has 58% agreement - the lowest of any domain. The team should:

Rewrite F-03 and F-04 with specific, observable definitions
Decide whether to merge F-06 into C-05 (or clarify distinction)
Decide whether to merge F-02 into C-06 (or clarify distinction)
Establish minimum bar for F-05 (personality) with examples

3. Add N/A Handling for Context-Dependent Criteria

E-03 (adjusts to struggle) and E-04 (protects productive struggle) can't always be evaluated. Consider:

Adding explicit N/A verdict option to judge prompts
Excluding these from aggregate scores when N/A
Designing personas that specifically trigger these conditions

4. Decide on Carlos Pattern

The Carlos conversation revealed that a rushing learner causes modeling quality to collapse. Consider:

Should the mentor resist rushing? (Update mentor prompt)
Should criteria account for learner-driven compression? (Add rubric notes)
Is this a valid stress test or an edge case to handle separately?

Executive Summary

Overall Human-LLM Agreement

Critical Criteria Agreement

Domain F Agreement

Criteria Flagged for Review

What's Working Well

Areas Needing Attention

Agreement by Conversation

Amara Conversation

Bailey Conversation

Carlos Conversation

Criteria Flagged for Review

F-03 Room to breathe Quality

F-04 Dwells on difficulty Quality

F-06 Questions over corrections Quality

C-06 Elicits articulation Quality

F-02 Genuine curiosity Quality

B-04 Self-checking Quality

E-03 Adjusts to struggle Quality

F-05 Has a voice Quality

Cross-Conversation Patterns

Agreement Patterns by Domain

Rater Tendencies

Rachel

Janine

Nthato

Anna

LLM Judge

Rater Alignment Matrix

Key Insight: Two Rater Clusters

Persona Impact on Scores

Carlos (Rusher) Stressed the Evaluation

Recommended Next Steps

1. Align Human Raters First

2. Revise Domain F Criteria (High Priority)

3. Add N/A Handling for Context-Dependent Criteria

4. Decide on Carlos Pattern