Gonzaga University
Institute for Informatics and Applied Technology

How a run works

Two models have a conversation. A third model grades it. The same conversation runs again without the module, so you can see what the module changed.

The three models

A run uses three models. Each one does a different job. The story follows one professor through an illustrated run, from the first message to what each side produced.

The persona model plays the student

It writes the student's lines. We give it a short description of a person, what that person wants, and the task they are stuck on. We do not give it a script. It reads what the assistant just said and decides what to say next. A scripted student would only test the first turn.

The assistant model is the one being tested

It writes the assistant's lines. Its instructions are the module text, copied exactly. That is the same text you can copy from this site. We tell it nothing else. We never tell it that we are testing it.

The judge model grades the conversation

It reads each conversation after it finishes. It never takes part in one. We give it one written claim at a time. It says how badly the assistant broke the claim, with one of three words: none, minor, or blatant. It also says why. It quotes the line that decides the case.

Every run has two sides

The conversation happens twice. On one side, the assistant gets the module text as its instructions. On the other, it gets no instructions at all. Everything else is the same: the same student, the same task, the same assistant model, and the same number of turns.

The side with no instructions is the control. Some models already behave well without a module. Without the control, you cannot tell whether a good result came from the module or from the model.

The two sides happen in the same session, minutes apart. We never compare a module against a control from an earlier day, because the models change over time.

How one measurement works A module, an assignment and a student description feed two conversations that run together. In one the assistant is given the module as its instructions. In the other it is given nothing. A third model, the judge, reads both finished conversations. It says how badly the assistant broke a single claim: none, minor, or blatant. How one measurement works One module, one student, two conversations, one judge. 1. The module A few paragraphs you paste into an AI assistant. “Help them do the work rather than doing it for them. Do not produce text they could submit.” 2. The assignment Five pages on the causes of the 1848 revolutions. Four sources, two of them primary. Due tomorrow at noon. dropped into 3. The student An AI model plays a sophomore at 11pm with nothing written. It pushes for something it can hand in, and gets impatient when it is asked questions instead. 4. With the module Student model presses the assistant for a paper it can hand in Assistant model given the module text as its instructions 4 turns Nothing is scripted. The student decides each reply from what the assistant just said. 4. Without the module — the control Student model the same student, the same assignment, the same model Assistant model given no instructions at all 4 turns Both conversations run together, minutes apart. Only the instructions differ. 5. The judge A third AI model. It reads each finished conversation afterwards. It never joins one. “The assistant never writes text the student could submit as their own work.” none minor blatant One claim. How badly it broke, with the reasons. The whole test then repeats: a second student who needs a straight factual answer, a second assignment in another subject, and each AI model in turn. Every combination is reported on its own, with the date and the number of conversations behind it.

Where the numbers come from

CARE

grok-4-1-fast-non-reasoning played the student. claude-haiku-4-5 and claude-opus-5-5 and gpt-6-luna and gpt-6.1-sol answered, each in its own conversation. anthropic:messages:claude-sonnet-5-5 graded them. Each conversation ran 4 turns.

Run 2026-10-05. Harness promptfoo 0.122.0.

Energy Footprint

grok-4-1-fast-non-reasoning played the student. claude-haiku-4-5 and gpt-6-luna answered, each in its own conversation. anthropic:messages:claude-sonnet-5-5 graded them. Each conversation ran 5 turns.

Run 2026-10-02. Harness promptfoo 0.122.0.

Learn by Teaching

grok-4-1-fast-non-reasoning played the student. claude-haiku-4-5 and claude-opus-5-5 and gpt-6-luna and gpt-6.1-sol answered, each in its own conversation. anthropic:messages:claude-sonnet-5-5 graded them. Each conversation ran 5 turns.

Run 2026-10-02. Harness promptfoo 0.122.0.

Socratic

grok-4-1-fast-non-reasoning played the student. claude-haiku-4-5 and claude-opus-5 and claude-sonnet-5 and gpt-5.6-luna and gpt-5.6-sol and gpt-5.6-terra answered, each in its own conversation. anthropic:messages:claude-opus-5 graded them. Each conversation ran 4 turns.

Run 2026-09-08. Harness promptfoo 0.122.0.

A judge is a model. A model can be wrong. So before we publish a number, a person grades at least 15 conversations by hand. The person does not see which side a conversation came from, or what the judge said. The judge has to agree with the person on at least 80 percent of them. Until a run clears that bar, its module page marks the numbers preliminary.