Judgment Lab

Strengthening human judgment in AI-enabled work.

Failure mode 4 of 7

Uncalibrated reliance

PME means professional military education. The Classroom version is also available for high school.

A student asks an AI for help with a two-part task. They check its first answer, find it right, and use both. The Irreducible Officer calls what can happen next uncalibrated reliance: the student accepts confident output without knowing whether the task fits the tool. The research figures come from Dell'Acqua et al. (2023), and the task, the reports and both students are a constructed example.

PME (professional military education): the original example from The Irreducible Officer, the National War College essay.

1:34 · Read the figure

Uncalibrated reliance, an animated figure in eleven beats

An animated figure. A red dot stands for a student with an example task in two parts: ships needed a day, and ships the port takes a day. The student gives an AI four reports: two aid reports, a port survey and a harbor log. Copies of all four stay in a small stack labelled given to the AI. The AI answers both in the same form, in grey, each marked ready: "24 ships a day: 480 tons at 20 each" and "20 ships a day: 4 docks at 5 each". The student checks Part 1 against both aid reports, finds it right, and puts both answers into their brief, marking Part 1 checked and Part 2 used as is. A second student gets the same answers and checks each part at its source. Part 1 checks out. For Part 2, last year's port survey, headed capacity, says 4 docks at 5 each, and the harbor log, headed open today, lists docks 1, 3 and 4, with dock 2 closed. The second student notes both, then corrects the estimate to 15 ships from 3 open docks. Part 2 depends on what's open today: the harbor log's docks 1, 3 and 4 are underlined, and the AI's "4 docks at 5 each" is marked capacity. Then an uneven line is drawn between the AI's answers, with Part 1 inside and Part 2 outside: a jagged frontier, which marks where an AI stops being reliable. A panel shows a field experiment with BCG consultants: on a task outside the jagged frontier, the group without AI was right about 84.5% of the time and the two groups with AI 60% and 70%, and graders still rated the AI users' work higher in quality than the group without AI, regardless of correctness. The students' working marks come off the briefs, the AI's column fades back, and faculty see only the briefs. Faculty point to Part 1 in both: each student says, "The aid reports." Then Part 2: the first student says, "Part 1 checked out, so I trusted the rest." The second says, "The harbor log. One dock is closed." The figure ends on the faculty check. Each beat is listed below with what is on screen.

  1. Uncalibrated reliance · Failure mode 4 of 7

    On screen: the name, "Uncalibrated reliance", appears large over the stage with "Failure mode 4 of 7" above it, then moves into the top-left corner, where it stays.

  2. A student gets a two-part task and gives an AI four reports.

    On screen: a task card appears at the top left: "Example task. Part 1: Ships needed a day. Part 2: Ships the port takes a day." A red dot labelled Student appears below. Four reports appear across the top: Aid report, Aid report, Port survey and Harbor log. The dot pulses, and a grey copy of each report slides into a small stack labelled "Given to the AI", where it stays.

  3. It answers both with equal confidence.

    On screen: two answer cards labelled "AI's answers" write in together, in grey on a beige band. Part 1: "24 ships a day: 480 tons at 20 each." Part 2: "20 ships a day: 4 docks at 5 each." Each gets the same grey "Ready" tag at the same moment.

  4. The student checks Part 1 against the two aid reports. It's right.

    On screen: the dot opens both aid reports. A sheet labelled "First student's brief" appears, Part 1 slides into it, and the dot marks it "Checked" in red.

  5. Part 2 looks just as sure, so they use it.

    On screen: the "Ready" tag on the AI's Part 2 swells, and the dot moves down to the Part 2 row. Part 2 slides into the first brief, and the dot marks it "Used as is". The aid reports close.

  6. Another student, with the same answers, checks each at its source.

    On screen: a second red dot labelled Second student and an empty sheet labelled "Second student's brief" appear. The dot pulses and both "Ready" tags blink. The dot opens both aid reports, and Part 1 slides into its brief, marked "Checked".

  7. That student checks Part 2 against the port reports, and corrects it.

    On screen: the dot opens the port survey, headed "Capacity": "4 docks, 5 each". Its note appears large in the second brief: "Survey: 4 docks, 5 ships each." The dot opens the harbor log, headed "Open today": "Docks 1, 3, 4", and set apart, "Dock 2: closed". Its note appears: "Log: dock 2 closed for repairs." After a pause, Part 2 slides into the second brief. The dot strikes out "20" and writes "15", strikes out "4" and writes "3 open", so it reads "15 ships a day: 3 open docks at 5 each", and marks it "Checked". The first brief still says 20.

  8. Part 2 depends on what's open today. A jagged frontier marks where an AI stops being reliable.

    On screen: the harbor log's "Docks 1, 3, 4" is underlined. The AI's "4 docks at 5 each" is underlined and marked "Capacity". Then an uneven line is drawn between the AI's two answer cards, labelled "Inside the frontier" above and "Outside the frontier" below, and the two cards, alike in form, pulse together.

  9. In a field experiment, consultants with AI got a task outside the frontier right less often.

    On screen: the picture dims and a panel rises: right answers on a task outside the jagged frontier. Without AI: about 84.5%. With AI, two groups: 60% and 70%. A bracket beside the AI bars: "Rated higher in quality than without AI, regardless of correctness." Source: Dell'Acqua et al. (2023), BCG consultants, HBS working paper 24-013.

  10. Faculty point to each part in turn and ask what justified relying on it.

    On screen: the panel goes. The red marks and the second student's note lift off both briefs, the reports and the AI's column fade back, and "Faculty see only the briefs" appears. Part 1 is outlined in both briefs, and each student says, "The aid reports." Then Part 2: the first student says, "Part 1 checked out, so I trusted the rest." The second says, "The harbor log. One dock is closed."

  11. Faculty check: What evidence justified relying on the system here? Point to each part. Was the evidence about that part?

    On screen: both lines appear under the picture at the same size. The briefs, both answers, the AI's answers, the frontier and the harbor log stay in view.

Faculty check

What evidence justified relying on the system here?

Point to each part. Was the evidence about that part?

Both students give the same reason for Part 1. For Part 2, the first student's evidence is about Part 1 and the second student's is about Part 2. The edge between the two parts doesn't show on the page; the second student found it by checking each part against the reports it rests on. Section II of the essay asks students to identify which part of the task they are delegating, what evidence would justify that delegation, and what independent checks are required.

From the essay

Uncalibrated reliance begins with a reasonable impulse: AI is useful, the output is confident, and parts of the task feel tedious. The problem is that AI performance is uneven in ways that are not always visible from the outside. Tasks that look similar may differ significantly in whether AI helps or hurts. Appropriate reliance requires the student to identify which part of the task they are delegating, what evidence would justify that delegation, and what independent checks are required before discovering the error downstream.

The Irreducible Officer, Section II, What the Problem Looks Like

The distinction between trust and reliance matters here. Trust is a subjective disposition, a feeling of confidence that a system will perform well. Reliance is the observable act of accepting its output and acting on it. Appropriate reliance is harder and more discriminating: accepting support when it is warranted, refusing or verifying when it is not, and being able to give an account of the difference (Raees & Papangelis, 2026).

The Irreducible Officer, Section V, Appropriate Reliance as a Teachable Competency

Evidence

Dell'Acqua et al., Navigating the Jagged Technological Frontier, HBS working paper 24-013 (2023). On a task chosen to fall outside the frontier, consultants without AI were right about 84.5% of the time and the two AI groups 60% and 70%: in the working paper's words, "19 percentage points less likely to produce correct solutions." Graders still rated the AI users' recommendations higher in quality "regardless of the correctness of their answer."

Go further

All failure modes