Judgment Lab

Strengthening human judgment in AI-enabled work.

Failure mode 1 of 7

Frame capture

PME means professional military education. The Classroom version is also available for high school.

An analyst at a health system asks an AI which patients most need a care program's help, and the AI proposes a reasonable way to frame the problem. The analyst works hard from there. What can happen next is frame capture: the first plausible frame becomes the working frame, unexamined. It can happen whenever a measure stands in for what you actually care about, like sales for customer value or tickets closed for problems solved. Illustration based on Obermeyer et al. (2019). The analysts, their AI and the task are constructed.

1:44 · Read the figure

Frame capture, an animated figure in fourteen beats

An animated figure. On the left, a red dot stands for an analyst, surrounded by six dashed considerations. An AI's grey frame takes in two of them, next year's costs and the program budget, and turns the task question, "which patients need the most help from the program?", into "which patients cost the most?". The analyst steps into the frame and builds and revises a ranking there. The view moves to the manager’s side, where the ranking arrives and the signs of the AI's frame drain out of it. A second analyst, also using AI, gets the same proposed frame, weighs all six considerations, keeps cost for now and writes down what would change it: if some patients cost less because they get less care. That analyst's ranking is identical. Asked why this frame and what would change it, the first analyst has nothing to check. The second checks two patients with the same predicted cost; one has more chronic conditions, so some patients are sicker. Their frame changes from next year's costs to chronic illness, and their ranking is rewritten as "which patients are sickest?". A note on the first ranking says sicker patients were ranked lower. The figure ends on the reviewer’s check, with the source under the picture: in the study, at a given risk score, Black patients were considerably sicker than White patients. Each beat is listed below with what is on screen.

  1. Frame capture · Failure mode 1 of 7

    On screen: the name, "Frame capture", appears large over the stage with "Failure mode 1 of 7" above it, then moves into the top-left corner, where it stays.

  2. An analyst gets an open care-program task.

    On screen: a task card appears at the top right: "Example task: Which patients need the most help from the program?" A red dot labelled Analyst appears on the left.

  3. Many things could matter. Framing means picking which ones define the problem.

    On screen: six considerations appear one at a time in dashed outline, three above an open space and three below: chronic illness, program budget, next year's costs, access to care, avoidable visits, and who it can help.

  4. The analyst asks an AI. It proposes a reasonable, narrower frame.

    On screen: the dot pulses. Grey corner brackets labelled "AI's frame" snap around the open space beside the analyst. A grey copy of the task question appears in the frame; "need" and "help from the program" fade, and "cost" appears, so it reads "Which patients cost the most?" Next year's costs and the program budget slide inside as grey headings.

  5. The analyst takes up the frame. Up close, it's all they see.

    On screen: the dot steps into the frame. The view moves in until the frame fills it, and the dot writes a rough first ranking in two columns under the AI's headings.

  6. The analyst revises twice, adding evidence and a firmer cut-off.

    On screen: the dot straightens the ranked rows, then adds a chart and a cut-off line across both columns. Red change bars mark each revision. The red label reads Analyst's draft 3.

  7. Step back with us: every revision stayed inside the AI's frame.

    On screen: the view pulls back to the whole picture while the dot stays where it is. The four dashed considerations sit where they were. Every red mark is inside the grey brackets.

  8. The manager receives the final ranking.

    On screen: a clean final copy of the ranking travels to the manager’s side and stops in the middle, large, labelled "What the manager receives: the final ranking". It is still inside grey brackets, and a key marks its tinted title and headings as "AI-supplied". A wipe crosses it, and the title and headings turn the same ink as the rest, keeping their words. The brackets and the key fade.

  9. Nothing in it shows the frame came from the AI.

    On screen: the ranking, now all one ink, moves to the top left, labelled "From this analyst".

  10. A second analyst, also using AI, weighs all six and keeps cost, for now.

    On screen: the label over the first ranking becomes "First analyst". A second red dot labelled Second analyst appears. Its AI proposes the same frame: grey brackets tagged "AI's frame" around next year's costs and the program budget, with the other four outside. The dot touches each of the six in turn, and each turns solid. Then the brackets turn red and the tag reads "Chosen"; the other four go under "Set aside", and a red note says what would change it: some patients costing less because they get less care.

  11. That analyst writes the same ranking; the weighing doesn't show.

    On screen: the second dot writes a ranking identical to the first, labelled "Second analyst". Then the weighing slides up behind the ranking and is hidden.

  12. Ask each analyst why this frame. One knows what would change it.

    On screen: "The manager asks each analyst: Why this frame?" appears. From behind each ranking, what it rests on slides into view. Under the first is only the AI's grey frame around its two considerations, with the first analyst inside. Under the second is the same frame in red, chosen, with the four set aside and the note on what would change it.

  13. The second analyst checks it: at the same predicted cost, some patients are sicker.

    On screen: under "Why this frame?", the manager's second question appears: "What would change it?". Then a red panel, "Same predicted cost", draws two patients: equal cost bars, and dots for chronic conditions, two for one patient and five for the other. Then: "Checked: some patients are sicker."

  14. So the frame changes, and so does the ranking. The cost ranking put sicker patients lower.

    On screen: in the second analyst's chosen frame, chronic illness replaces next year's costs, which joins the set-aside four. Then the second ranking is rewritten with red change bars under a new title, "Which patients are sickest?". Then a red note on the first ranking, "Sicker patients ranked lower", and under the first analyst's frame, "Nothing to check".

  15. Reviewer’s check: What problem did the analyst choose to solve? Why this frame? What would change it?

    On screen: the questions appear under the picture, at the same size. The task card's "need" and the first ranking's title, "Which patients cost the most?", with its note "Sicker patients ranked lower", are in view beside the revised ranking, "Which patients are sickest?". Under the picture: "Illustration based on Obermeyer et al., Science (2019). In the study, at a given risk score, Black patients were considerably sicker than White patients."

Reviewer’s check

What problem did the analyst choose to solve?

Why this frame? What would change it?

Before the check, both analysts in the figure would answer the first question the same way. The second line separates them: one can say what would change their frame, and check it.

The study behind this example

The study looked at a commercial prediction algorithm of the kind health systems use "to identify and help patients with complex health needs". It was a prediction model. No generative AI was involved.

The paper states its finding this way: "At a given risk score, Black patients are considerably sicker than White patients, as evidenced by signs of uncontrolled illnesses. Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%."

The authors trace it to what the algorithm was built to predict: "The bias arises because the algorithm predicts health care costs rather than illness, but unequal access to care means that we spend less money caring for Black patients than for White patients."

Cost looked like a good stand-in by some accuracy measures, and the bias was there anyway: "despite health care cost appearing to be an effective proxy for health by some measures of predictive accuracy, large racial biases arise."

Their conclusion is about choosing the measure, the step this figure calls the frame: "We suggest that the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias in many contexts." Science's editor's summary of the paper adds: "Reformulating the algorithm so that it no longer uses costs as a proxy for needs eliminates the racial bias in predicting who needs extra care."

In the figure, the second analyst writes down what would change the frame and then checks it. The check follows the study's comparison: how sick patients were at the same risk score, where the score was the algorithm's prediction of cost. The figure uses chronic illness as its measure of need.

Source: Obermeyer, Powers, Vogeli & Mullainathan, "Dissecting racial bias in an algorithm used to manage the health of populations," Science 366 (2019): 447–453. Quotes are from the abstract (p. 447) and the journal's editor's summary. The analysts, their AI, the six considerations, the rankings and the check are constructed. The figure shows none of the study's data.

From the original PME essay

Frame capture occurs when the model supplies the first plausible frame and the student never achieves enough distance to revise it. The danger is not that the frame is obviously wrong; it may be entirely reasonable. It may simply define the problem too narrowly, privilege one set of interests over others, assume a theory of adversary behavior, or treat a structural constraint as fixed when it should be contested. Once accepted, later revisions improve the answer inside the wrong boundary. The frame capture is invisible inside the final product.

The Irreducible Officer, Section II, What the Problem Looks Like

Go further

The failure mode and the reviewer's question come from the essay below. This page shows the same move in a health system's analytics team.

All failure modes