Prompting strategies that did not improve calibration

This is a negative result, published because the techniques in question are widely recommended and, in our testing, do not work. Five prompting strategies that are supposed to improve how well a model's confidence matches its accuracy produced no reliable improvement in any configuration we tried.

What we tested

We selected the five strategies most often cited in practitioner guides:

• Asking the model to state its confidence as a percentage before answering
• Asking the model to list reasons it might be wrong before answering
• A two-pass prompt that answers, then re-estimates confidence with the answer visible
• Persona prompts that frame the model as a cautious expert
• Few-shot examples with carefully calibrated confidence annotations

Each strategy was run on 6,000 questions spanning math, factual recall, and reading comprehension, across three open models. Calibration was measured with expected calibration error and Brier score, with bootstrap confidence intervals over 1,000 resamples.

What happened

Nothing moved. Expected calibration error shifted by less than half a point in either direction for every strategy, model, and domain combination, and no shift survived the confidence intervals. Two strategies changed the stated confidences cosmetically: the persona prompt made numbers rounder, and the two-pass prompt made them higher. Neither made them more accurate.

A model that says 80 percent and is right 55 percent of the time is not calibrated, no matter how politely the number was produced.

Why publish this

Collectively, these five strategies appear in a large share of the deployment guides we reviewed. Teams adopt them believing they bought epistemic discipline for the price of a prompt. They did not. If you need calibrated confidence, measure it on your task and adjust for the bias you find. There is no wording trick that does it for free.