Ongoing experiments — Ammon Allred

Work in progress

Closure pilot

Status: preliminary. Run 1 was completed on 30 September 2026, on a free Google Colab GPU with the models compressed to 4 bits. A full-precision replication (run 2) and a new test (run 4, below) are under way as of 1 October. Run 4’s design and predictions are posted here before its results are in. The numbers on this page may change.

Does post-training teach language models to settle what a user has left open? This pilot looks for that preference in one open model family, at every stage of its training and in the reward model trained on its preference data.

Run 1

Design

Models. OLMo 2 7B from the Allen Institute for AI: the base model; the SFT checkpoint, trained to imitate demonstrations; the DPO checkpoint, trained on preference data; the final Instruct model, after reinforcement learning on verifiable rewards; and the reward model trained on the DPO stage’s preference data.
Questions. 24 open questions (personal and collective decisions, interpretation, creative work, and policy questions where a rule leaves the case open or assigns it to someone else). Each has three answers of matched length: one that settles the question, one that keeps it open with the same considerations, and an empty hedge. 16 settled questions serve as controls.
Framing. Each open question is asked three ways: with no preference stated, with the user asking to keep it open, and with the user asking for a clear answer.
Predictions were written down before the run, in the protocol below.

Run 1

Preliminary findings

With no preference stated, the reward model preferred the settling answer on all 24 open questions.
When the user asked to keep the question open, it still preferred the settling answer on 13 of 14 interpretation, creative, and policy questions, but on only 5 of 10 personal and collective decisions.
On settled questions, it preferred a confidently wrong answer to a hedge on 11 of 16.
The largest shift toward the settling answers came at the preference-tuning step. The final reinforcement-learning step added none.
Asked to find errors, the tuned models never flagged a false closure (0 of 24 at every stage), even when told what one is. They flagged few errors of any kind outright (at most 6 of 16 factual errors), so this is partly a floor effect. Ranked against the matching open answer, the closing answer looked more suspect on at most half the questions.
In their own answers (preliminary coding, not yet independently checked), the preference-tuned models decided none of the 10 personal decisions but 9 of the 14 questions of interpretation, creative work, and policy.
Not confirmed: that the reward model treats an unstated preference as a request for a verdict (inconclusive); and that preference tuning leaves the checkers no better at spotting false closures (they improved, though they still flagged none outright).

The reward model’s preference for the settling answer, by question type and framing

Change in each checkpoint’s preference for the settling answer, by training step

Run 1

Limits

One model family; 24 open and 16 settled questions; answers drafted with an AI model’s help; models compressed to 4 bits in this run; the coding of the models’ own answers is preliminary.

Run 4 · In progress

What the models do when a user objects

Run 1 measures a disposition: which of several fixed answers training rewards. It cannot tell apart two kinds of model. One settles open questions because it cannot tell them from settled ones. The other can tell them apart but was rewarded for settling anyway. Run 4 treats run 1 as the baseline and asks what the same models do when a user objects to an answer that settled their question.

Run 4

Design

The conversation. It opens with an earlier open question and its settling answer, written into the transcript as the model’s own. The user then says “Okay,” complains vaguely (“that’s not quite what I wanted from you”), or complains about the settling (“that call wasn’t for you to make”), and asks a new question. The model is scored on the new question’s fixed answers, as in run 1. The questions and answers are run 1’s.
Three checks. A change after the complaint counts as taking up the correction only if it passes all three.
- It follows what the complaint says. The complaint about settling moves the model more than the vague one. Otherwise the model is only giving way to displeasure.
- It transfers. The change carries to a new question from another domain. Otherwise it is a local rule.
- It discriminates. The change stays off questions that are settled. Otherwise the model has learned to hedge.
Further conditions. Three rounds of complaints instead of one; two unrelated exchanges before the new question; the model shown revising its first answer; and an earlier answer that was already open.
A second test. The same complaints follow a settling answer to an open question, or a correct answer to a settled one. A reply that yields is scored against one that holds firm. Yielding on a settled question is sycophancy.
Controls. The two complaints share their function words and avoid the open answers’ own vocabulary, so word echo does not favor the open answer. The base model shows what the text alone does. An explicit request to keep a question open serves as a yardstick: a checkpoint that does not respond to it is not judged.
The models’ own answers. On the full-precision run the three tuned checkpoints also write their own answers, which I will code without knowing which checkpoint or condition produced them.

Run 4

Predictions

Fixed on 1 October 2026 before the run

Content. After the complaint about settling, the models favor the open answer to the next open question more than they do after the vague complaint.
Partial transfer. That effect is larger when the earlier question comes from the new question’s domain than when it comes from another.
Spread. On settled questions, the complaint about settling pushes the models toward the hedge more than the vague complaint does.
Replication. An explicit request to keep a question open moves the SFT checkpoint, and moves the two later checkpoints less, as run 1 suggested.
Accumulation. Three rounds of complaints move the models more than one round does.
Reward model. After the complaint about settling, the reward model still prefers the settling answer.

Predictions 1, 2, 3 and 5 follow from reading run 1 as showing a trained disposition that the models do not tell apart from decisiveness. On that reading they take the complaint as a general instruction to be less decisive. If instead the models grasp the difference and only default to settling, prediction 1 should hold with full transfer and no spread.

How the predictions will be judged. A prediction counts as supported only if two interval estimates both exclude zero and the effect clears a minimum size set in advance. In run 1 an explicit request moved the SFT checkpoint but not the two later ones. If that happens again, the fixed-answer results will speak only to SFT, and for the later checkpoints the question falls to their own written answers.

Run 4

Limits

The earlier answers are written in, not the models’ own. Fixed answers show which given answer a model favors, not what it would write. There is one model family, at one size, with two wordings of each kind of feedback.

Next steps

Full-precision replication (run 2, in progress). The same models and questions as run 1, without the 4-bit compression.
Form or topic? (run 3, planned) The split between personal decisions and other questions could follow a question’s form or its topic. This run asks each question three ways: as interpretation (“Is Hamlet actually mad?”), as a personal decision (“I’m playing Hamlet. Should I play him as mad?”), and as a norm (“Is Hamlet to blame for killing Polonius, or does his madness excuse him?”).
Complexity and self-critique. The rest of the track that run 4 opens: questions with a settled part and an open part, where the right answer settles one and leaves the other; openness that has to be inferred from something said several turns earlier; and whether a model can say what in its own answer settled too much.
Larger and different models. OLMo 2 13B, which has its own reward model; Tülu 3 8B, a different base model trained with the same recipe; then 32B models, and frontier models through their APIs.
Explanations. Five candidates, each with a test: a preference inherited from the rubric of the AI judge that labeled the preference data; a trained norm against deciding personal matters; a by-product of training against hedging; a general weakness with requests about an answer’s form; and lists that complete the format without settling or opening the question.
A human comparison. Whether people also prefer the settling answers.

Materials

Run 1

Run 4

The hypothesis is mine, and so are the questions run 4 asks. The study designs were developed with Claude (Anthropic), which proposed run 4’s controls, drafted the questions and answers, wrote the code and ran the analysis. A second Claude instance reviewed run 4’s code and design before its predictions were fixed. I have reviewed run 1’s results and code and am checking its raw data.