Mohith
All writing

Twenty Bits: better questions, unreliable stopping

RL taught Qwen3-8B to ask better questions in 20 Questions, but not when to stop.

On this page

Valid questions, little information

Before training, Qwen3-8B gained just 0.158 expected bits per question on noiseless animal games. A perfectly balanced yes/no split can give 1 bit. The model was leaving most of that information on the table.

It also failed to make a final guess in 34 of 56 of those games. So the question was about more than winning: could an outcome-only reward improve the questions, and would the model learn when to stop asking them?

Good search divides the remaining candidates as evenly as possible. RL improved question quality in every trained seed. Winning told a less consistent story: some policies guessed too soon; another kept asking after its budget was gone.

A table, a target, a yes or no

Each game has a table of items and binary attributes. The model sees the complete table, but not the hidden target. It asks a Boolean question and receives YES or NO from a script. There is no judge model.

ASK: a AND NOT bGUESS: exact_item_name

The grammar supports exact attribute names or table codes, AND, OR, NOT, parentheses and item == name. A wrong guess ends the game. Invalid questions consume budget.

For N candidates, the limit is 2 × ceil(log2 N) questions, followed by one final-guess opportunity. Another question then loses.

After every answer, the environment supplies the remaining candidates. With truthful answers this is the exact consistent set. With noise, it supplies candidates above posterior probability 0.001, plus the most likely candidate. It does not reveal numeric probabilities.

This is a study of question selection given supplied candidate state. It does not establish that RL teaches the model to track the search state unaided.

Step through a saved game. Watch which candidates survive each answer.

One saved target, with the raw base and both truthful-trained policies. Use the policy selector and step controls to compare their questions. The example is selected by a disclosed behavior rule, not as a representative performance sample. Selection rule and full transcripts.

Reward wins. Measure information.

R=+1W-0.03Q-0.1I

W marks a win, Q counts questions, and I counts invalid questions.

Expected information gain, or EIG, asks how much uncertainty a question should remove before its answer arrives:

EIG = H(C) − Ea[H(C | a)]

Here C is the hidden candidate and a is the answer. Entropy, H, measures uncertainty about that candidate. The table makes EIG exactly computable, averaging over both possible answers and accounting for noise. Evaluation includes losing games and invalid questions.

EIG is measured and never rewarded. The model has to discover informative questions through the game outcome and penalties. Adding an information bonus would answer a different question.

All 3 runs start from raw Qwen3-8B. A11 and A23 use truthful answers; B41 uses independently flipped answers with probability 10%. Each gets 30 updates, sampling groups of 8 games from a shared start state and using group-normalized return advantages.

The animal curriculum moves through 4, 8, 16, 32, 64 and 128 candidates. Thinking is disabled; each turn has a 2,048-token output cap. Frozen evaluation cases and transfer domains never enter training.

Better questions in every seed

On noiseless animals, EIG per question rises from 0.158 for raw base to 0.634, 0.390 and 0.437. The paired confidence intervals for all these increases stay above zero. None reaches the 1-bit ceiling.

Expected bits per question on noiseless animals improve in all three trained seeds, with confidence intervals, while staying below the ceiling. Exact values follow in the table.
Equal weight on the smaller and larger animal cells. Intervals resample games within each fitted endpoint, not independent training seeds. Source data. Open full-size chart.
Expected bits per question, with 95% confidence intervals
PolicyEIG / questionInterval
Raw base0.1580.112 to 0.211
A11 · truthful0.6340.594 to 0.677
A23 · truthful0.3900.333 to 0.450
B41 · noisy0.4370.384 to 0.490

The accepted questions use attributes rather than legal item-name enumeration. That does not mean the model never tried name search: the audit below catches attempts the grammar rejected. Improved question quality is the clearest result; optimal search and general synthetic transfer are not established.

Similar wins, opposite failures

A11 wins 24 of 56 noiseless animal games. A23 wins 25 of 56. A single win count makes these policies look similar.

A11 loses the other 32 through ambiguous wrong guesses. A23 leaves 31 games without guessing. B41 improves its questions but also gambles: 36 ambiguous wrong guesses and 19 wins on the same animal slice.

Stacked outcomes separate wins, ambiguous wrong guesses and no-guess endings, showing A11 gambling and A23 stalling despite similar animal win counts.
The animal slice and the complete primary bank are shown separately. Wins can include lucky guesses made while several candidates remain. Source data. Open full-size chart.
Watch the policies part ways. The same target reveals A11 gambling and A23 stalling. The selection rule asks for these failures without filtering for a base win. This illustrates the difference; it does not estimate its frequency. Full text alternative.

One line removes the stalling

The primary prompt states the maximum number of questions, but never shows how many are left. A separate evaluation adds this line on every turn:

Questions left: k of K

At exhaustion: No questions left. You must GUESS now.

No further training is involved. The available endpoints replay the same frozen cases, with new prompt hashes kept separate from the primary results.

A23’s no-guess endings fall from 136 to 0. Its wins rise from 138 to 186 out of 280. Its stalling depended on missing explicit budget state.

B41 still makes ambiguous guesses with budget left in 188 games, down from 202. Showing the counter removes the final extra questions, but it does not remove gambling.

Primary versus visible-budget outcomes: A23 no-guess endings disappear and wins increase, while B41 continues to make many ambiguous guesses with budget left.
The prompt changes from the opening turn, so it can affect question choices too. This does not isolate the final instruction alone, and it does not identify an internal mechanism for gambling. Source data. Open full-size chart.
Same weights, an explicit counter. A23 with and without the remaining budget. The visible-budget rollout wins while several candidates remain, so this is not a demonstration of certainty. Full text alternative and selection rule.

Budget tracking and stopping strategy are different failures. A23 follows an explicit final-guess instruction. B41 continues to gamble under both prompts. The experiment does not establish that any single reward term causes that behavior.

No clear help from noisy training

Under the primary prompt, raw base wins 52 of 140 noisy games. A11, A23 and B41 win 31, 29 and 36. Every trained endpoint wins fewer than base, including the noisy-trained policy.

Raw base wins more noisy games than each trained policy under the primary prompt. The noisy-trained endpoint does not recover the base win rate.
There is only one noisy training seed. Its comparison with the mean truthful-trained policy has a win-rate interval that includes no improvement. Source data. Open full-size chart.

Question quality and error correction are different measurements. With 10% independent answer noise, even a balanced question has a ceiling of about 0.531 expected bits. Re-ask opportunities also depend on each policy’s chosen questions, so raw re-ask counts do not establish a general checking strategy.

Loose ends and failed attempts

Identifier errors varied by seed. A11’s synthetic code-plus-suffix errors did not repeat at the same scale in A23. Raw base was already strong on the relabeled tables. The study does not show a general improvement in synthetic transfer.

The grammar blocked attempted name search. A post-hoc audit finds 142 invalid A23 training questions with exact candidate-name references. Bare names and malformed lists were rejected. Proper item == name lists remain legal. The audit changes no score.

One update collapsed. A23’s update 23, at 64 candidates, has 0 wins and a burst of invalid and truncated output. Later behavior partly recovers. The cause is not established.

Sampler and trainer probabilities differed. Maximum token logprob gaps are 2.704, 1.621 and 3.803 nats for A11, A23 and B41. Native token sequences match; the backend cause and behavioral effect are unknown.

What this study cannot tell us

  • Candidate state is supplied. This is question selection, not unaided posterior tracking.
  • The animal tables are authored profiles. Held-out rows can share underlying animal types with training rows.
  • The fixed Boolean grammar removes free-form interpretation and blocks some malformed strategies.
  • At 256 synthetic candidates, both domain and candidate-set size differ from training. This cannot isolate pure domain transfer.
  • Thinking is disabled. Visible reasoning does not measure internal computation.
  • There are 2 truthful seeds, 1 noisy seed and 28 games per frozen cell. Game-level confidence intervals do not capture training variability.
  • Each training group shares a table and target. The rollout count overstates the number of independent search worlds.
  • The A11 hosted checkpoint was lost. Its deferred hard-mode and parser comparisons could not run.
  • The no-candidate-list hard-mode comparison is tiny, and the replay examples are deliberately selected to show failures.

About $24 on Tinker

Recorded project usage is $23.681223, plus $2.083181 held pending reconciliation. These are observed-token charges at pinned prices, not a reconciled invoice. Holds cover unresolved requests and checkpoint storage.

The useful result is narrower than “RL learns optimal search.” Outcome-only training improved attribute-based questions in every seed. Reliable stopping and noisy play remained unresolved. Supplying the budget fixed one missing piece of state, not the whole policy.

The code, frozen cases, confidence intervals, failures and saved replays are public. This page uses the completed study’s artifacts and makes no new model calls.

Training and model sampling used Tinker by Thinking Machines.