Loop Dial: more depth, less written reasoning
I varied a research model’s loop count. Math and code improved by three passes, reasoning got shorter, and extra depth kept costing more.
A model whose depth I can dial
This model has a depth setting. I can run the same weights with one pass through its looped layers, or give those layers several passes before the next token.
I wanted to see what that buys: better answers, less written reasoning, or just more time on the GPU.
I tested thinkingmachines/TML-d24-varloop-research, the varloop research checkpoint. These are not results for shipped Inkling 2 Mini. The model card describes the distinction.
Looping gives the model more internal computation without changing its weights. Written reasoning is a separate output. I measured both the answers and the amount of reasoning text, rather than treating a longer trace as more computation.
The same problems at different depths
Every depth answered the same problems within each comparison. I used greedy decoding, maximum concurrency 32, an 8,192-token output cap and a pinned SGLang image on one RTX PRO 6000.
The official Inkling parser separated reasoning text from final answers. Math was graded by numeric equivalence, code by sandboxed tests, and the knowledge control by the official SimpleQA Verified rubric and GPT-4.1 snapshot.
| Task | Problems | Passes |
|---|---|---|
| GSM8K | 150 | 1, 2, 3, 4, 5 |
| MATH-500 | 200, equally split by level | 1, 2, 3, 4, 5, 6 |
| HumanEval | 164 | 1, 2, 3, 4, 5 |
| BBH, five tasks | 60 | 1, 3, 5 |
| SimpleQA Verified | 200 | 1, 3, 5 |
I also compared shared and separate KV caches at three and five passes on MATH and HumanEval. Two extra runs at one and three passes used the same 100 MATH problems and all HumanEval tasks. The full report keeps those runs separate instead of counting repeated questions as new evidence.
Three passes reached the math and code plateau
On MATH, accuracy rose from 92.5% at one pass to 99.0% at three. The three-pass 95% confidence interval was 96.4% to 99.7%.
Four, five and six passes scored 97.5%, 97.5% and 98.0%. None demonstrated an improvement over three in the paired tests. The three adjusted McNemar p-values were all 0.75.
Code followed a similar pattern. HumanEval rose from 80.5% at one pass to 95.7% at three, with a three-pass 95% interval of 91.5% to 97.9%. Four passes stayed at 95.7%. Five scored 95.1%.
I would call this a plateau on these math and code subsets, not a general limit of looping. MATH is already close to its ceiling here.
Less reasoning on the page
One to three passes reduced mean visible reasoning by 41.4% on MATH and 41.7% on HumanEval. The paired 95% intervals for the reductions were 26.7% to 53.6% and 29.2% to 52.3%.
Some one-pass responses kept writing until the token cap. That could make the average look longer just because the model failed. I checked the questions answered correctly at both depths. Reasoning still shrank by 27.8% on MATH and 20.4% on code.
Hard MATH problems saved more tokens in absolute terms because they started with longer responses. I did not resolve a larger proportional reduction on hard problems than easy ones. More passes also did not keep shortening the MATH and code means after three.
A small knowledge control
I added SimpleQA Verified as a knowledge control. The model received the bare questions, with no search or tools. Each depth answered the same 200 questions.
Accuracy was 7.0%, 5.0% and 5.5% at one, three and five passes. The one-versus-three McNemar test gave p = 0.289. It did not resolve an accuracy difference. It also did not prove equivalence.
| Passes | Accuracy | 95% CI | Mean reasoning tokens |
|---|---|---|---|
| 1 | 7.0% | 4.2% to 11.4% | 1,577.5 |
| 3 | 5.0% | 2.7% to 9.0% | 1,033.6 |
| 5 | 5.5% | 3.1% to 9.6% | 829.2 |
Reasoning still shrank by 34.5% from one to three passes: 543.9 fewer tokens on average, with a paired 95% interval of 271.8 to 839.0 fewer. The control has one run per depth, and an automated grader can make mistakes. I cannot turn this into a claim that knowledge stays flat in general.
Shorter writing still cost more
Shorter reasoning did not consistently mean a faster answer. On MATH, measured batch GPU time per correct answer went from 0.685 seconds at three passes to 1.338 at six. On the same GPU, that nearly doubled inference cost.
This is the whole batch time divided by the number of correct answers. It includes the work on wrong answers, but excludes model loading and grading. It is a throughput cost at concurrency 32, not the time an individual user waits for an answer.
My interpretation
This is an interpretation, not a measured internal mechanism. More internal passes and less written working can occur together. The math and code result survives restricting the comparison to answers correct at both depths, so it is not entirely an effect of capped failures.
That still does not show that the loops replaced particular steps of written reasoning. I measured outputs and timing, not the model’s hidden computation. The comparisons with similar observed compute are descriptive, not controlled experiments with a fixed reasoning budget.
My read is narrow: written length is not the whole compute budget. More depth can reduce the text the model writes, while still making each correct answer more expensive.
What this study cannot tell us
- The checkpoint saw passes one through five in later midtraining, sampled in the ratio 1:2:6:2:1. It was post-trained at three. Passes four and five are midtraining only; six is beyond the trained range. The model card is the source for those labels.
- MATH has little room left in this sample. At three passes, level five scored 38 out of 40, with a 95% interval of 83.5% to 98.6%. Fixing both remaining errors would add only one point overall. That is headroom, not evidence that more passes recover it.
- The repeat accuracy ranges reached one percentage point on the matched MATH subset and 1.83 on code. Small higher-depth score differences need to be read against that noise.
- BBH rose from 55/60 at three passes to 57/60 at five, but its paired p-value was 0.625. The changed outcomes were all in bracket completion. This small task mix does not establish a broad reasoning gain.
- Shared cache did not improve aggregate accuracy in the measured cells. Its timing effects varied by domain and by the responses generated.
- These are public benchmark subsets, one checkpoint, one runtime, greedy decoding and a fixed output cap. Benchmark familiarity and systematic grading errors are not captured by the confidence intervals.
Cost and sources
The original full-study resource upper bound was approximately $4.79. The knowledge control added approximately $1.75. These estimates come from GPU timestamps and returned grader-token usage. They are not verified invoices.
The repository contains the saved scored outputs, subset IDs, licences, full tables, paired tests and scripts to regenerate the analyses and charts without a GPU or API calls.
Thinking Machines had already reported results on looping and shorter reasoning. This study adds these fixed subsets, the deeper pass settings, answer lengths, timing, cache comparisons and repeated-run measurements. It is a study of the varloop research checkpoint, not shipped Inkling 2 Mini.