A ball leaves a platform. Before watching the rest of the clip, we already expect it to fall. That expectation is useful in everyday life. It becomes a liability when the video shows a world whose gravity points upward.
A model can give a convincing physical explanation without following the motion in the video. In an ordinary scene, learned physics and observed motion often support the same answer. A correct response then tells us little about which source the model followed. InfiniteWorld makes the two sources disagree in a controlled way: change the dynamics, retain the observation, and measure whether the answer follows the observed motion.
When physical knowledge stops being enough
Physical knowledge is a prior over what is likely to happen. Visual reasoning must also infer what is happening in this particular scene. If friction is lower than expected, or an unfamiliar force bends a trajectory, the answer should depend on the motion rather than the object's familiar appearance.
This distinction matters for an agent that must operate in an unfamiliar environment. Explaining how a familiar object usually behaves is different from identifying its current dynamics. The evaluation therefore needs a way to isolate the contribution of observation while keeping the question itself fixed.
InfiniteWorld uses simulator-controlled physical scenes to create that comparison. Its six factors cover gravity direction and strength, friction, restitution, air dissipation, and chiral deflection. Air dissipation changes velocity-dependent damping; chiral deflection adds a sideways acceleration determined by motion direction. Selected two-factor combinations test situations in which more than one aspect of the familiar world has changed.
The clips are controlled examples from the existing project assets. Their role is to make the intervention visible. The scored questions and reference answers come from recorded simulation state, rather than from a viewer's guess about the clip.
Hold the question fixed. Change the motion.
The central experimental unit is a question evaluated in five matched settings. Text only provides its wording and options. Simulator ordered adds chronological frames. Simulator shuffled presents the same frames in a deterministic permutation. Translated ordered and Translated shuffled repeat the comparison with a depth-conditioned visual translation of the simulator video.
Across those settings, the wording, answer options, reference answer, and answer extractor stay fixed. This removes a common ambiguity: a score difference cannot be attributed to asking an easier question in one condition. Frame availability, chronology, and rendering can instead be examined through separate paired contrasts.
Visual gain = simulator ordered score − text-only score
Order effect = simulator ordered score − simulator shuffled score
Rendering gap = simulator ordered score − translated ordered score
These are percentage-point differences computed before rounding. They answer different questions. A model may benefit from seeing frames but make little use of their order. It may follow a clean simulator trajectory yet struggle when the same event receives another visual rendering.
Translation is part of the measurement, not a guarantee that every relevant visual detail is preserved. The rendering gap includes any question-relevant content changes introduced by translation. A higher simulator score should therefore be read as an observed difference between these inputs, rather than a pure causal estimate of photorealism alone.
From noticing an anomaly to inferring a law
The benchmark contains 15,229 questions linked to 3,382 distinct simulator videos. Its five tasks deliberately ask for different forms of reasoning. A model that can name an altered factor may still fail to estimate its amount or transfer that inference to another scene.
| Task | Questions | What must be inferred |
|---|---|---|
| Law identification | 6,764 | Which listed factor differs from the reference, or whether none does. |
| Parameter estimation | 1,230 | A gravity-related quantity or successive bounce-height ratio. |
| Temporal reasoning | 3,024 | An object's relative height over an observed interval. |
| Analytic counterfactual comparison | 2,715 | Height or distance relative to an Earth-normal analytic reference. |
| Law induction | 1,496 | Whether a query and two references share a generator category. |
Law identification uses two arrangements of the options for each video. The meaning of the correct answer stays the same when its position moves. This allows the analysis to distinguish a preference for an answer position from a repeated semantic judgment.
Parameter estimation adds a different demand: return a number. Full and half credit depend on task-specific tolerances, so a near estimate is distinguished from a large error. Temporal questions compare positions within the observed video; because frames have no timestamp annotations, aligning the textual interval with the sequence is itself part of the task.
Law induction is more demanding still. Two reference videos share a generator category, and the model must decide whether a query belongs to that category. A category identifies both the altered factor or factor combination and its amount category. Recognizing “there is damping” is insufficient if the question requires matching how much damping there is.
What the matched comparisons reveal
Adding frames usually improves the full-set score, but the magnitude differs substantially across models. In the reported comparison, Qwen2-VL-7B gains 0.42 percentage points from simulator-ordered frames, while GPT-5.6 Sol gains 19.25. Under the smaller output-token limit, Qwen3-VL-32B has the largest visual gain, 14.13 points.
The comparison also separates chronology from the presence of images. Shuffling simulator frames lowers GPT-5.6 Sol's score by 12.15 points, compared with 0.03 for Qwen2-VL-7B. Those differences describe how useful ordering is for the measured questions; they should not be collapsed into a single claim that a model either does or does not understand time.
Model scale gives no uniform shortcut to this behavior. Qwen3-VL-32B leads the smaller members of its family in Simulator ordered, whereas Qwen2.5-VL-32B trails Qwen2.5-VL-7B by 0.17 points. Qwen2.5-VL-3B loses 3.25 points when frames are added. More parameters and more observations do not guarantee that the model will use the added frames effectively.
A small overall visual gain can conceal large opposing task changes. Qwen2-VL-7B improves on law identification and temporal reasoning while losing on analytic counterfactual comparison. Full-set scores weight each question equally; task-macro scores weight the five tasks equally. Both views are useful, but they summarize different distributions.
The familiar answer can survive contradictory motion
Overall accuracy does not identify the kind of mistake a model makes. InfiniteWorld therefore measures the Earth-normal error rate: among altered law-identification questions, how often does the model choose the answer appropriate to ordinary physics?
For Qwen3.8-27B, ordered simulator frames improve altered-scene accuracy relative to Text only. Yet the gap between Earth-normal and altered questions remains large, and the Earth-normal answer continues to dominate its responses on altered questions. Frames can help without fully overriding the prior.
Moving that answer to another option position exposes a further distinction. A model can repeat the same wrong physical explanation even when the answer's position changes. Semantic agreement measures this stability, while splitting agreement into correct and wrong pairs tells us whether the stability is desirable.
For Qwen3-VL-8B, Simulator ordered increases agreement over Text only, but repeated wrong answers remain more common than correct agreement. Shuffled frames yield slightly more agreement while lowering law-identification accuracy. An evaluation that rewarded consistency without checking its content would miss this difference.
A shared factor is not a shared amount
Law induction groups the questions into Match, Different factors, Earth-normal, and Different amount. Match requires Yes; the other groups require No. Reporting these groups separately exposes response preferences that a single accuracy number can hide.
GPT-5.6 Sol's strong overall law-induction performance combines high Match accuracy with poor Different amount performance under Simulator ordered. GPT-5.4 Mini shows another pattern: it rejects most nonmatching queries but also rejects most matching ones. These are different failure modes, even if an aggregate metric makes them look similar.
Separating familiar physics from observed motion
The main contribution is an experimental separation. By holding questions fixed and changing what can be observed, the benchmark distinguishes answer quality from visual gain, sensitivity to ordering, and sensitivity to rendering. By altering dynamics, it creates situations in which familiar knowledge and observed motion can lead to different answers.
The measurements retain important boundaries. Simulator-controlled factors cover a selected family of environments, not all physical reasoning. Frames sample a trajectory rather than provide continuous observation. A translated video may change question-relevant content. Generator-category matching is the operational target for law induction, rather than unrestricted discovery of a symbolic equation.
Text and option structure also carry information. The paper examines answer patterns and descriptive template-based references rather than treating Text only as a content-free chance baseline. Comparisons should preserve the same questions, input settings, answer extraction, and output-token grouping before interpreting differences between models.
The broader question is how an agent should revise its beliefs when the world disagrees with its expectations. InfiniteWorld makes that disagreement measurable. A capable physical reasoner must use prior knowledge to form a hypothesis, then let the observed motion determine whether that hypothesis survives.
