T2-z: Test-Time Optimization of Environment Latents for Robotic World Models

arXiv:2610.22364v1 | 2026-10-17

Problem

Predicting how the world responds to an action requires more than what can be seen: robots must account for environmental factors that are neither directly observable nor explicitly specified, such as friction, mass, and human-defined rules. Most robotic world models assume full observability or rely on pre‑specified system parameters, which limits their ability to generalize to new environments or cope with unmodeled dynamics.

Method

We introduce T2‑z, a test‑time optimization method that augments a robotic world model with a learnable environment latent. During training, T2‑z alternates between two stages: (1) fixing the latent and learning general action‑conditioned dynamics across environments, and (2) freezing the world model and optimizing the latent to capture environmental variation purely from prediction errors. At test time, the world model remains frozen and a newly initialized environment latent is optimized from one or more interaction trajectories, adapting the model to previously unseen configurations.

Findings

Across seven simulated settings and four real‑world robotic tasks, T2‑z reduces the object prediction error metric from 15.20 to 7.00 and raises action success from 33.73% to 72.96%. Ablations show that effective latents can be inferred from as little as a single test‑time interaction trajectory. The emergent latents capture structured representations of individual and compounded factors (for example friction and mass) across diverse visual scenes.

Limitations

The method is evaluated on a limited set of simulated and real‑world tasks; performance may vary with more complex dynamics or higher‑dimensional latent spaces. The approach assumes the world model architecture is sufficiently expressive to benefit from latent conditioning, and does not address potential optimization instability during test‑time latent search.

Practical Significance

T2‑z provides a lightweight, online adaptation mechanism that enables robots to quickly adjust their internal dynamics model to new environments without requiring explicit system identification or retraining. This improves robustness and sample efficiency for model‑based control in uncertain, changing real‑world settings.

Interactive Walkthrough: optimizing an environment latent at test time

This toy animation illustrates the mechanism described in the paper, not a reported rollout. A frozen world model keeps predicting futures, while a compact latent is updated from prediction error until it better explains the hidden environment.

Paused · stage 1 of 4.

Toy diagram of T2-z test-time latent optimization A frozen world model predicts a next state from an observation, action, and environment latent. As stages advance, a prediction-error arrow updates the latent, and the predicted ball trajectory shifts to match a heavier-friction environment. Toy latent-optimization loop Observed interaction same push, different outcome Frozen world model shared dynamics stay fixed latent zₑ Predicted next state target outcome prediction error updates zₑ Reused latent now conditions planning in this environment. Stage 1/4: initialize a new environment latent for an unseen setup. Stage 2/4: the frozen world model predicts the wrong next state because the hidden factor is not captured yet. Stage 3/4: optimize only the latent using prediction error from observed interactions. Stage 4/4: reuse the optimized latent so the same frozen model predicts this environment more accurately.
The paper's key move is test-time latent optimization, not retraining the world model: the shared predictor stays frozen while a compact environment code adapts to friction, mass, or other hidden factors.