Generative Point Tracking and Forecasting

Abstract

Motion forecasting predicts where points will move in the future, while motion tracking predicts where they are in the present. Despite these similarities, existing approaches to the two problems are quite different. In this paper, we propose a unified model that can address both tasks. We train a causal, video-conditioned flow matching model to predict point positions. The resulting model can easily toggle between point tracking and forecasting by changing its visual signal. Despite our model's simplicity, we find that it outperforms prior work in point forecasting and obtains performance that is competitive with the state-of-the-art on the TAP-Vid benchmark.

Qualitative Demo on DAVIS

We qualitatively demonstrate our method's forecasting ability in unstructured scenes. We train the model on the pseudolabeled Kinetics dataset and evaluate it on DAVIS. For this experiment only, we use a model at the DiT-B scale. In the examples below, the model tracks the first 8 frames and forecasts the next 16; see the bottom of the page for failure-case analysis.

GT Video Tracking (8F) + Forecasting (16F)

Qualitative Demo on DriveTrack

Our model can infer motion from scene context in real-world driving videos. In the examples below, the model forecasts future trajectories from a single conditioning frame.

Initial Condition GT Trajectory Forecasting

Qualitative Demo on Physics101

Our model can forecast complex, nonlinear dynamics in physical interaction videos.

Initial Condition GT Trajectory Forecasting

Method

We propose a single generative model that performs both forecasting and tracking using a diffusion transformer, operating with image conditioning for tracking and without image conditioning for forecasting. Noise is added to the point locations across time, and the model is trained to denoise these trajectories. Each diffusion block contains three types of attention: temporal attention, applied across the same point over time; spatial attention, applied among all points within the same frame; and cross attention, applied between point location tokens and image feature tokens.

Failure Cases

Our method exhibits failure patterns similar to video generative models: (a) it can struggle to fit highly complex motion; (b) forecasts may drift when extrapolating beyond the 16-frame training window; and (c) performance degrades on out-of-distribution data, such as animals.