Back to Pixels: Lifting Latent Video Diffusion into Pixel Space

Overview

Frontier video generation models work in a VAE latent space that compresses video in both space and time. This compression made training tractable, but now limits high-resolution, controllable, and interactive generation – doubling resolution quadruples token count, conditions apply to groups of frames rather than individual ones, and each response must wait for the rest of its multi-frame block.

We present PixVid, which finetunes a pretrained latent video model to generate every frame directly in pixels. PixVid keeps the diffusion transformer and replaces the VAE with a convolutional head that renders its outputs into pixels. We find that predicting clean video rather than velocity, together with a much larger noise shift, is critical to quality.

Finetuned from Wan2.1-14B, PixVid reaches quality competitive with the original latent model within 20K iterations. Working frame by frame in pixels, it generates 2× higher-resolution video with the same number of tokens per frame, keeps depth-conditioned video editing aligned under fast motion, and responds to each action at the next frame.

Method

PixVid keeps the DiT backbone of a pretrained latent video model and replaces the VAE: a linear patch embedding maps each frame's pixels to tokens, and a causal convolutional detailer head renders the DiT tokens into pixels.

Lifting a latent DiT to pixel space

Use the arrows or click the figure to step through.

An offline linear pixel–latent alignment is fused with the pretrained patch embedding to initialize the pixel patch embedding, and a timestep calibration matches the pretrained signal-to-noise ratio. The DiT is initialized from the latent model, and a detailer head replaces the linear head and VAE decoder to predict clean frames directly.

Higher resolution at the same token count

To adapt from patch size p to 2p, 2×2 average downsampling is fused with the 1×p×p patch embedding, and a new up block L+1 (initialized as nearest-neighbor upsampling) is prepended to the detailer head. With the token grid unchanged, we train at 2× resolution at the same DiT cost.

Token grid at 480p and 960p

480p 832 × 480

30 × 52 tokens

One token: 16 × 16 pixels

Baseline cost

960p, latent model 1664 × 960

60 × 104 tokens

One token: 16 × 16 pixels

4× tokens, 16× attention vs. 480p

960p, PixVid 1664 × 960

30 × 52 tokens

One token: 32 × 32 pixels

Same tokens and DiT cost as 480p

Causal detailer head

DiT featuresnot generated yet

droppedcache

pixels, frame t

All 3D convolutions in the detailer head are causal in time (kt = 3): output frame t reads only frames t−2 to t. In frame-by-frame generation, the head caches features from the previous two frames and decodes each frame as soon as its DiT tokens arrive.

Low-latency autoregressive generation

An autoregressive model generates video in blocks, and a block can start only once the commands for all of its frames have arrived. In a temporally compressed latent model, each block spans at least one multi-frame latent, so new commands wait for the rest of the block; PixVid generates single-frame blocks, so each action takes effect on the next frame.

Control-to-screen latency

With temporal compression (N = 4 here), each generation step commits 4 frames to a single action and queues incoming actions until the block finishes, adding up to 133 ms of delay for a 4-frame block at 30 FPS. PixVid generates one frame per step, so every action is reflected in the next generated frame.

Responsiveness to abrupt camera controls

PixVid on camera paths with abrupt direction changes. The overlay shows the keys pressed on each frame.

Interactive demo

Press Start and steer with the arrow keys or W, A, S, D. The left model uses 4-frame blocks and the right model uses 1-frame blocks, both at 30 FPS.

Press Start, then hold the arrow keys

Latent model (4-frame blocks)

Lag behind input: 0.0 frames, 0 ms

PixVid (1 frame per step)

Lag behind input: 0.0 frames, 0 ms

The latent model samples input only at the start of each 4-frame block and holds that direction for all 4 frames, quantizing every key press to whole blocks. PixVid samples input every frame and applies each change on the next frame.

Results

Text-to-video at 960p

PixVid samples at 1664 × 960 with patch size 32, using the same 30 × 52 tokens per frame as the 832 × 480 model.

Text-to-video comparison

PixVid, Wan2.1-T2V-14B and PixelUMM on the same MovieGen prompt.

Depth-conditioned video editing

Each model re-renders the source clip from its per-frame depth and a caption. Latent (VACE-1.3B) is finetuned with the same control branch, data and settings as PixVid; VACE, Cosmos and Wan-Fun are used without finetuning.

Camera control

Top: one scene under three camera paths of increasing difficulty. Bottom: PixVid and SolarWM-5B on the same camera path.

BibTeX

@article{song2026back,
  title={Back to Pixels: Lifting Latent Video Diffusion into Pixel Space},
  author={Song, Yanke and Chen, Shuo and Rockwell, Chris and
          He, Guande and Liu, Nan and Xie, Desai and
          Gu, Tianpei and Park, Yong-Hyun and {V S}, Aryan and
          Menn, Dennis Yuren and Yang, Ming-Hsuan and Huang, Xun},
  journal={arXiv preprint arXiv:2611.17336},
  year={2026}
}