UniCam: Taming Unified Diffusion Models in Noise Space for Camera-controllable Video Rendering

Haoyu Zhao; Zuxuan Wu; Yu-Gang Jiang
Fudan University

* This project page contains a large number of videos, please wait patiently for them to load.

Abstract

Controlling camera pose in video diffusion models is essential for novel-view video rendering, yet existing approaches struggle to achieve precise control. Methods that directly inject numerical camera parameters into the diffusion backbone often fail to capture subtle viewpoint variations and lead to structural distortions or visual artifacts. To overcome these limitations, we propose UniCam, a unified framework that introduces a temporally coherent stochastic representation, termed CameraNoise, warped from camera intrinsic and extrinsic parameters. Unlike conventional approaches, CameraNoise embeds camera poses directly into the noise space. This makes our approach independent of scene appearance while faithfully encoding camera motion. Specifically, we introduce a novel Geometry-guided Reprojection Flow along with a CameraNoise warping algorithm, which jointly preserves the Gaussian prior of diffusion and ensures consistent noise propagation under camera transformations. By integrating CameraNoise into the diffusion process, the UniCam framework produces stable, high-quality videos with precise camera control across text-to-video, image-to-video, and video-to-video generation tasks. Extensive experiments on three public benchmarks demonstrate that our approach significantly outperforms prior methods in both fidelity and controllability.

[TODO #2]1. Camera-controllable Image-to-Video (I2V) Generation

[TODO #3] Our method transforms camera poses into CameraNoise, a noise-space representation of camera motion. This appearance-independent representation is introduced into the denoising process of the video diffusion model, enabling the generated videos to accurately capture camera motion trajectories.

[TODO #2]2. Warped CameraNoise from Camera Poses

[TODO #2]The proposed CameraNoise is an appearance-agnostic noise representation.

[TODO #3] You can pause the video at any frame to examine the Gaussian distribution of a single frame. At that moment, you can notice that the motion information in the video has disappeared.

[TODO #2]3. More Camera Controllable I2V Results

[TODO #3] Each video is generated with reference image, camera poses, and prompt.

[TODO #4]4. Camera Controllable Text-to-Video (T2V) Generation

[TODO #5] Each Videos is generated with a textual prompt and camera poses.

[TODO #4]5. More Camera Controllable T2V Results

[TODO #2]6. Camera Controllable Video-to-Video (V2V) Generation

[TODO #2]Within UniCam, we can easily change the camera motion in the original video.

[TODO #2]7. Overview of UniCam Framework

Grapefruit slice atop a pile of other slices
[TODO #3] Overview of our framework. We introduce CameraNoise, a controlled noise signal that encodes temporal correlations of camera poses into video diffusion. Our method is constructed via Geometry-guided Reprojection Flow (GRFlow) and a Gaussian-preserving warping algorithm, and injected into the video diffusion to enable precise viewpoint control. We use bold green arrows to illustrate the flow of control signals from camera poses to the synthesized video.

[TODO #2]8. Geometry-guided Reprojection Flow (GRFlow)

[TODO #2]A Reprojection of Camera Poses in 2D Grid.

[TODO #3] In this example, we show the GRFlows generated for the leftmost video under different alpha values in Eq. (5).

[TODO #2]9. Our Appearance-agnostic CameraNoise via GRFlow

[TODO #2]v.s.

[TODO #2]Appearance-motion entangled optical-flow-based noise

[TODO #3] * To enhance the visualization of motion and appearances in the noise, we speed up the video x2.
[TODO #3] Since optical flow inherently contains object contours and appearance information, the warped noise derived from it inevitably carries appearance priors. During inference in diffusion models, such information can conflict semantically with the noise prior and control conditions, ultimately leading to generation failure.

[TODO #2]10. Dynamic and ourdoor scenes with different camera poses.

Camera1: Move-Up Shot.

Camera2: Counterclockwise Rotation Shot.

Camera3: Move-Down Shot.

We demonstrate dynamic results across multiple scenes under three different types of camera motion. Given a reference image, camera poses, and textual descriptions, our model is able to generate the corresponding dynamic scenes.

[TODO #2]11. Dynamic single scene across different camera poses.

Scene1: A vibrant forest scene is filled with various birds flying, surrounded by trees, green mossy ground, and sunlight.

Scene2: A golden retriever stands in a sunlit grassy field, with trees and open green space in the background.

Scene3: A cowboy rides a horse along a winding dirt road through a golden and sunlit field with fences.

We showcase the dynamic motion of the same scene under different camera poses, demonstrating that our model exhibits strong robustness across various scenes. The camera poses used are sourced from the MultiCamVideo dataset.

[TODO #2]12. Dynamic vehicle driving scene.

Vehicle driving scenarios are among the most challenging dynamic scenes. We test our model of image-to-video generation using data from the DrivingDoJo dataset, and the results show that it can effectively handle common driving situations, including daytime, nighttime, straight driving, and turning.

[TODO #2]13. Camera pose transfer: source video to generated videos.

Given an input video, we can obtain its camera parameters using the VGGT model, and then convert these parameters into CameraNoise with our proposed algorithm to provide the model with camera control capability. We select two scenes from the MultiCamVideo dataset and transfer the camera motion under three different reference image conditions. The results demonstrate that our model can achieve lossless transfer of camera motion from the input video.

[TODO #2]14. Comparison with previous State-of-the-Arts.

[TODO #3] Since previous CameraCtrl, MotionCtrl, and Go-with-the-flow methods are trained on different datasets, we visualize their zero-shot results on the MultiCamVideo dataset to ensure a fair comparison. It is important to note that our focus is on controlling the camera motion in the scene, rather than dictating how the people within the scene move. GT means the ground truth.

We observe that all these methods exhibit varying degrees of degradation when applied to new scenes:

1) CameraCtrl shows declines in both camera control accuracy and visual content quality;

2) MotionCtrl almost completely loses its camera control capability in the new scenes;

3) Go-with-the-Flow suffers from a noticeable drop in visual content quality.

15. Comparison with State-of-the-Arts under OOD scenarios.

Camera pose type 1: move-up shot.

Camera pose type 2: move-down shot.

Camera pose type 3: move-left shot.

Camera pose type 4: move-right shot.

Camera pose type 5: move-clockwise shot.

Camera pose type 6: move-in shot.


We evaluate methods MotionCtrl, CameraCtrl, Go-with-the-Flow, GEN3C, and our CameraNoise in six typical out-of-distribution (OOD) scenarios: valleys, fields, lakes, deserts, forests, and amusement parks (see the reference images in the first column). For each scene, we test these methods using six representative camera motions provided by GEN3C.

Based on these results, we summarize the characteristics and limitations of current mainstream methods under OOD conditions:

1) MotionCtrl and CameraCtrl: exhibit large deviations in camera following, indicating limited robustness in camera control;

2) Go-with-the-Flow: prone to excessive camera motion and occasional content collapse;

3) GEN3C: produces static scenes where objects cannot move, resulting in rigid video content. Additionally, due to its reliance on 3D feature modeling, it is susceptible to scene penetration issues (e.g., camera pose 5);

4) Our method: demonstrates superior performance in OOD scenarios in terms of camera control accuracy, content consistency, and motion dynamics.