JAWS: Just A Wild Shot
for cinematic transfer in neural radiance fields
CVPR2023
Xi Wang*,1
Robin Courant*,1, 2
Jinglei Shi3
Eric Marchand1
Marc Christie1
Inria, IRISA, CNRS, Univ Rennes1
VISTA, LIX, Ecole Polytechnique, IP Paris2
VCIP, CS, Nankai University3
* Equal contribution

[Paper]
[Code]

Abstract

This paper presents JAWS, an optimization-driven approach that achieves the robust transfer of visual cinematic features from a reference in-the-wild video clip to a newly generated clip. To this end, we rely on an implicit-neural-representation (INR) in a way to compute a clip that shares the same cinematic features as the reference clip. We propose a general formulation of a camera optimization problem in an INR that computes extrinsic and intrinsic camera parameters as well as timing. By leveraging the differentiability of neural representations, we can back-propagate our designed cinematic losses measured on proxy estimators through a NeRF network to the proposed cinematic parameters directly. We also introduce specific enhancements such as guidance maps to improve the overall quality and efficiency. Results display the capacity of our system to replicate well known camera sequences from movies, adapting the framing, camera parameters and timing of the generated video clip to maximize the similarity with the reference clip.



Overview video



Gallery

Vertigo effect - Jaws, 1975




Rotating - House of Cards, 2013




Crash zoom - Kill Bill: Volume 2, 2004



Stable push-in - The Matrix Reloaded, 2003



Handheld push-out - Inception, 2010


Ablation

[Left: Result | Right: Reference]
Animation timing




Focal length




Paper and Supplementary Material

JAWS: Just A Wild Shot
for cinematic transfer in neural radiance fields

Wang Xi*, Robin Courant*, Jinglei Shi, Eric Marchand
and Marc Christie
In Conference, CVPR 2023.


[Paper]
[Supp]
[Bibtex]
[Code]


Acknowledgements

We extend our gratitude to Anthony Mirabile and Nicolas Dufour for their invaluable contributions in constructing the 3D scene and proofreading. In addition, we thank the anonymous reviewers for their time and valuable comments.
Finally, thanks to Phillip Isola and Richard Zhang for the project page template; the code can be found here.