李修-China3DV-2025.pdf

1、Bridging 3D Generation and Video Generation by Modeling MotionXiu LiBytedance SeedMini3DVWith the advancing of video generation models,is 3D still important?2024/10/27More important,for World ModelWhat is World ModelWorld model is the PDE model of worldis the system inputAutoregressive LLM sets a go

2、od exampleModeling the cause and dynamics of world variableshttps:/ models are generative AI models that understand the dynamics of the real world,including physics and spatial properties.They use input data,including text,image,video,and movement,to generate videos.Are world models simply video gen

3、eration models?Or something else?F(t,x(t),y(t),y(t),y(n)(t)=0 x(t)y(t)is the observation we are interested inSolving this PDE is estimatingy(t)y(0)with initial valueand continuous system input x(t)A Picture of World Model1.Modeling the states of world,kinematics and dynamics(dynamics of 3D parameter

4、s)2.Must be able to take action input,be interactive.3.Able to predict next state by understanding the reason that causing it.World Model as the Reasoning Model of Autoregressive Video Modely(0)y(0)y(1)y(1)y(t)y(t)x(t)y(t+1)y(t+1)Past world 3D statesNext world 3D statesActionVideo ObervationReasonin

5、gNext FrameMonstr3RF(t,x(t),y(t),y(t),y(n)(t)=0ReasoningQARStep.1 2D Next Frame Prediction without Reasoningy(0)y(0)y(1)y(1)y(t)y(t)x(t)y(t+1)y(t+1)Past world 3D statesNext world 3D statesActionVideo ObervationNext FrameAutoregressive Video Generation for Portrait Animation Tokenization Next Motion

6、Prediction Tokenize portrait video to ID token+motion tokens Asymmetric Encoder-Decoder Design Various regularization VQ on motion tokenID FrameN FramesMotion EncoderIDEncoderDiffusion Decoder(Per-frame)IDOther Prompts(text,audio)Autoregressive TransformerAutoregressive Video Generation for Portrait