Timezone: »

 
Poster
NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis
Jian Liang · Chenfei Wu · Xiaowei Hu · Zhe Gan · Jianfeng Wang · Lijuan Wang · Zicheng Liu · Yuejian Fang · Nan Duan

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #707
in Poster Session 5 »
Infinite visual synthesis aims to generate high-resolution images, long-duration videos, and even visual generation of infinite size. Some recent work tried to solve this task by first dividing data into processable patches and then training the models on them without considering the dependencies between patches. However, since they fail to model global dependencies between patches, the quality and consistency of the generation can be limited. To address this issue, we propose NUWA-Infinity, a patch-level \emph{``render-and-optimize''} strategy for infinite visual synthesis. Given a large image or a long video, NUWA-Infinity first splits it into non-overlapping patches and uses the ordered patch chain as a complete training instance, a rendering model autoregressively predicts each patch based on its contexts. Once a patch is predicted, it is optimized immediately and its hidden states are saved as contexts for the next \emph{``render-and-optimize''} process. This brings two advantages: ($i$) The autoregressive rendering process with information transfer between contexts provides an implicit global probabilistic distribution modeling; ($ii$) The timely optimization process alleviates the optimization stress of the model and helps convergence. Based on the above designs, NUWA-Infinity shows a strong synthesis ability on high-resolution images and long-duration videos. The homepage link is \url{https://nuwa-infinity.microsoft.com}.
Keywords: image synthesis · video synthesis

Author Information

Jian Liang
Chenfei Wu (Microsoft)
Xiaowei Hu (University of Alberta)
Zhe Gan (Microsoft)
Jianfeng Wang (University of Science and Technology of China)
Lijuan Wang
Zicheng Liu (Microsoft)
Yuejian Fang (Peking University)
Nan Duan (Microsoft Research Asia)

More from the Same Authors