Copiloting Creative 3D Scene Modeling and Visualization with Generative Agents

Creating 3D scenes has been an important task in digital content creation with applications including interior designs, game design, visual effects, product marketing, etc. Recent advancements in generative AI have brought potential improvements to 3D scene creators, as significant progress has been made in AI-assisted automated design and 3D content generation.New developments in AI-assisted 3D content generation include text-to-3D mesh generation, procedural model generation, and room layout generation.However, although current systems can assist 3D content creators in labor-intensive tasks, few research studies have been conducted to improve 3D creation workflow and inspire idea explorations.In this work, we present a novel copilot system that leverages language and image-generative agents to allow high-quality and iterative 3D scene modeling and visualization and accelerate the creation of creative 3D content.Compared to traditional 3D modeling and visualization workflows, our system provides more natural and intuitive control and can significantly reduce the time required to create 3D content.