Text-and-Image-Conditioned content GENeration (TICGEN)

Wargaming exercises are essential to preparing the U.S. Navy for large-scale combat with a near-peer adversary. However, constructing naval wargames means relying on a human-centric approach to predict potential adversary actions. This runs the risk of introducing to wargames human errors, biases, and omissions, potentially distorting the insights and analysis the Navy seeks to glean from wargaming. Thus, it is desirable to utilize AI-based immersive technology (e.g., augmented, virtual, mixed, and/or extended reality) to construct C5ISR environments for wargames. This in turn calls for leveraging generative AI to create time-varying digital content for use in immersive simulations. The effort we propose would contribute a component to the Virtual Sea Shield (VSS) content generation environment for wargame content.The proposed effort focuses on a specific aspect of VSS: generating static 3D assets that represent objects found in real-world images, using text prompts for controllable generation. This capability is a powerful tool in the context of 4D content generation for the naval domain. For example, using text-conditioning, we could ìupgradeî a 3D model of a scene to 4D by leveraging a text source providing updates on scene changes over time. This would enable seamless integration of text-based ISR reports with imagery to assess and simulate the state of the battlefield. Additionally, using image-conditioning, we could extract photorealistic shapes and textures from open-source imagery intelligence, enabling us to ìremixî scenes by recombining them in fresh orientations, resulting in novel situations that enhance the generality and adaptivity of constructed wargames.Any successful approach to the task of extracting models of domain-relevant objects from open-source imagery must utilize the massive amount of available open-source 2D image data while also leveraging components designed specifically for 3D generation (in particular, it must avoid naÔve use of VLMs to generate meshes). The approach must also simultaneously condition on text and images, whereas prior state-of-the-art approaches have conditioned solely on either text (e.g., MeshDiffusion) or images (e.g., ReconFusion).ÝWe present TICGEN, a novel neural architecture for Text-and-Image-Conditioned content GENeration. Inspired by DreamFusion and ReconFusion, the general structure of the proposed architecture is as follows: given images containing a common object, we first extract the relevant pixels from each image and separate these extracted images into sources and targets. We learn a Neural Radiance Field (NeRF) conditioned on the sources which faithfully reproduces the targets. This NeRF is regularized via a Diffusion that has been previously trained on internet-scale data. When provided source images as input, the NeRF can be exported to an explicit representation of the object of interest.