In the animation industry, 3D modelers typically rely on front and back non-overlapped concept designs to guide the 3D modeling of anime characters. However, there is currently a lack of automated approaches for generating anime characters directly from these 2D designs. In light of this, we explore a novel task of reconstructing anime characters from non-overlapped views. This presents two main challenges: existing multi-view approaches cannot be directly applied due to the absence of overlapping regions, and there is a scarcity of full-body anime character data and standard benchmarks. To bridge the gap, we present Non-Overlapped Views for 3D Anime Character Reconstruction (NOVA-3D), a new framework that implements a method for view-aware feature fusion to learn 3D-consistent features effectively and synthesizes full-body anime characters from non-overlapped front and back views directly. To facilitate this line of research, we collected the NOVA-Human dataset, which comprises multi-view images and accurate camera parameters for 3D anime characters. Extensive experiments demonstrate that the proposed method outperforms baseline approaches, achieving superior reconstruction of anime characters with exceptional detail fidelity. In addition, to further verify the effectiveness of our method, we applied it to the animation head reconstruction task and improved the state-of-the-art baseline to 94.453 in SSIM, 7.726 in LPIPS, and 19.575 in PSNR on average.
The overall pipeline of NOVA-3D. NOVA-3D utilizes front and rear viewpoint images as input. The dual-viewpoint encoder extracts features from the images, which are then used by the generator to produce two tri-planes. The tri-planes are sampled to obtain sampling features, and the direction-aware attention module is employed to fuse the features. Finally, the reconstruction loss and GAN loss modules are used to calculate the overall loss.
360-degree displays of the head of 3D anime character model.
360-degree displays of the full-body of 3D anime character model.
Left: Some examples of front and back images, where the front image contains more high-frequency information and the back image contains more low-frequency information. Right: Directly concatenating the features of the front and back view will lead to ghost face in the back view.
3D reconstruction visualization.
Left: NOVA-3D can generate novel views with richer details that are closer to the appearance of real characters. Right: NOVA-3D eliminates the symmetrical ghosting phenomenon.
Here are some anime characters from various works in the ACG (Anime, Comic and Game) field showcased in the NOVA-Human dataset. It includes well-known characters in the ACG community such as Hatsune Miku, Uzumaki Naruto, Kiana, Ram, Megumin, and Kamado Nezuko.
Here we present the training data sampled from a 3D anime character model in NOVA-Human. On the left are 16 randomly sampled views, while on the right are four fixed orthogonal sampled views.
Here are the anime characters with a futuristic or near-futuristic Cyberpunk style showcased in the NOVA-Human dataset.
Here are the anime characters with a Gothic style showcased in the NOVA-Human dataset.
Here are the anime characters with a Dark Horror style showcased in the NOVA-Human dataset.
Here are the anime characters with different body types and races showcased in the NOVA-Human dataset, including Beast ear Niang, Elf, Demons, Robots, and more.
Here are anime characters with niche styles showcased in the NOVA-Human dataset, including various unique and unconventional characters.
Here we showcase additional sampled views from the NOVA-Human dataset. Our dataset encompasses a diverse range of 3D anime characters.
