Snap Researchers Introduce NeROIC for Object Capture and Rendering Applications
AI researchers at Snap have collaborated with the University of Southern California to introduce a path-breaking neural application called NeROIC.
NeROIC stands for Neural Rendering of Objects from Online Image Collection. This machine learning-trained model acquires and analyzes object representations in 2D and 3D frames to enable various types of object-centric rendering applications. Let’s examine NeROIC and how its application can simplify the image acquisition and rendition processes.
What is NeROIC?
NeROIC is an innovative two-stage model to acquire object representations from online image collections.
Stage 1:
The first stage model captures images of an object from various angles as ‘input.’ Then, with the help of a camera, different poses of the object are created and trained using Neural Radiance Fields-based networks (NeRF). Using density functions, the model then computes the surface normal across normal extraction layer to come up with image in a neural rendition plane.
The NeRF model is optimized in various conditions and then decoupled in stage 2 to improve rendition and training capabilities of the novel view.
Stage 2:
The image rendition is done by applying lighting conditions and synthesizing the novel views in different environment and lighting conditions.
This process is explained in the paper here:
Why Use NeROIC for Image Rendition?
The internet is flooded with images and videos. Website creators selling products have a hard time describing the features and dimensions. NeROIC model will solve the various problems that image creators and website owners face in describing and rendering similar items in different environment and backgrounds.
NeROIC uses NeRF for novel view synthesis, enabling a superior object capturing and object rendition approach.
The model utilizes the PyTorch framework, trained on 4 NVIDIA V100s with the batch size of 4096. The model is then tested on a single NVIDIA V100.
By building NeROIC, AI researchers have demonstrated the role of neural networks and novel view synthesis in image capture, composition, and relighting approaches for multi-level image rendition using AI ML. This opens new avenues for AI ML tools specifically built for online image collections that are cropped or captured from different environment and backgrounds.
To access GitHub code for NeROIC project, c*********
Project authors:
ZHENGFEI KUANG∗, University of Southern California, USA
KYLE OLSZEWSKI, Snap Inc., USA
MENGLEI CHAI, Snap Inc., USA
ZENG HUANG, Snap Inc., USA
PANOS ACHLIOPTAS, Snap Inc., USA
SERGEY TULYAKOV, Snap Inc., USA
Comments are closed.