NYU and Intel Researchers launched Image Sculpting, a new artificial intelligence framework for editing 2D Images
The researchers at New York University(NYU) and Intel developed the AI-powered Image Sculpting method to address the limitations in 2D image editing by integrating 3D geometry and graphics tools. This approach allows direct interaction with the 3D aspects of 2D objects, enabling precise editing such as pose adjustments, rotation, translation, 3D composition, carving, and serial addition.
The research includes exploration into generative models like GANs, which have broadened the scope of image editing to encompass style transfer, image-to-image translation, latent manipulation, and text-based manipulation. However, text-based editing has limitations in precisely controlling object shapes and positions. ControlNet is one of the models that address this by incorporating additional conditional inputs for controllable generation. Single-view 3D reconstruction, a longstanding problem in computer vision, has seen advancements in algorithmic approaches and training data utilisation.
Although Image Sculpting presents promising capabilities, it faces limitations in controllability and precision through textual prompts. Requests regarding detailed object manipulation remain challenging for current generative models. The method relies on the evolving quality of single-view 3D reconstruction, and manual efforts may be required for mesh deformation. Output resolution falls short of industrial rendering standards, and addressing background lighting adjustments is crucial for realism. Despite its innovative approach, Image Sculpting represents an initial step, and further research is essential to overcome these limitations and enhance its overall capabilities.
