People moving in and out of photographs used to be reserved for the world of Sci-Fi and the likes of Harry Potter.
“This is a very hard fundamental problem in computer vision,” said co-author Ira Kemelmacher-Shlizerman, an associate professor at the UW’s (University of Washington) Paul G. Allen School of Computer Science & Engineering. “The big challenge here is that the input is only from a single camera position, so part of the person is invisible. Our work combines technical advancement on an open problem in the field with artistic creative visualization.”
The algorithm, Photo Wake-Up, enables a 2D image of a person from a picture or a work of art to run, walk or jump out of the frame. The system also allows users to view the animation in three dimensions using augmented reality (AR) tools. The researchers presented their results on June 19 at the Conference on Computer Vision and Pattern Recognition in Long Beach, California.
How is it done?
Photo Wake-Up starts by identifying a person in an image and making a mask of the body’s outline. The algorithm then matches a 3D template to the subject’s body position. With this algorithm, the 2D image is warped to look like the person in the photo in 3D. The algorithm then manipulates the picture between 2D and 3D to present the desired effect.
The software then records each pixel of the warped 3D information, its distance from the camera or artist and how a person’s joints are connected. The algorithm then color match the 3D template with the original image on the 2D sample. The color matching involves front and back presentation “stitching” to make a 3D person who will be able to turn around.
To create the perfect movement away from the original image, the background is filled by Photo Wake-Up as not to leave a blank space and pixel match it to the rest of the image.
“Photo Wake-Up is a new way to interact with photos,” Weng said. “It can’t do everything yet, but this is just the beginning.”
This research was funded by the National Science Foundation, UW Animation Research, UW Reality Lab, Facebook, Huawei, and Google.
