“How to See the World’s Reflection From a Bag of Chips: Computer Scientists Reconstructed the Image of a Whole Room Using the Reflection from a Snack Package. It’s Useful for AR/VR Research—And Possibly Spying”, 2020-03-26 (; similar):
Technically speaking, the researchers didn’t actually use chips; they reconstructed a room using a Korean brand of chocolate-dipped corn puffs called Corn Cho. But whether it’s corn puffs or potato chips, the snack bag acts like a bad, warped mirror. A heavily-distorted reflection of the room is contained in the glint of light that bounces off the bag, and the team developed an algorithm that unwarps that glint into a blurry but recognizable image. In one instance, the researchers were able to resolve the silhouette of a man standing in front of a window. In another, the bag reflections allowed them to see through a window to the house across the street clearly enough to count how many stories it had. The algorithm works on a variety of glossy objects—the shinier, the better. Using the sheen of a porcelain cat, for example, they could also reconstruct the layout of the surrounding ceiling lights.
…To reconstruct the environment, the researchers used a handheld color video camera with a depth sensor that roughly detects the shape and distance of the shiny objects. They filmed these objects for about a minute, capturing their reflections from a variety of perspectives. Then, they used a machine learning algorithm to reconstruct the surroundings, which took on the order of two hours per object. Their reconstructions are remarkably accurate considering the relatively small amount of data that they used to train the algorithm, says computer scientist Abe Davis of Cornell University, who was not involved with the work.
The researchers could achieve this accuracy with so little training data, in part, because they incorporate some physics concepts in their reconstruction algorithm—the difference between how light bounces off shiny surfaces versus matte surfaces, for example. This differs from typical online image recognition tools in use today, which simply look for patterns in images without any extra scientific information. However, researchers have also found that too much physics in an algorithm can cause the machine to make more mistakes, as its processing strategies become too rigid. “They do a good job of balancing physical insights with modern machine learning tools”, says Davis.
…However, some experts caution that future versions of the technology are ripe for abuse. For example, it could enable stalkers or child abusers, says ethicist Jacob Metcalf of Data & Society, a nonprofit research center that focuses on the social implications of emerging technologies. A stalker could download images off of Instagram without the creators’ consent, and if those images contained shiny surfaces, they could deploy the algorithm to try to reconstruct their surroundings and infer private information about that person. “You better believe that there are a lot of people who will use a Python package to scrape photos off Instagram”, says Metcalf. “They could find a photo of a celebrity or of a kid that has a reflective surface and try to do something.”