The LiDAR on your smartphone can now see around corners

Researchers have used weak reflected light to create rough three-dimensional renderings of hidden objects. A technology that once required extremely expensive laboratory equipment is now opening the way to entirely new forms of machine vision and spatial perception on affordable devices.

Photo: Aaron Young/MIT Media Lab

The LiDAR sensor, which some smartphones typically use to measure depth, has gained a whole new ability: detecting objects that are not directly in front of it. Researchers led by Siddhartha Somasundaram of the MIT Media Lab have shown that a smartphone-based LiDAR sensor can reconstruct hidden 3D objects, track movement around corners, and even help determine the camera's position in space using objects outside its direct field of view. The system doesn't create a sharp photo of a hidden space, but it does reconstruct rough shapes from extremely low light and detect movement.

This technology could in the future allow autonomous vehicles to detect pedestrians before the driver or camera sees them, help robots navigate in crowded spaces, and allow augmented reality headsets to track a user's hands even when they're out of their field of vision. Somasundaram pointed out that a capability that once required specialized imaging equipment has been successfully put into the hands of robotics and augmented reality developers.

The sensors work by emitting laser pulses and measuring the time it takes for the light to return. Modern consumer LiDAR sensors can measure time-of-flight in the picosecond range, which is enough to detect differences in centimeters. When imaging outside the line of sight, some of the light hits a wall or floor, scatters toward a hidden object, bounces back off the wall, and returns to the sensor. Although this signal is extremely weak, it still carries useful information.

The team used a portable LiDAR system with about 100 pixels, each of which combines a laser emitter and a single-photon detector. Because a single image was too noisy, the researchers combined multiple images using a technique called “motion-induced aperture sampling,” which took advantage of the natural movement of the device in the hand to combine the weak signals into a clearer image.

The system successfully demonstrated three main capabilities: reconstructing the 3D shapes of static hidden objects (such as a U-shaped object), tracking moving objects (including a bouncing ball around a corner), and using hidden objects as landmarks to determine the camera's position. The results were best with reflective objects, as typical diffuse surfaces return significantly less light.

The entire experiment was performed on off-the-shelf hardware that costs less than an estimated $83, and the researchers have made the code and data publicly available. The research results were published in the journal Nature, and Jessica Rosenworcel, executive director of the MIT Media Lab, also expressed her excitement over the achievement.