Recommendation algorithms that power Amazon, Netflix can improve satellite imagery, too

Algorithms that help consumers decide what to stream or buy online can do more than predict customers’ habits: They can help satellites see the Earth better, according to a Rutgers study.

Optical satellites lose sight of the Earth’s surface when it is covered by clouds, and researchers have long relied on inaccurate tools to fill the blind spots, particularly along coastlines. By adapting a recommendation algorithm first built for Netflix, Ruo-Qian (Roger) Wang, an assistant professor of civil and environmental engineering at the Rutgers School of Engineering, created a system that is more accurate and faster at predicting cloud-covered landscapes in coastal areas than conventional data-filling tools.

The findings are published in the International Journal of Applied Earth Observation and Geoinformation.

“E-service platforms like Alibaba and Amazon use recommender systems, which leverage large datasets to provide customized product recommendations to aid customers’ decision making,” Wang said. “Interestingly, the way recommender systems process data is not unlike the process for predicting coastal landscapes obscured by clouds.”

In the open ocean, cloud-filling algorithms used in remote sensing measure continuous data—such as water temperature, color and algae content—to make predictions of what is hidden. But these solutions break down along the coast, where “errors are magnified due to increased cloud cover, vegetation and other variables,” said Wang, adding recommender systems “could do a better job in this objective.”

To test his hypothesis, Wang built a cloud-filling model atop the work of Simon Funk, a software developer who won a Netflix recommending tool competition. The algorithm, called Funk-SVD, plots consumer reviews on a matrix. This data is then used to predict the viewing habits of users who did not record a review.

It’s a similar process for cloud filling: Each coordinate on a map is represented by a pixel on a photograph and that pixel can either be water or land, with clouds representing unrecorded data. Wang’s adaptation of Funk-SVD makes guesses on what is beneath the clouds based on other data points.

Using an image database of 258 frames derived from Landsat missions in the Delaware Bay, Wang trained Funk-SVD to complete the cloud-filling scheme. His solution was more accurate than the most widely used cloud-filling tool, DINEOF (Data-Interpolating Empirical Orthogonal Functions), and achieved similar accuracy to another popular tool, Datawig, which is powered by machine learning. While Datawig uses vast computer power and can take days to process, Wang’s solution took 30 seconds.

Wang said his solution has numerous applications for long-term Earth observation. The method could be used to measure crop production, for instance, or to map the pace of urbanization across large areas. It can do so faster and cheaper than conventional methods, too.

“Any general land-use change could be monitored using this tool,” said Wang.