From the stage of F8, Joaquin Quinonero, Facebook’s Director of Applied Machine Learning, described a new technique the company is using to improve the watching experience for 360 videos. The format is challenging to deliver because of its size, but Facebook is using machine learning to reduce the number of pixels that have to be rendered at any one time. By predicting where a viewer will look next, rendering priority can be given to that location  — particularly helpful for users with lower quality internet access.

The status quo for 360 videos is reactive rather than proactive rendering. Mike Coward, engineering director for Facebook’s VR video team echoed the frustration of users to me when he described the unpleasantness of turning your head in VR only to see a blurry scene.

One partial fix is to optimize compression. But teams at the company are already using machine learning to select across the thousand-plus compression techniques for individual snippets of video. The other way to reduce the streaming load is to just cut down on what you’re rendering. And rather than reduce quality across the board, Facebook’s approach improves resolution for exactly what you’re most likely to look at next.

Mike Coward, engineering director for Facebook’s VR video teamStep one was to use the resources of the company to monitor where people actually do look when watching 360 videos. Facebook’s VR video team created a heat-map that highlighted the most popular spots that users looked at within videos. From there, Facebook built a generative saliency map using a deep neural network. This model makes it possible to perform predictions on new videos that haven’t previously been watched or studied.

If a human were to be given the task of predicting where someone might look, they might study their natural environment and look for anomalies that could catch one’s interest — think birds or a car driving by.

Abstracting away to the neural net, the physical cars and birds cease to matter. Facebook’s model was trained on a massive corpus of videos to identify interesting subsets of a video frame. Coward told me that the model, when faced with a surfer in the ocean, is capable of picking selecting the surfer as most interesting, despite the fact that both are fast moving entities.

After implementing the prediction model, Facebook was able to increase resolution by 39 percent on VR devices. Aside from improving resolution and making 360 videos accessible to people without great network connections, the technology could some day make it possible to offer preemptive suggestions to creators on how to make videos more engaging.

Today Facebook open sourced Caffe2. The deep learning framework follows in the steps of the original Caffe, a project started at the University of California, Berkeley. Caffe2 offers developers greater flexibility for building high-performance products that deploy efficiently.

This isn’t the first time that Facebook has engaged with the Caffe community. Back in October, Facebook announced Caffe2Go, what effectively was a mobile CPU and GPU optimized version of Caffe2 (they even both have Caffe2 in their names if you parse it right). Caffe2Go received attention at that time because its release coincided with Style Transfer.

Notably, the company also released extensions to the original Caffe. The majority of these changes make Caffe more attractive to developers building services for large audiences. For projects where resources are of no consequence, Facebook has historically turned to Torch — a library it finds optimal for research use cases.

Every tech company wants to tout the scalability of its machine learning framework of choice. I asked Yangqing Jia, the lead author on Caffe2, what he thought of MXNet and the noise Amazon has been making about its ability to scale. Reasonably, he was cautious about dropping benchmarking numbers for comparison. These numbers can have meaning, but they are heavily influenced by the actual implementation of a machine learning model and subject to a fair amount of “DIY” volatility.

Yangqing Jia, the lead author on Caffe2 and Alex Yu, leader of business development

“All frameworks are more or less at a similar scalability factor,” explained Jia. “We’re pretty confident that Caffe2 is probably a little bit better than the rest.”

Facebook is pouring a lot of resources into both Caffe2 and PyTorch. Today’s release accompanies partnerships at the hardware, device and cloud levels. Alex Yu, leader of business development for Caffe2, explained to me that Facebook aimed to include the market leaders in each category. This meant Nvidia and Intel on the hardware side, Qualcomm on the device side and Amazon and Microsoft on the cloud side. And while Google wasn’t targeted, a GCP partnership wouldn’t be out of the question going forward.

Prior to release, Caffe2 was deployed at scale across Facebook. The team also took considerations for the developer communities familiar with the original Caffe. Caffe models can be easily converted to Caffe2 models with a utility script. Facebook is releasing documentation and tutorials and has put Caffe2’s source code on GitHub.