Facebook bets on realistic face tracked avatars as key to VR's future

Facebook calls the avatar system detailed this week 'Codec Avatars'. The work has been talked about at Facebook and Oculus developer conferences over the last few years.
Dozens of people are working at Facebook Reality Lab in Pittsburgh, Pennsylvania on research into ultra-realistic avatars that could redefine communication in the 21st century.

I spoke by phone with Yaser Sheikh, a Carnegie Mellon University associate professor running VR research in the city for Facebook since 2015. Sheikh offered context on a technical blog post released this week which details the company's research toward ultra-realistic avatars. I followed up with Facebook over email for clarification after the call.

"Facebook Reality Lab Pittsburgh can accommodate close to 100 people," the email explains. "To accommodate the team's growth, FRL Pittsburgh will be moving into a larger building in Pittsburgh later this year."

That suggests a big investment toward the future of the team. The group is also not the only one at Facebook researching more realistic and personalized avatars. To be clear, Facebook says this technology is "years away" from being realized in consumer headsets.

Facebook calls the avatar system detailed this week 'Codec Avatars'. The work has been talked about at Facebook and Oculus developer conferences over the last few years. These latest details, however, provide insight into the work and their path which could one day carry the research into a shipping product.
To create each avatar, Facebook uses a specialized capture studio consisting of 132 cameras pointed toward the center. After a few hours with a person making various facial expressions, the system generates a unique photorealistic avatar face. Another newly-built system captures bodies.

"Creating lifelike avatars currently requires capturing large quantities of high-quality audio and video of research participants using a highly-customized system of cameras and microphones in Facebook Reality Lab's Pittsburgh office," the email from Facebook explains. "This data is used to train AI systems that may one day allow users to quickly and easily build their Codec Avatar from just a few snaps or videos. "

The current process won't work with consumers for obvious reasons so a main focus of the research is to use machine learning to generate avatars of this detail. The potential is that one day your existing photos on Facebook or Instagram could be enough. I asked Sheikh if he'd tried feeding his own photos from Facebook to its system to see if he can build an avatar from it. He said they need to understand more about how light interacts with skin before trying.

"When we do such work, we want to make sure it's with the consent of the user," he added.

Made on
Tilda