AInsights: Executive-level insights on the latest in generative AI

Just when you’ve seen it all, there’s always something new that will surprise you, almost to the point, where you may lose the magic of surprise. We live in some incredible times, don’t we? As OpenAI co-founder and CEO Sam Altman said recently, “This is the most interesting year in human history, except for all future years.”

Well, I just read a research paper published by Microsoft Asia that blew my mind. 🤯 And as you can imagine, it takes a lot to blow me away!

The paper essentially introduces what it calls the VASA framework for generating lifelike talking faces with “visual affective skills” (VAS).

Its first iteration, VASA-1, is a real-time, audio-driven talking face generation technology. It can create lifelike animated faces that closely match the speaker’s voice and facial movements, with, get this, single portrait picture, a same of speech audio, control signals such as main eye gaze direction and head distance, and emotion offsets, create a real-time hyper-realistic talking head video…all with scarily convincing gestures.

Unless you knew the person, and even then, it would be difficult for the untrained eye to detect that they were watching a machine-produced video (or in some cases, a deepfake). 😳

AInsights

Certainly, Microsoft Research is exploring the boundaries for what’s possible with the best of intentions. So, in this piece, let’s focus on this technology with that perspective. From that point of view, key benefits and use cases of VASA-1 include:

Highly realistic and natural-looking animated faces: VASA-1 can generate talking faces that are indistinguishable from real people, enabling more immersive and engaging virtual experiences.

Real-time performance: The system can produce the animated faces in real-time, allowing for seamless integration into interactive applications, gaming, and video conferencing.

Broad applicability: VASA-1 has potential use cases in areas such as virtual assistants, video games, online education, and telepresence, where lifelike animated characters can enhance the user experience.

Potentially interesting use cases could include:

Virtual avatars and digital assistants: VASA-1 can be used to create virtual avatars and digital assistants that can engage in natural, human-like conversations. These avatars could be used in video conferencing, customer service, education, and entertainment applications to provide a more immersive and engaging experience.

Dubbing and lip-syncing: The ability to accurately synchronize facial movements with audio can be leveraged for dubbing foreign language content or creating lip-synced animations. This could streamline the localization process and enable more seamless multilingual experiences

Telepresence and remote collaboration: It can enhance remote communication and collaboration, allowing participants to maintain eye contact and perceive non-verbal cues as if they were physically present.

Synthetic media creation: VASA-1 could generate create highly realistic synthetic media, such as virtual news anchors or digital characters in films and games. This could open up new creative possibilities and streamline content production workflows.

Accessibility and inclusion: VASA-1 could improve accessibility for individuals with hearing or speech impairments, providing them with more natural and engaging communication experiences.

Microsoft Research Asia: Sicheng Xu*, Guojun Chen*, Yu-Xiao Guo*, Jiaolong Yang*‡, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, Baining Guo Microsoft Research Asia *Equal Contributions ‡Corresponding Author: jiaoyan@microsoft.com

Please subscribe to AInsights, here.

If you’d like to join my master mailing list for news and events, please follow, a Quantum of Solis.