AI multi-speaker lip-sync has arrived

For a long time, there has been a lack of synchronisation between lip movements and voices in dubbed content. Experts believe this is one of the reasons why dubbing is relatively unpopular in English-speaking countries. In fact, lip movements make localised content more realistic and therefore more appealing to audiences.

There is a study by Yukari Hirata, a professor known for her work in linguistics, which says that watching lip movements (rather than gestures) helps to perceive difficult phonemic contrasts in the second language. Lip reading is also one of the ways we learn to speak in general.

Today, with Rask’s new feature, it’s possible to take localised content to a new level, making dubbed videos more natural.

The AI automatically restructures the lower face based on references. It takes into account how the speaker looks and what they are saying to make the end result more realistic.

According to Maria Chmir, founder and CEO of Rask AI, the new feature will help content creators expand their audience. The AI visually adjusts lip movements to make a character appear to speak the language as fluently as a native speaker.

The technology is based on generative adversarial network (GAN) learning, which consists of a generator and a discriminator. Both the generator and the discriminator compete with each other to stay one step ahead of the other. The generator clearly generates content (lip movements), while the discriminator is responsible for quality control.

Unified Magazine provides expert insights on business, technology, and innovation. We partner with leading entrepreneurs to provide strategic advice and market analysis, empowering professionals and entrepreneurs with resources for success and growth.