London-based Flawless AI’s ‘True Sync’ tech is a revolutionary approach to film dubbing

When British director Scott Mann’s latest film, Fall, was on the precipice of receiving an “R” rating from the MPAA over the number of “F” bombs dropped over its one hour and 47-minute run time, he did what any reasonable person would: he used artificial intelligence to digitally alter the actor’s performances in order to change the swear words into more palatable terms. A stroke of fricking genius, if you ask us.

For those who are curious: about 35 “F” words stood between a PG-13 rating and an R rating. Mann’s dilemma, then, became trying to figure out how to preserve the integrity of his movie without reshooting or dubbing. This is, of course, harder than it sounds.

On a traditional movie, you’d just bite the bullet and do reshoots. It’s costly, but it’s better than having your audience lose immersion because they’ve just heard a line delivered differently than what they clearly saw the actor say on screen. It’s a lot easier to read lips in 4K.

But Fall isn’t a traditional movie. It’s a horror movie about two women who climb up a 2,000-foot radio tower. I won’t spoil anything, but it was shot practically. That means they built a giant tower and filmed their actors on the top of it.

When you think about the logistics involved, it become apparent why reshoots aren’t really an option. Not only would they be even costlier than usual, but the vulgar language was, in many cases, ad-libbed by the actors due to the emotionality of the scene.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

This means Mann likely got the performances he was looking for, an accomplishment we can assume no director would willingly erase over something so trivial as some bad language. Unfortunately, R rated films typically only gross about half what their PG-13 counterparts due.

Luckily for Mann, he happens to be in a unique position among Hollywood directors. He’s the co-CEO and co-founder of Flawless AI, a technology company that specializes in digitally altering video to match an actor’s performance to a dubbed audio track.

For Fall, Mann and the Flawless AI team had to record the actors saying alternative phrases such as “fricking” to replace the film’s vulgar language. Then, they ran both the new audio and the video through the company’s bespoke neural network, or what they call “True Sync” technology. This allows the team to, essentially, deepfake the actors faces into saying the new lines.

As far as results go, with a “certified fresh” rating on Rotten Tomatoes, it would seem nobody noticed the AI at work when they watched the drama unfold on the big screen.

The system is not only cost-effective versus reshoots but, according to Mann, it’s also relatively easy to insert into production. He told TNW via email that about 80% of the workflow is automated and that it was designed to handle everything in post — it doesn’t require any special equipment or direction during film production. He even says that “vubbing,” or video dubbing, can save up to 50% in costs when it comes to reshoots.

But Flawless AI wasn’t just founded to help filmmakers save money in post-production. In fact, the company’s main focus is on accessibility. According to Mann, “Accessibility is super important for us because we believe in the ability to share stories as they were meant to be and preserve the performances of actors and how the directors planned for the story to be told.”

The company was founded after Mann watched one of his previous films, Heist, starring Rober Deniro, in traditional dubbed-over format. He found the status quo dubbing methods to be “limited and broken.”

“I saw how badly the traditional foreign dubbing harmed the movie, as the writing, performance and meaning were lost,” Mann told TNW. “This led me to reading a white paper from the Max Planck Institute on neural networks and ultimately led to the founding of Flawless.”

Going forward, Mann and the Flawless AI team hope their work will allow storytellers to reach audiences around the world in a more intimate way. It’s one thing to know what an actor is saying because you’re hearing dubbed audio in a language you understand, it’s an entirely different experience to see them deliver the line with all the nuance and character it was given in the original performance.

We tip our hat to Mann and the Flawless AI team for using AI for good. Art is for everyone.