AI dubbing and localisation for video: what it actually does

Learn in your browser · Chapter 01 of 06

AI dubbing and localisation get talked about as if they are one simple button: upload a video, pick a language, done. This chapter breaks that claim into what is actually happening underneath, and how it differs from subtitling.

AI dubbing replaces a video's original speech with new, AI-generated speech in another language, timed to fit the original clip. It sits inside the wider job of localisation: adapting a piece of content, video included, so it makes sense and feels natural to a different language or region. Dubbing is one route into localisation. Subtitling is another: it adds translated text on screen and leaves the original audio untouched. Neither is inherently better. Subtitling is faster, cheaper and keeps the original performance intact, but it asks the viewer to read while watching. Dubbing replaces the audio, which suits video where reading is inconvenient, such as a phone-scrolled short or a training video played in the background, but it asks more of the translation and the voice.

An AI dubbing tool is not one piece of software doing one job. It is a short pipeline of separate steps, usually run one after another: transcribe the original speech to text, translate that text into the target language, generate new spoken audio from the translated text (either a generic synthetic voice, or one built to resemble the original speaker), stretch or compress that new audio so it roughly fits the original clip's timing, and, in some tools, adjust the speaker's visible mouth movements so they look closer to the new words. Each of those five steps can go wrong on its own, and a mistake early in the chain, in the transcription or the translation, carries through to every step after it.

A worked example (invented). A small business has a three-minute English product demo video on its website and wants a French version for French customers, without hiring a studio or a bilingual voice actor. Run through an AI dubbing tool, the video's English narration is transcribed, translated into French, and spoken again by a synthesised voice timed to the original three minutes. The business gets a usable French video in minutes rather than weeks. What it does not get, without checking, is any assurance that a French idiom landed correctly, that a product name was not mistranslated, or that the new voice's timing matches the on-screen product shots exactly. That gap between usable and finished is the subject of this whole workbook.

Learn in your browser · Chapter 02 of 06

Getting from an English video to a usable French, Spanish or Japanese one depends on three separate steps working well together: transcription, translation, and fitting the result back into the original timing. This chapter looks at where each one commonly breaks.

Transcription turns spoken audio into text, and it is the foundation everything else is built on. It struggles with the same things speech recognition has always struggled with: background noise, overlapping speakers talking over each other, strong or unfamiliar accents, and technical terms, brand names or people's names it has never encountered. A single mistranscribed word early in the pipeline gets translated, spoken and lip-synced as if it were correct, because nothing downstream knows to question it.

Machine translation has genuinely improved, but it still translates meaning far more reliably than it translates feel. A literal, grammatically correct translation can still land wrong: a joke that depends on wordplay in the original language, an idiom with no equivalent, a tone that reads as too formal or too casual for the new audience, or a brand or product name translated when it should have stayed as-is. None of these are transcription errors. The words were all correctly translated. The judgement about what to leave alone is what a machine translation step does not reliably make on its own.

Once the new language's audio exists, it rarely lasts exactly as long as the original. Languages differ in how many syllables they need to say the same thing, so a line that took four seconds in English might need five in Spanish or three in Japanese. Fitting the new audio back into the original clip's timing is called isochrony, and most tools handle it by speeding up or slowing down the generated speech slightly, within limits, before it starts sounding unnatural. A translation that is accurate but too long or too short for isochrony to fix gracefully is a real, ordinary failure mode, not an edge case.

Scroll sideways to see every column.

Learn in your browser · Chapter 03 of 06

Getting the words and the voice right is only half the job on video. The mouth on screen was filmed saying the original language. This chapter covers what lip-sync adjustment tools try to do about that, and where it still looks obviously wrong.

Traditional human dubbing studios solve the mouth-mismatch problem without touching the video at all: a lip flap pass rewrites the translated script so its words roughly match the rhythm and mouth shapes of the original performance, close enough that a viewer's brain fills in the rest. Some AI tools take a different route: lip-sync adjustment, sometimes called visual dubbing, uses a video model to reshape the speaker's mouth and jaw region on screen so it looks closer to the new audio, rather than adapting the script to fit the original mouth.

Certain sounds are consistently the hardest for either approach. Sounds made with fully closed lips, such as 'm', 'b' and 'p', need the mouth to actually close. If the new audio needs one of these sounds where the original video's mouth was open, the mismatch is unusually visible, because a closed-mouth sound spoken with an open mouth looks obviously wrong even to a casual viewer, not just an attentive one.

Scroll sideways to see every column.

Learn in your browser · Chapter 04 of 06

Marketing for these tools tends to show its best case. This chapter is the honest version: what genuinely works well today, and what still needs a person checking it before you publish.

AI dubbing tends to work well on short, clear, single-speaker content in a common language pair: a straightforward product demo, an internal training video, a factual explainer, or narration with little overlapping dialogue or slang. The clearer and more literal the source material, the less there is for translation and lip sync to get wrong.

• Overlapping dialogue or several people talking at once
• Strong regional accents or fast, casual speech
• Humour, idiom and culturally specific references
• Emotional nuance and pacing, such as a joke's timing or a serious pause
• Brand names, technical terms and people's names
• Tight lip sync on close-up or side-profile shots

None of this means the tools are not useful. It means the honest claim is 'AI dubbing gets you most of the way, quickly', not 'AI dubbing replaces a human review pass'. A human review pass, someone who speaks the target language checking the transcript, translation, voice and final clip before it goes out, is still the difference between a usable draft and a video you would put your name to.

Scroll sideways to see every column.

Learn in your browser · Chapter 05 of 06

Search for AI dubbing and you will meet some striking numbers: a market worth billions, cost cuts of most of a traditional budget, adoption rocketing in a few years. This chapter shows you exactly where three of those numbers actually come from, and how to read the next one you meet.

The AI dubbing tools market was valued at $1.35 billion in 2026 and is forecast to reach $2.56 billion by 2030, growing at roughly 17.7% a year, according to a report from The Business Research Company, a market-research firm. That figure appears on the firm's own public summary page for a report that is otherwise paywalled, and it is repeated, without independent checking, on at least one dubbing-tool vendor's own site. A market-size estimate like this is a genuine attempt to size a real, growing market, but it is still one firm's modelled estimate, not a measured fact, and this workbook has not read the full paywalled report behind it.

Vozo, a company that sells AI dubbing software, publishes a statistics page reporting cost reductions of roughly 60% to 90% against traditional studio dubbing, and a separate page giving per-minute pricing examples as low as $1 to $10 for AI dubbing against $50 to $200 for human dubbing. Vozo also claims elsewhere on its site to be 'trusted by 7 million+ creators and companies', which is its own user count, not a market-wide measure. Novadub, a different vendor, claims a far bigger cost drop on its own blog (roughly EUR 50 to 100 or more per minute for traditional dubbing against about EUR 1 to 2 per minute for AI dubbing), and separately states that creator adoption rose from about 5% in 2023 to about 35% in 2026, projecting roughly 70% by 2028, with no methodology or source given anywhere on the page.

The general skill this chapter is teaching is not specific to dubbing. Whenever a tool's marketing quotes a striking market size, cost saving or adoption figure, ask four questions before you repeat it. Who published it, and do they sell the product it flatters? Is a source or methodology actually cited, or does the number just appear on the page? Is the underlying report something you, or anyone, can actually check, or is it paywalled and taken on trust? And does the figure describe the whole market, or just that one vendor's own customers?

Learn in your browser · Chapter 06 of 06

Everything so far has been about quality. This closing chapter is about the two things quality alone does not cover: whether you had the right to clone a voice, and whether your audience knows what they are watching. It ends with a practical checklist and your own pilot.

Voice cloning for legitimate dubbing and voice cloning for a scam use the same underlying technique, covered from the other side in Deepfakes and misinformation: how to tell what is real. What separates them is not the technology. It is two things: consent, explicit permission from the person whose voice is being cloned, or the rights holder representing them, before it is used to generate new speech, and disclosure, being honest with your audience that a video's speech has been AI-generated or AI-dubbed, rather than letting people assume the person on screen personally speaks the language they are hearing.

• Which language pairs does it actually support, and how well, rather than how many it merely lists
• Is voice cloning opt-in, and does it ask for or require the original speaker's consent
• Does it include a step to review and edit the transcript and translation before the final audio is generated
• Is lip-sync adjustment included, a separate paid step, or not offered at all
• What does it cost per minute or per project, and does that scale to how much video you actually need to localise
• Where is your video and audio processed and stored while the tool works on it

If you are reading this because your company needs to localise video content, such as training or product material, without sending that footage to a third-party cloud dubbing tool, Mickai builds bespoke AI workflows and code enhancements for businesses, running on Mickai's own sovereign operating system with zero data egress: your video, audio and code never leave your own environment. See what that looks like at mickai.co.uk/bespoke.

What you can now do. You can describe what an AI dubbing tool actually does, step by step, rather than treating it as one button. You know which kinds of video it handles well today and which still need a human review pass. You can read a vendor's own adoption or cost figure and say what it does and does not prove. And you have a short, practical checklist for consent, disclosure and choosing a tool before you localise anything of your own.

Keep this workbook. It is yours to keep, print and write in. Come back to chapter 4's table and this chapter's checklist the next time you are choosing a tool or judging a result.

https://trust-agent.ai/exam/ai-dubbing-and-localisation-for-video

This is free educational content published by Mickai LTD, not professional, legal, financial or compliance advice, and Mickai accepts no liability for how it's used.

Trust Agent publishes workbooks like this one for free because Mickai believes people who understand what a tool actually does, rather than what its marketing implies, make better decisions about when to trust it.