Making an AI Short Drama Solo, Part 3: Dialogue and Lip Sync, Voice First and Picture Second

Behind the scenesOct 18, 2026 · 6 min read

Why the voice comes before the picture

When I started, I did it the "obvious" way: generate the shots, then add voices on top. Every episode turned into a fight. A line ran longer than its shot, so the picture froze or looped. A shot ran longer than its line, so a character stood there staring into space. I spent more time stretching and trimming than making anything.

So I flipped the order. Once the script is locked, every line is voiced first. Then I measure the real audio, and that length becomes the length of the shot. The picture is made to fit the sound, never the other way around.

This also makes my "under one minute" rule honest. A script that looks short on paper can easily run long once it is spoken. The only number that counts is the length of the actual audio, so that is what I check before a single image gets generated.

One character, one voice, forever

Every character has a single fixed voice for the whole series. Viewers recognize a character by voice faster than by face, and if the voice drifts, the character feels like a different person.

Each line in the script also carries an emotion label: annoyed, smug, panicked, deadpan. That label nudges the speed, pitch and volume a little. But here is the lesson I learned the hard way: each character has a ceiling on how far those nudges can go. Without it, one episode pushed a character's voice up for an "excited" line, another pushed it down for a "tired" one, and side by side they sounded like two different actors.

The AI-assistant character gets the tightest ceiling of all. A machine voice has to be steady. If the AI suddenly sounds emotional, the joke about the calm, overconfident machine falls apart.

Timing is the joke

A one-minute sketch has no room for flat delivery, so I treat rhythm as something I direct on purpose:

• Setup lines run a little fast. The audience needs the information, not the scenery. Brisk setup keeps them leaning in.
• The punchline slows down and drops lower. It is the one line that gets space to breathe.
• Pauses go to the extremes. A quick comeback almost steps on the previous line. The pause right before the punchline is long, long enough that you feel it coming. Medium pauses everywhere are what make dialogue sound robotic.

One small, boring fix made a surprising difference. I used to leave a generous bit of silence after every line, just to be safe. Across a whole episode those small gaps added up to a lot of dead air, and the pacing felt sleepy without any single shot looking wrong. Now the silence after each line is tight, and the only deliberate pause is on the landing beat, where the audience needs a moment to laugh.

The unglamorous audio details

What the voice says vs. what the subtitle shows

A show about programmers is full of tech words, and a voice-synthesis model mispronounces a lot of them. My fix is to keep two versions of a line: a phonetic spelling that goes to the voice, so it sounds right, and the correct spelling that goes into the subtitle, so it reads right. Viewers never see the weird spelling.

Every line at the same loudness

Some voices are simply much quieter than others. Put a soft voice next to a loud one and the viewer keeps reaching for the volume button. So every line is loudness-normalized on its own. When I actually want a whisper or a shout, I set it as a relative adjustment on top, so it stays a deliberate choice instead of an accident of which voice I picked.

Some lines can't be spoken

A character staring at a broken build might "say" nothing but "???". A voice model can't voice that, and it usually fails outright. Lines like that go on screen as text instead.

Never mix two voices in one episode

If the voice service fails halfway through an episode, the pipeline stops. It does not quietly fall back to a different voice engine. One character sounding slightly different in the second half of an episode is worse than waiting and rerunning. Every finished line is also cached, so unchanged lines are never regenerated, and a rerun only touches the lines I actually edited.

Trim the accidental pauses

Voice models sometimes take a strangely long breath in the middle of a sentence, especially around punctuation. Those pauses are trimmed automatically, so the only long pauses left are the ones I wrote.

Lip sync: the voice drives the mouth

Once the audio is done, it does a second job. When a shot with dialogue is generated, the voice track for that shot is handed to an audio-driven video model as the target audio, starting from the very first frame. The model animates the face to match that sound, so the mouth follows the real line instead of a guess.

A few rules came out of watching a lot of bad takes:

1. Shots where nobody speaks get silence as their target. If I leave the audio empty, the model happily invents something on its own: humming, murmuring, a mouth that moves for no reason.
2. In a two-person shot, each person gets their own track. Each track is the shot's audio with the other person's lines muted, and each side of the frame is driven separately. Otherwise both characters mouth every line together, like a duet.
3. Listeners are directed, too. The person not talking is told to keep their lips closed and their eyes on the speaker. A good listening face sells a joke as much as the line does.
4. Phone calls stay on the listener. The caller is off screen, so their voice gets a phone filter and drives no mouth at all. The camera stays on the person listening, because their reaction is the funny part.

And right after the punchline, I usually add a short silent reaction shot. No line, just a face processing what was just said. It gives the audience a beat to laugh, and it often gets a bigger laugh than the line itself.

Next up

Part 4 is about the hardest visual problem in AI video: keeping characters consistent. How the same person stays the same person from shot to shot, and how the show grew from 2D stills into 3D widescreen.