Auto-captions out of sync: why it happens and how to fix it
The symptom
The clip opens well. The first captions track the voice. Halfway through, something slips: the word appears on screen before it is spoken, and by the end it is a whole sentence ahead.
This is the most common complaint about automatic captioning, and it is almost always misdiagnosed. It is not a speech recognition accuracy problem — the text is usually correct. It is a timing problem.
The cause: estimating instead of measuring
Most transcription tools return text grouped into sentences, with a start and end time per sentence. They do not return a time per word.
To display captions word by word, those timings have to be invented. The usual method divides the sentence duration by the word count, or proportionally by character count.
The problem is that nobody speaks that way. A real sentence has long and short words, pauses before the ones that matter, hesitations in the middle. Proportional division is wrong on almost every word, and the errors accumulate: the fifth word inherits the error of the previous four.
That is precisely why drift grows across the clip instead of staying constant. A fixed delay could be fixed with a global offset. A cumulative error cannot.
The fix: forced alignment
The solution is to measure rather than estimate.
Forced alignment starts from an advantage: the text is already known. You do not need to guess what was said, only where. The audio is searched for the position of each word of the known text, and the result is measured timings, not calculated ones.
We measured the difference across a batch of real clips:
| Median drift | |
|---|---|
| Timings estimated from the sentence | 1.92 s |
| Word-level forced alignment | 0.04 s |
A median drift of 1.92 seconds means that in half the captions the viewer is reading something they have not heard yet. Four hundredths of a second is indistinguishable from perfect.
The second failure: filtering too hard
There is a related defect that shows up often: whole sentences with no caption at all.
The usual cause is voice activity detection. These filters exist to save processing by skipping stretches without speech. When set too aggressively, they classify quiet, hesitant or noise-overlapped speech as silence — and that speech disappears from the transcript.
In our measurement, enabling voice detection on the source transcription cost about ten per cent of the text: 13,703 words against 15,242 without it, on the same recording.
The practical conclusion is that the two steps want opposite things. Source transcription wants maximum recall, so the filter should be off. Per-clip alignment wants timing precision, and there a moderate filter helps. Using one setting for both means choosing between losing text and losing precision.
How to check your own clips
You do not have to take anyone's word for it. This is measurable:
1. Take the finished clip, with captions burned in. 2. Transcribe it again with a tool that returns per-word timings. 3. Compare when each word is spoken against when its caption appears. 4. Take the median of the differences.
Below 100 ms is fine. Above 300 ms is noticeable. Above 500 ms is the defect described at the top.
It is also worth measuring uncovered speech: how many seconds of voice have no text under them. A natural pause without a caption is normal; a spoken sentence without one is the filter eating content.
Why this matters more in vertical
In a horizontal video watched with sound, a caption slightly ahead is a detail.
In vertical, in the feed, with no sound in the first seconds, the caption is the content. It is what the viewer uses to decide whether to stay. A caption out of sync in the first three seconds costs the whole clip.
At Lanoar AI Clips we measure the drift on every clip before delivery, and a clip that fails the threshold does not ship. Not because alignment is a selling point, but because it is the difference between a clip that gets watched and a clip people swipe away from.
For where this sits in the full process, we wrote this guide.