When you speak, you don't pause between words. The audio signal is continuous. The gaps you think you hear are largely reconstructed by your brain, based on what you know about the language. A machine receiving this signal therefore has to do two things at once: segment, and recognise. And neither works without the other.
We reported this morning that transcription had dropped to ten cents an hour. Here is what happens for that price.
From sound to numbers
A microphone measures air pressure thousands of times per second. The result is a long sequence of numbers, with no structure at all: no words, no sentences, no identified silences.
The first step is to turn this sequence into a more useful representation. Rather than keeping the raw amplitude, it is sliced into chunks of a few hundredths of a second, and for each chunk the frequency distribution is measured.
This produces a kind of image: time on the x-axis, frequencies on the y-axis, intensity as colour. It is this image that the system analyses, and it is why techniques developed for vision have transferred so well to audio.
From sounds to words
Then comes the hard part. The system must map sound sequences to text without knowing in advance where each word begins and ends.
Older approaches worked in separate stages: identifying elementary sounds, grouping them into words, correcting with a dictionary. Current systems do it all in one go, learning the mapping between a signal and a string of characters directly, from enormous amounts of transcribed audio.
The key point is that these systems make heavy use of linguistic context. Faced with an ambiguous sound, the model picks what is most plausible given what came before. This is exactly the mechanism of the language models we describe in our reference article, applied to sound.
Because the system favours what is probable, its errors never look like gibberish. An unknown proper noun becomes a common word that sounds the same. A technical term becomes its pronunciation neighbour. A faulty transcription reads perfectly, and that is what makes it dangerous: nothing signals that an error has occurred. This is the same problem we described for hallucinations. Always check names, figures and rare terms.
What remains difficult
Several people at once. Separating overlapping voices is a distinct problem, and it is not solved. This is why meeting transcriptions degrade as soon as people talk over each other.
Accents and variants. A system trained mostly on one standard pronunciation works less well elsewhere. This is a real source of inequality, of the same order as the one we described for under-resourced languages.
Proper nouns. A system cannot guess a name it has never encountered. Most tools let you provide a list of expected terms, and it is the most cost-effective setting that nobody uses.
Noise. An outdoor recording, a distant microphone, background sound: signal quality remains the primary factor in output quality.
How to get better results
Four tips, in order of effectiveness.
Improve the capture. A microphone close to the mouth beats any model. It is the most effective and least technological lever.
Provide a vocabulary. Participant names, jargon, acronyms. Errors on these items account for most of what needs correcting.
Separate the tracks if you can. One recording per person removes the overlap problem.
Proofread what matters. Figures, dates, names, amounts. The rest can stay approximate without consequence.
What to remember
Speech recognition is a good example of a technology that has become so mundane that we forget how hard it is. Segmenting a continuous stream into words, in any language, with accents and noise, is a problem computing took fifty years to handle properly.
It also illustrates a general rule of these tools: they are excellent at what is frequent and weak at what is rare. Your ordinary sentences will be perfect. Your colleague's name, your department's acronym and the exact amount deserve a second look.