What diarization produces
Without it, a transcript of a two-person interview is one block of text. With it, the same transcript reads as:
How it works
- Voice activity detection. Find the parts of the audio where someone is speaking.
- Segmentation. Cut speech into short chunks at likely speaker changes.
- Speaker embeddings. Turn each chunk into a numeric fingerprint of the voice.
- Clustering. Group similar fingerprints; each cluster becomes a speaker label.
- Alignment. Merge the labelled time ranges with the words from speech recognition.
Open-source pipelines such as pyannote implement these steps and are what most local tools, Scrieb included, build on.
Why Whisper doesn't include it
Whisper was trained to map audio to text. It has no representation of who is speaking, so it cannot label turns. Products that advertise "Whisper with speaker labels" are running a second, diarization model and merging the results. That matters for privacy: a tool can run Whisper locally and still send the audio to a server for the diarization step. Ask about both.
What makes it fail
Overlapping speech
Two people at once is the hardest case; most models assign the segment to one of them.
Similar voices
Two speakers of the same pitch and accent may be merged into one label.
Poor audio
A laptop microphone across the room, phone-quality audio or heavy noise blur the voice fingerprints.
Many speakers
Panels of ten sound-alike voices are split unreliably; two to six speakers taking turns is the sweet spot.
Practical advice for recordings, and how it runs offline in Scrieb: speaker identification.
Frequently asked questions
- What does diarization mean?
- Diarization is the task of answering "who spoke when" in a recording: dividing the audio into segments and assigning each segment to a speaker. The output is a set of labels such as Speaker 1 and Speaker 2 attached to time ranges. It comes from "diary", as in keeping a log of turns.
- Is speaker diarization the same as speaker identification?
- Almost. Diarization separates and labels speakers without knowing who they are: Speaker 1, Speaker 2. Speaker identification, strictly, matches a voice to a known person. In transcription apps the terms are used interchangeably for the first thing, and that is how Scrieb uses it: labels by voice.
- Does Whisper do speaker diarization?
- No. OpenAI's Whisper converts speech to text and has no notion of speakers. Diarization requires a separate model, usually a segmentation and speaker-embedding pipeline such as pyannote, run alongside Whisper and merged with its output.
- How accurate is speaker diarization?
- It is measured by diarization error rate, the share of audio attributed to the wrong speaker, missed or falsely detected. On clean recordings with two to six speakers and little overlap, modern models achieve single-digit error rates. Overlapping speech, similar voices and poor audio raise it quickly.
- Can diarization run offline?
- Yes. The models are small enough to run on a laptop CPU or GPU. Scrieb runs diarization locally alongside transcription; WhisperX does the same from the command line. Most cloud services run it on their servers.
- How many speakers can diarization handle?
- There is no hard limit, but accuracy falls as speakers multiply and voices resemble each other. Two to six speakers who take turns is the range where it works well.