AI Transcription and Tagging: How Much to Trust the Text

What does each part do?

They are two different tasks:

1. Transcription: what was said. It turns sound into words.
2. Diarization: who said it. It assigns each fragment to a speaker, for example “Participant 1” or “Moderator.”

An error in the first changes the text. An error in the second changes the attribution: a participant’s sentence appears as if the moderator had said it. In research, that second error is serious, because it changes the meaning of the quote.

How much error should you expect?

There is no fixed number. A 2025 study evaluated speech-to-text systems on about ten hours of PennSound, a collection of poetry readings and conversations with very different recording conditions. Error differences between systems were small, and Whisper was the best open-source option only if its hallucinations were avoided, meaning text the model produces that nobody said.

Another study, published in 2026, transcribed 1,108 primary care visits recorded between 2002 and 2006 with WhisperX. Compared with manual transcripts of 92 of them, it measured a 45.2 percent word error rate. The authors believe the figure is overestimated because of de-identification placeholders and non-speech annotations in the manual transcripts, as well as poor audio quality. Read it as a reminder that difficult audio can produce a lot of error, not as a benchmark.

Accents and speech varieties matter too. A 2020 PNAS study tested five commercial systems (Amazon, Apple, Google, IBM, and Microsoft) on US English interviews and measured an average word error rate of 0.35 for Black speakers versus 0.19 for white speakers, with the same gap on identical phrases. A Koji guide, citing a 2024 study, says Whisper was more accurate with American and Canadian accents than with British and Australian ones, and that spontaneous speech was transcribed worse than read speech. That second finding comes from the guide, not from the original source.

The practical conclusion: error rises with noise, overlapping voices, accents, and spontaneous speech, and you cannot assume a transcript is correct without reviewing it. The Koji guide proposes a working rule, not a standard: if the error rate on critical terms (product names, figures, negations) exceeds roughly 5 percent in any participant group, that group’s quotes should be verified by a person before use. To measure it, compare a sample of the transcript with the audio and count only those terms.

How do you improve quality before transcribing?

The recording is what you control most, and it improves the result more than switching models:

• Use a microphone close to each person, or one microphone per participant in groups.
• Avoid rooms with echo, fans, or traffic.
• Ask each person to speak in turn and avoid overlapping.
• Record a 30-second test and transcribe it before the real session.
• Note proper names and technical terms during the session so you can check them later.

How do you tag without losing context?

Once you have a reviewed text, tagging is the step where codes are assigned to fragments. A practical flow:

1. Review the full transcript of at least one session while listening to the audio.
2. Correct names, figures, products, and technical terms.
3. Confirm speaker attribution in the key fragments.
4. Tag with a short codebook, with examples.
5. Store each quote with its session and timestamp, so you can return to the audio if needed.

This step connects with interview synthesis and with research repository tools, which usually store transcripts together with their tags.

When should you NOT use it?

• With poor-quality audio and no time to review it: the text will be unreliable.
• With clinical, legal, or minors’ data, without checking the data processing terms.
• When speaker attribution decides the finding and you have not verified it in the audio.
• When the transcript is used as a verbatim quote in a report without review.

Example

Illustrative example: a team records four interviews in a café with background music. The automatic transcript comes out with many changed words and two participants confused. The team decides to re-record the next sessions in a quiet room with one microphone per person. For the first four, they correct the key sentences by hand, mark the doubtful parts, and only then tag.

What they learn: before blaming the model, check the recording conditions, which are what the team controls.

How do you anonymize a transcript?

Anonymization is part of the review, not an afterthought. Before a transcript leaves the session folder:

• Replace names with participant codes, and keep the key in a separate, access-controlled file.
• Remove identifiers that are not names: company names, neighborhoods, school names, unusual job titles, dates of rare events.
• Mark the removal with a neutral placeholder such as [COMPANY] or [CITY], so the meaning of the sentence stays readable.
• Check the audio, not just the text: a recording can carry a voice or a name that the transcript did not capture.

Note that a placeholder does not guarantee anonymity. A combination of age, job, and city can identify someone in a small community. When the sample is small or the context is specific, ask whether the detail is necessary for the finding at all.

Common mistakes

• Trusting the transcript without listening to any part of it.
• Ignoring speaker attribution, which changes the meaning of a quote.
• Leaving proper names misspelled in the repository.
• Recording in a noisy space and blaming the model.
• Uploading audio with personal data to a tool without checking its terms.

Sources

• Evaluating Speech-to-Text Systems with PennSound (arXiv 2504.05702)
• Racial disparities in automated speech recognition (Koenecke et al., PNAS, 2020)
• WhisperX (official repository)
• Depression Detection at the Point of Care: Automated Analysis of Linguistic Signals from Routine Primary Care Encounters (arXiv 2604.06193)
• Accents, Dialects and AI Transcription: How to Check Your Voice Research Works for Everyone (Koji)