Speaker Diarization of Known Speakers

[tobiasschmidt89]

Hi,

I really enjoy using this Library for speaker diarization to create labeled transcripts in combination with Whisper: Speaker 1: ..., Speaker 2: ..., Speaker 1: ...
Currently I then search and replace the anonymous speaker labels with the real names.

I have some meetings that always have the same speakers (D&D game sessions). Therefore I am searching for a way to kind of create "voice embeddings" of each speaker by recording them in isolation for a minute or so. Then I want to do a speaker diarization using these embeddings to get labels like: Max: ..., Maria: ..., Tobi: ..., Max: ..., Unknown: ...

I would be interested if someone has some pointers on how I could achieve this with Pyannote. I expect I need to do the following:

• Use an audio embedding model to create a normalized embedding for each speaker (with an isolated audio file).
• Do a speaker diarization of the entire audio file to get the audio segments.
• Create an audio embedding of each audio segment.
• Calculate the similarity score of each audio segment embedding with the precalculated embedding of each speaker to identify the most likely speaker.
• Use whisper to transcribe each audio segment attaching the label of the speaker.

Crossed items I know how to do.

I am very comfortable with text embeddings. Audio embedding is a new topic for me.
I would really appreciate any pointers or example scripts.

Thank you
T.

[desicochrane]

Hey @tobiasschmidt89 I'm just started working on this too (for podcasts). Digging into pyannote I see it uses speechbrain "speechbrain/spkrec-ecapa-voxceleb" embedding model. https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb

I can share the approach I have in mind:

1. I am probably going to store k representative embeddings per speaker (I'll likely do k-means clustering of al samples of each speaker).
2. Run pyannote diarization. I might provide a high number of speakers, since I'll later be mapping them anyway, I am ok with low precision at this stage to get higher recall
3. Take these and perform dense retrieval against the embedding db. Probably just cosine distance and take, say, 20 nearest neighbours.
4. Then, I'll do a rerank (cross encoder pattern) to get a probability score.

I'll see how that goes.
Likely I can do more here, I have the text of the audio intervals, so I can also compare the text embeddings along with audio embeddings. Also, in my case, I know some stuff about the prior probabilities ahead of time (the hosts of the podcast are most likely. You might have prior knowledge in your meeting use case too).

Let me know how you get on with it. Feel free to hmu if you wanna collaborate a bit on this.

[PhilipAmadasun]

@desicochrane Could you please explain this "Rerank Cross Encoder Pattern" thing? I'm doing something similar to what you guys are trying, might need that. Also, if you take multiple isolated voice samples(embeddings) of the people you know will be in the conversation, will then have to compare cosine distance on every single one of the voice samples you have on one individual person. I ask because let's say you have 10 voice samples per person, that's 100 cosine distance comparisons no?

[desicochrane]

Hey mate, sorry late response.

"Rerank Cross Encoder Pattern" thing?

Checkout the retrieve and rerank pattern here: https://www.sbert.net/examples/applications/retrieve_rerank/README.html

if you take multiple isolated voice samples(embeddings) of the people you know will be in the conversation, will then have to compare cosine distance on every single one of the voice samples you have on one individual person. I ask because let's say you have 10 voice samples per person, that's 100 cosine distance comparisons no?

You could do that, sure. 100 cosine distances is lightning fast and vectorizable. 1M cosine distances can be done in milliseconds.

[PhilipAmadasun]

@desicochrane How can you calculate many cosine distances in real time as you mentioned? Would you happen to have some example code?

[desicochrane]

you could put the 1M vectors into a matrix and do a dot product (if they're normalised), or you could put them in a vector index like Faiss. Maybe something like this:

!python -m pip install faiss-gpu

import faiss
import numpy as np
import time

# Number of vectors and dimensions
num_vectors = 1000000
dimensions = 1000

# Generate random vectors (1M vectors + 1 query vector)
database_vectors = np.random.rand(num_vectors, dimensions).astype('float32')
query_vector = np.random.rand(1, dimensions).astype('float32')

# Normalize the vectors (FAISS requires normalized vectors for cosine similarity via the inner product)
faiss.normalize_L2(database_vectors)
faiss.normalize_L2(query_vector)

# Create a GPU FAISS index (inner product for cosine similarity, as vectors are normalized)
res = faiss.StandardGpuResources()  # Use all GPU resources
index = faiss.IndexFlatIP(dimensions)  # Inner Product (IP) index
gpu_index = faiss.index_cpu_to_gpu(res, 0, index)  # Move index to GPU 0

# Add vectors to the index
gpu_index.add(database_vectors)

# Perform the search
k = 10  # Number of nearest neighbors to find
start_time = time.time()
D, I = gpu_index.search(query_vector, k)  # D = distances, I = indices of the nearest neighbors
end_time = time.time()

print("FAISS-GPU search time:", end_time - start_time)
print("Distances:", D)
print("Indices:", I)

Running that in google colab I got
FAISS-GPU search time: 0.05401325225830078
Distances: [[0.7987112 0.79602313 0.79593 0.7954348 0.79437023 0.7937976
0.7922267 0.7918892 0.79174185 0.79161 ]]
Indices: [[388597 671101 817522 743371 101155 474046 17489 622588 665845 341719]]

[ymednis]

Hi @desicochrane, @tobiasschmidt89 .

I'm exploring ways to improve speaker diarization in recordings with multiple speakers, but my primary goal is to accurately detect a specific target speaker. I have additional enrollment samples for that speaker (which I suppose I can use to generate an embedding), and I'm wondering if anyone here has achieved high precision for this particular use case.
Thanks, in advance.

[tobiasschmidt89]

No, not yet. I am for my use case (D&D Session Recordings) utilizing LLMs for the speaker assignment. It is like a riddle for the LLM. Here are X speakers, here is the transcript and possible speakers. What is the name of each speaker?

[alunkingusw]

Hi, here is my gist that applies embedding on a wav of a known speaker, and compares it to the output of the diarisation. I am building an API that manages meetings and audio from meetings to transcribe it.
Let me know if this is helpful.
https://gist.github.com/alunkingusw/4275a26ba79cefd49f5a2e91d91c4da2

[alunkingusw]

Once I have this working, my API (along with a frontend) will accept audio files and transcribe them, automatically labelling known speakers. The sample audio used for the embeddings could be provided by the user, or manually selected from a recording. Currently I'm looking at how I can take the segments from a meeting and order them by the most accurate detections of speech (https://pyannote.github.io/pyannote-metrics/reference.html), so that the comparisons of embeddings can be more accurate.

[mroconnell]

In case it's helpful, there's a dev who did this process for public meetings in NYC and developed a great prompt engineering methodology to have LLMs identify speakers and then map the speaker name across very lengthy sessions. If you're running something like a 5+ hour D&D session, you might have a lot of 'lost in the middle' problems where the LLM does a much better job IDing speakers at the beginning and end of the file. He has a lot of tricks to fix that. He does mention toward the end that you can include a list of the possible speaker names in the prompt as a way to eliminate small variations and spelling mistakes that occur if you just set the llm loose without that parameter.

Link to his speaker id prompts: https://gist.github.com/voberoi/3d82f6b2a55e79b7cd014847853be8bf
Video presentation on the topic: https://www.youtube.com/watch?v=FjDQu2IbC1M

[mroconnell]

Now that I think about it, if your D&D friends are like mine, they might be voice acting and changing their voices very deliberately as they narrate actions. That would pose an interesting challenge for this specific task! Wonder how that will go.

[voberoi]

That dev is me! So fun to see this here, because I'm investigating creating voiceprints for everyone in my database.

(The site is citymeetings.nyc)

Are Pyannote's service's voiceprints just embeddings? Can I do nearest neighbor searches to find most-alike voices?

[voberoi]

Btw annotated slides for the talk are here if you're interested:

https://vikramoberoi.com/posts/how-citymeetings-nyc-uses-ai-to-make-it-easy-to-navigate-city-council-meetings/

There's a lot of stale info there now. I gave another talk at School of Data in 2025 but it's not up yet for some reason.

[mroconnell]

Are Pyannote's service's voiceprints just embeddings? Can I do nearest neighbor searches to find most-alike voices?
tldr Yes, and I think yes (and it will return scores for each embedding you want it to compare to)

I didn't dive very deeply into this and I'm a real rookie. The test I ran let me upload specific speaker videos so pyannote could create some embeds (I think that was the right language) and then when I ran a whole meeting past it with diarization, it would look at each diarization segment and give a match score for each embed I had (I think this is the NN part of your question) and it would spit out the one with the highest score. I did a mediocre job with the embeds just to see how it all worked and it got 3 out of 5 council members right on my first run through. I was in a hurry to ship something for a demo so I switched it to manual speaker mapping where user just clicks on the "Unknown Speaker 1" label and edits it in the doc. https://www.loom.com/share/dfa4b5c783304d44b53c2cd6cf3109fa

What I intend to do is let users indicate start and end times for uninterrupted speech and upload it to pyannote so the users can create their own embeds for local electeds. But that is a long way away on the road map! :) and naturally you're in a different place where you have a very large but finite list of elected officials you could do a great job for so you might have a smarter path for nyc.

[voberoi]

Interesting! Thank you for the context.

[Nazkab10]

@desicochrane Did you end up figuring this out for podcasts. Looking into the exact same thing rn.

[Nazkab10]

@mroconnell Thanks, taking a look at this

[alexkondor03]

Hi @tobiasschmidt89 Are you running your DND sessions on a meeting platform? If you are, it might be easier to get speaker labels by using an API that supports transcription and speaker diarization.

[tobiasschmidt89]

Yes, in the meantime we moved to Google Meet and I use Google Meet transcript, a custom Canvas app created with Gemini Canvas to clean the transcript using the free Gemini API integration and Notebook LM to then actually create session summaries based on transcript and context data.

Then I built a Gemini canvas app to transform the summary in a advanced storybook with images using the Gemini API again (it is better than the default storybook gem). Really nice summaries.

Automating end to end feels unnecessary now as stitching such things together takes me maybe 5 minutes, so no problem. ;)

[rmatinha-fctunl]

Hi, I am an absolute newbie to these matters, and I'd like to know how you solved the problem of assigning the transcriptions from whisper to the diarization results?

From my testing the diarization model identifies things ok enough, but the timestamps don't align at all with the ones provided by the transcription model (Systran/faster-whisper-large-v3). I was wondering if there was a way of passing the timestamps of the start and end of a transcribed phrase and having the model identify the speaker in that period.

[alunkingusw]

You can do it one of two ways - either line them up, or diarise first and send the audio in chunks to whisper for transcribing bit by bit (not so great for lots of overlap)
here is an example of doing it segment by segment
https://gist.github.com/alunkingusw/75ac37c44ca5b00ed9fdd5f50b391afa

and here is an example of using the whisper-pyannote library to line up the timestamps and transcribe it as one whole
https://gist.github.com/alunkingusw/2eb29682a98f94a714d10080ed0f4896

I found that doing it as one whole increased the likelihood of hallucination from whisper, but then I was transcribing podcasts where there isn't a lot of speaker overlap. Have a go of both and see how you get on! There are alternative ways of cleaning up the hallucinations if you get them.

[rmatinha-fctunl]

I will take a look, but from my past experiences, the diarization model tends to halucinate more than the transcription, usualy when there are interruptions between speakers, the model swaps the people talking around. But then again I just noticed that this thread is for KNOWN speakers, and in my case it will have one unknown speaker.

[alunkingusw]

I'm being pedantic here, but the diarisation (sic, I'm UK based) model doesn't 'hallucinate'. What you are describing there is the DER - diarisation error rate. It's one of the biggest problems for diarisation, a good way to improve that is to train the diarisation model on annotated data that has more overlapped speaking, but there are lots of things along the pipeline that could help. Known or unknown shouldn't be different, as this thread is discussing identifying the speaker after diarisation has occurred anyway.
The Whisper model 'hallucinates' because it uses language model reasoning if the audio is not generating a strong enough signal.
Good luck with your project!

[rmatinha-fctunl]

Ok, update on this issue, I did uv pip install omegaconf, added the import omegaconf to the import block and omegaconf to the depencies at the start which seemed to fix that issue.

Then I came across another error involving the brouhaha module likewise not being properly installed, which I think I solved by running git clone https://github.com/marianne-m/brouhaha-vad.git , changing numpy==1.24.4 to numpy>=1.24.4 in the requirements.txt in the brouhaha path, and then adding the dependency with "brouhaha @ file:///path/to/file/.venv/brouhaha-vad".

However now I seem to have stumbled upon a compatibility issue with:

Traceback (most recent call last):
  File "/path/to/file/scripts/diaritranscribe3.py", line 69, in <module>
    from pyannote.audio import Inference, Model, Pipeline
  File "/.cache/uv/environments-v2/diaritranscribe3/lib/python3.12/site-packages/pyannote/audio/__init__.py", line 29, in <module>
    from .core.inference import Inference
  File "/.cache/uv/environments-v2/diaritranscribe3/lib/python3.12/site-packages/pyannote/audio/core/inference.py", line 35, in <module>
    from pyannote.audio.core.io import AudioFile
  File "/.cache/uv/environments-v2/diaritranscribe3/lib/python3.12/site-packages/pyannote/audio/core/io.py", line 60, in <module>
    ) -> torchaudio.AudioMetaData:
         ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'torchaudio' has no attribute 'AudioMetaData'

Will update if I find a fix, but if someone knows how to do it, feel free to share

[rmatinha-fctunl]

You can do it one of two ways - either line them up, or diarise first and send the audio in chunks to whisper for transcribing bit by bit (not so great for lots of overlap) here is an example of doing it segment by segment https://gist.github.com/alunkingusw/75ac37c44ca5b00ed9fdd5f50b391afa

and here is an example of using the whisper-pyannote library to line up the timestamps and transcribe it as one whole https://gist.github.com/alunkingusw/2eb29682a98f94a714d10080ed0f4896

I found that doing it as one whole increased the likelihood of hallucination from whisper, but then I was transcribing podcasts where there isn't a lot of speaker overlap. Have a go of both and see how you get on! There are alternative ways of cleaning up the hallucinations if you get them.

Ok, I attempted to implement this on my machine and I got a bunch of compatibility issues, specifically between Brouhaha needing pyannote==3.3.0 and the way the diarization token is read.

Traceback (most recent call last):
  File "/home/user/diarization/repos/scripts/diaritranscribe3.py", line 621, in <module>
    main()
  File "/home/user/diarization/repos/scripts/diaritranscribe3.py", line 589, in main
    diarization = diarize_audio(
                  ^^^^^^^^^^^^^^
  File "/home/user/diarization/repos/scripts/diaritranscribe3.py", line 208, in diarize_audio
    pipeline = Pipeline.from_pretrained(MODEL_ID, use_auth_token=tokens["diarization"])
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/uv/environments-v2/diaritranscribe3-3f9949c47f20e532/lib/python3.12/site-packages/pyannote/audio/core/pipeline.py", line 89, in from_pretrained
    config_yml = hf_hub_download(
                 ^^^^^^^^^^^^^^^^
  File "/home/user/.cache/uv/environments-v2/diaritranscribe3-3f9949c47f20e532/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 88, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
TypeError: hf_hub_download() got an unexpected keyword argument 'use_auth_token'

changing pipeline = Pipeline.from_pretrained(MODEL_ID, use_auth_token=tokens["diarization"]) to pipeline = Pipeline.from_pretrained(MODEL_ID, token=tokens["diarization"]) does not fix it

Can you list what pyanotte, torch, versions are you using?

[Karen86Tonoyan]

Thanks, that’s a fair correction. You’re right — “hallucination” is not the right term for diarisation errors. What I meant was diarisation error rate: incorrect speaker attribution, segmentation mistakes, and degradation under overlapped speech. I’ll update the wording to avoid mixing DER with Whisper-style hallucination. The distinction is important: diarisation fails as a speaker segmentation/attribution problem, while Whisper may hallucinate transcript content when the acoustic signal is weak and the language model prior dominates. Good point also on overlapped speech. I’ll frame the improvement path as training/evaluating diarisation on annotated data with more overlapping speakers, rather than treating known/unknown speakers as the core issue after diarisation. Thanks for the precise correction. pon., 27 kwi 2026, 11:15 użytkownik Alun King ***@***.***> napisał:

…

I'm being pedantic here, but the diarisation (sic, I'm UK based) model doesn't 'hallucinate'. What you are describing there is the DER - diarisation error rate. It's one of the biggest problems for diarisation, the best way to improve that is to train the diarisation model on annotated data that has more overlapped speaking. Known or unknown shouldn't be different, as the thread is discussing identifying the speaker after diarisation has occurred anyway. The Whisper model 'hallucinates' because it uses language model reasoning if the audio is not generating a strong enough signal. Good luck with your project! — Reply to this email directly, view it on GitHub <#1667 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BUHB4E4O7KWVH2JVXUE74534X4QMVAVCNFSM6AAAAACX2YK6W2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTMNZSGY3DGMI> . You are receiving this because you are subscribed to this thread.Message ID: <pyannote/pyannote-audio/repo-discussions/1667/comments/16726631@ github.com>

[Karen86Tonoyan]

ModuleNotFoundError: No module named 'omegaconf'

…

On Mon, Apr 27, 2026 at 6:16 PM rmatinha-fctunl ***@***.***> wrote: I do apologise if I seemed rude. English is not my native tongue. I've tried giving it a couple tests, and changing it to work from CLI instead of colab/jupyter and I seem to be getting an error when it calls the embeding model File "/diarization/repos/scripts/diaritranscribe3.py", line 356, in apply_reference_speaker_labels embedding_model = Model.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^ File "/.cache/uv/environments-v2/diaritranscribe3-3f9949c47f20e532/lib/python3.12/site-packages/pyannote/audio/core/model.py", line 602, in from_pretrained loaded_checkpoint = pl_load( ^^^^^^^^ File "/.cache/uv/environments-v2/diaritranscribe3-3f9949c47f20e532/lib/python3.12/site-packages/lightning/fabric/utilities/cloud_io.py", line 73, in _load return torch.load( ^^^^^^^^^^^ File "/.cache/uv/environments-v2/diaritranscribe3-3f9949c47f20e532/lib/python3.12/site-packages/torch/serialization.py", line 1579, in load return _load( ^^^^^^ File "/.cache/uv/environments-v2/diaritranscribe3-3f9949c47f20e532/lib/python3.12/site-packages/torch/serialization.py", line 2190, in _load result = unpickler.load() ^^^^^^^^^^^^^^^^ File "/.cache/uv/environments-v2/diaritranscribe3-3f9949c47f20e532/lib/python3.12/site-packages/torch/serialization.py", line 2179, in find_class return super().find_class(mod_name, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'omegaconf' Do you happen to know if there is something I could do to fix this? I did not use the forked whisper version which is the largest deviation I think I made. — Reply to this email directly, view it on GitHub <#1667 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BUHB4E42SYCEXFONZEHLARL4X6BXHAVCNFSM6AAAAACX2YK6W2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTMNZTGE2DGNQ> . You are receiving this because you commented.Message ID: ***@***.*** com>

[rmatinha-fctunl]

I installed omegaconf with
uv pip install omegaconf
and adding an import omegaconf to the import block but it is still giving me errors

[rmatinha-fctunl]

Ok, adding "omegaconf" to the dependencies fixed it

[arcticloud]

Your plan is exactly right — enroll one embedding per known person, then match each diarized speaker to the closest enrollment. Concrete version:

from pyannote.audio import Pipeline, Inference, Model
from scipy.spatial.distance import cosine

emb = Inference(Model.from_pretrained("pyannote/embedding", use_auth_token=TOKEN),
                window="whole")          # window="whole" -> one vector per input
dia = Pipeline.from_pretrained("pyannote/speaker-diarization-3.1", use_auth_token=TOKEN)

# 1. enroll: one averaged embedding per known person (isolated ~1 min clips)
references = {name: emb(path) for name, path in {
    "Max": "max.wav", "Maria": "maria.wav", "Tobi": "tobi.wav"}.items()}

# 2. diarize the full session
diar = dia("session.wav")

# 3. assign each diarized speaker to the nearest known person (or "Unknown")
def match(vec, thr=0.5):
    sims = {n: 1 - cosine(vec, r) for n, r in references.items()}
    name, score = max(sims.items(), key=lambda kv: kv[1])
    return name if score >= thr else "Unknown"

names = {}
for spk in diar.labels():
    seg = max(diar.label_timeline(spk), key=lambda s: s.duration)  # longest clean segment
    names[spk] = match(emb.crop("session.wav", seg))

labeled = diar.rename_labels(names)

Two things that matter in practice: use window="whole" so each enrollment clip becomes a single vector, and tune the thr threshold on your own recordings (D&D sessions with crosstalk will need a bit of experimentation — lower = stricter, more "Unknown"). For robustness you can average several segments per diarized speaker instead of just the longest one. The embedding model is the same ECAPA/wespeaker net pyannote uses internally, so enrollment and diarization live in the same vector space.