Inference time of the distractor generation module

[sunhaozhepy]

Hi Kristian,

Thanks for the code, it is really helpful.
I'm wondering about one thing: how do you ensure that your distractor module will output a sequence with 2 tokens in it? Because it appears to me that training with three distractors doesn't necessarily make you 100% certain that the model will generate outputs of this form. Or do you do multiple samples and just concatenate them?
Thanks in advance.

[kevinmalana]

I ran into similar performance issues with distractor generation. In quizworld.xyz, I sidestepped the problem entirely by using Groq's llama-3.1-8b-instant for question generation with few-shot prompting instead of a fine-tuned model — inference is fast and the quality is good. You might find the approach useful: https://github.com/kevinmalana/quizworld