Beyond the Obvious Multi-choice Options: Introducing a Toolkit for Distractor Generation Enhanced with NLI Filtering

The process of generating challenging and appropriate distractors for multiple-choice questions is a complex and time-consuming task. Existing methods for an automated generation have limitations in proposing challenging distractors, or they fail to effectively filter out incorrect choices that closely resemble the correct answer, share synonymous meanings, or imply the same information. To overcome these challenges, we propose a comprehensive toolkit that integrates various approaches for generating distractors, including leveraging a general knowledge base and employing a T5 LLM. Additionally, we introduce a novel strategy that utilizes natural language inference to increase the accuracy of the generated distractors by removing confusing options. Our models have zero-shot capabilities and achieve good results on the DGen dataset; moreover, the models were fine-tuned and outperformed state-of-the-art methods on the considered dataset. To further extend the analysis, we introduce human annotations with scores for 100 test questions with 1085 distractors in total. The evaluations indicated that our generated options are of high quality, surpass all previous automated methods, and are on par with the ground truth of human-defined alternatives.