Software that uses artificial intelligence (AI) to turn a recorded lecture into a quiz can look like one feature, but it rests on a chain of separate systems: speech recognition, passage retrieval and question generation. Published research on each stage shows where the chain holds up and where errors can enter.
From speech to transcript
The first stage converts audio into text. A well-documented example is Whisper, the OpenAI speech recognition system described in a 2022 paper. Its authors trained it to predict transcripts of audio found on the internet, scaling to 680,000 hours of multilingual and multitask supervision, and reported accuracy and robustness approaching human levels. They also released the models and inference code for further work.
High average accuracy does not rule out serious individual errors. A 2024 study titled Careless Whisper evaluated the service as of 2023 and found that roughly 1% of audio transcriptions contained entire phrases or sentences that did not exist in the audio. The authors found that 38% of those hallucinations included explicit harms, such as inaccurate associations or implied false authority.
The errors occurred disproportionately for speakers with longer non-vocal stretches, a common symptom of aphasia. In a study product, an invented sentence is more than a transcript defect, because later stages may treat it as something the lecturer said.
Finding the right passage
The next design question is what the question generator gets to read. One established approach is retrieval-augmented generation, or RAG, set out in a 2020 paper accepted at the NeurIPS conference. A retriever pulls relevant passages from an index, and a language model uses them to write its output.
The authors, whose index was built from Wikipedia, reported that RAG models produced “more specific, diverse and factual language” than a baseline without retrieval. They also described provenance, or showing where an answer came from, as an open problem for language models working alone. In study software, a lecture transcript can take the place of that index, which makes it possible to tie a question to a specific passage.
Writing the questions
Question generation has been tested more directly. In a Carnegie Mellon University study prepared for the 2024 Australian Computing Education Conference, researchers used GPT-4 to write 651 multiple-choice questions for 246 learning objectives across six Python courses, then compared them with 449 human-written questions.
The team found that 81.7% of generated questions passed all of its evaluation criteria. The failures were specific: 4.9% had more than one correct answer, against 1.1% of human-written questions, and 4.0% had wrong options that gave away the answer, against 0.9%.
The system worked from course descriptions and learning objectives rather than lecture transcripts. The authors also said they did not evaluate the questions in live classrooms, so their teaching impact is not known.
Commercial products package these stages in their own ways. Serona, for example, says students can upload an old test and receive new practice questions written in the teacher’s format.
What retrieval research supports
The case for quiz features comes from research on retrieval practice, the act of recalling information rather than rereading it. A 2019 review in Frontiers in Education covered 23 studies run in real classrooms and reported reliable testing effects across all experimental conditions in 19 of them.
Format matters for product design. The reviewers found multiple-choice and fill-in-the-gap tests highly beneficial, while short-answer tests were “not so consistently beneficial.” On corrective feedback, they said the literature did not allow them to conclude that it adds a significant benefit, partly because too few studies compared conditions with and without it.
The review also limits the claim. It found retrieval more beneficial than “weak” or “superficial” learning strategies, but said it “does not seem to be more beneficial than ‘stronger’ learning strategies.”
Why it matters
For software teams, the research points to concrete build decisions: keep a link from each question to its source passage, screen generated items for multiple correct answers and wrong options that give the answer away, and treat transcription errors as a risk to the whole pipeline rather than a cosmetic flaw.
For schools and students comparing products, the key distinction is between evidence for a method and evidence for a tool. The classroom studies measure testing as a learning strategy. They do not show that any particular AI product delivers the same benefit.
The caveats
AI study tools can reduce the effort of making quiz material, but they can also carry a transcription or generation error into what a student memorizes. Checking generated questions against the lecture, slides or textbook remains the safer approach. The studies cited here also cover specific models, courses and time periods that may not match products on the market today.


