CHRIS QUIRK
AI in the Classroom: A Learning Process
Tom Mitchell, Founders University Professor in the Machine Learning Department (MLD), uploaded an instructional PDF document describing how muscles contract to NotebookLM, a new AI tool by Google that analyzes documents and provides a summary of the contents. One output option is a conversational recap of the document read aloud by two synthesized voices, and it sounds just like a podcast.
Mitchell fired up the audio, a four-minute chat between a virtual woman and man describing the muscle contraction process described in the PDF, complete with myofibrils, myosin and sarcomeres, all delivered in an engaging, easy patter.
It is a startling experience to watch a language model generate a podcast on a complex biological process in short order, and as you listen it is easy to feel as though you are absorbing complex information in a relaxed way thanks to the chirpy back-and-forth of the voices and the strategic repetition of key terms. But will it stick? Does this way of learning work better than simply reading a text for understanding and retention?
“We have to test that and find out,” said Mitchell. “But in that podcast when it says there’s no ‘dimmer switch,’ a muscle fiber fully contracts, or it doesn’t contract? The notion of a dimmer switch is not mentioned in the PDF. It’s a much drier description. It’s added by the podcast, and it works like an earworm.”
The podcast example — complete with its apt and catchy analogy spun out of air — shows some of the capabilities and potential power of AI as an educational tool, and, as Mitchell indicates, it also gives a sense of the work yet to be done in this burgeoning field.
“AI is an exciting new opportunity, an exciting new tool, but it really is just the medium,” said Marsha Lovett, co-coordinator of Carnegie Mellon University’s Simon Initiative, director of the Eberly Center for Teaching Excellence & Educational Innovation, and teaching professor in the Department of Psychology. “If you have a podcast that has this nice, lead-in music and lots of sound effects in the background, that may make it more entertaining, but what we know from learning science is that extraneous stimuli can be distracting. I think AI could help or hinder retention depending on how it’s used to design the instruction. But it is really not the technology, rather how it’s used.”
A recent report by the Institute of Education Sciences, a research branch of the U.S. Department of Education, points out several advantages that AI can provide students. Implicit in the report is that AI in education is here to stay. The authors cite benefits AI can provide, like customizing the learning experience for students, and using predictive models to identify a student’s strengths and weaknesses. The report also cautions that AI is best used as an add-on, rather than a substitute for human instruction in education.
Mitchell agrees. Teachers are not going to be replaced by computers, he said. “With the possible exception of where we can make it very inexpensive. Duolingo offers lessons for free, and there’s no teacher involved. And I do learn something.”
Tom Mitchell, Founders University Professor in MLD
Marsha Lovett, Co-coordinator of Carnegie Mellon University’s Simon Initiative and Director of the Eberly Center for Teaching Excellence & Educational Innovation
Robin Schmucker, CMU Postdoctoral Researcher
When thinking about the future of AI in education, Mitchell envisions AI being employed as an aid and resource for teachers to improve student learning outcomes. For the past six years Mitchell has been partnering with CK-12, a California-based nonprofit foundation that aims to increase access to low-cost education in the U.S. and abroad, producing automated media and lessons for K-12 students. In the process, the foundation has amassed a huge amount of anonymized data on how students learn. “They’ve had more than 250 million students visit their website and use their material during their 15-year existence,” Mitchell said. “They’ve logged keystroke-level data of each of those students. They know exactly what each one of those students did as they went through the curriculum.”
These data sets can show researchers how to more effectively keep students engaged and help them master information and concepts. For example, the data might show a particular student spent five minutes and 23 seconds reading a lesson. Then the student watched a video on the topic, which is five minutes long, but dropped out after three minutes and 20 seconds. Then the student was given a multiple-choice practice question, which they got wrong, spending 12 seconds on the question. Recognizing this, the AI provides a hint, and the student gets the question right in just under three seconds.
Mitchell takes this data and the data from thousands of other student learning sessions and processes it through machine learning tools to find deeper insights. “The two problems we looked at are a diagnosis problem and a therapy problem,” Mitchell explained. “If you’re going to teach a student, you want to diagnose what they do and don’t understand right now. We’ve trained our machine learning models to look at the keystroke-level data of a new student who’s partway through the course.” Mitchell’s model can predict with 80% accuracy if the student will correctly answer a particular question. “That’s a substantial ability.”
“That type of information could be useful to teachers,” said Robin Schmucker, a CMU postdoctoral researcher who has been working with Mitchell. “As a teacher, if I can see gaps in a student’s prerequisite knowledge in a subject, or there’s missing information from prior school years that the system could bring to my attention, that would help me target lessons or provide learning opportunities for the student outside the classroom,” Schmucker said.
For the therapy side of the analysis, Mitchell analyzed how various hints can help students succeed on lessons. Teachers provided six hints for each of the thousands of questions Mitchell examined. “We’re able to look at what happened to those students downstream and tell which hint actually works best,” he said. “Our system was so successful in improving student learning outcomes that CK-12 deployed it pretty much across the board in January 2024.”
Schmucker went on to say, “It’s an example of how big data from literally millions of students allows us to do what we’ve never done before.
“I think the more these systems teach, the better they’ll teach, because they will be able to improve by watching what works and what doesn’t,” said Schmucker.
It’s an example of how big data from literally millions of students allows us to do what we’ve never done before.”
— Robin Schmucker, CMU Postdoctoral Researcher
Engagement remains an issue with automated teaching platforms, and drop-off rates can be high. In one grim case, a massive online open course (MOOC) in bioelectricity offered by Duke University, only 3% of registered users took the final examination. Mitchell’s early data show that his customization innovations can increase the percentage of students who stay with a lesson and complete it.
Mathematical word problems will never win popularity contests in classrooms, but tailoring them to the student’s interests can keep that student in the game. Mitchell gives the example of two people running to meet each other from opposite ends of a field, with one going twice as fast as the other. Where do they meet? “If the student is interested in basketball or field hockey or soccer, or has a favorite team, we could automatically personalize the problem for that.” Down the line, Mitchell envisions that a large language model (LLM) could have a chat with the student if they got the question wrong, and coach them to the right answer. “If the student said, ‘halfway,’ the LLM could ask why they thought that and offer a hint such as, ‘If one is moving faster, does it make sense that they meet in the middle?’ The hints could address the problem directly, but also provide a comprehensive understanding of the concepts.”
Mitchell thinks homework is another area where AI will change the status quo for students and help teachers in a variety of ways at the same time. For example, automated grading of online lessons could save time for teachers. Additionally, homework could be generated automatically to meet the particular educational requirements of the student, and the teacher will have much richer information from students’ homework about how they are faring with the lessons. “When I was a student, I got the same hard copy handout homework that you would get,” said Mitchell. “Go home: no hints along the way. No change to the question sequence. I just answered what I could.
“Now with AI, your homework won’t be the same as my homework, I can listen to a podcast on the bus or riding my bike home, and when I get there and do the assignment, I get an online learning experience where I’m coached along the way.”
As researchers develop these educational tools, Lovett believes that educators and designers need to work together to create an iterative process of improvement. Lovett believes the focus should be on aligning the technology with what we already know about how people learn, then studying the AI tools we build to improve them. “We have a lot of guidance from research on the principles of good instructional design. I’m hoping that more people start to weave those principles and strategies for effective learning into how they combine AI and content.”
Mitchell shares this generally optimistic outlook. “There are some downsides, but opportunities far outweigh them. I’m even more convinced now than I was when I jumped into this area, that if AI is ever going to make a difference in education, this is the decade when that has to happen.” ■