Jonathan Bechtel on AI tutoring (from my email)

You recently mentioned the Alpha School and their claims about AI tutoring. I share the skepticism expressed in your comments section regarding selection bias and the lack of validated academic benchmarks.

I wanted to highlight a more rigorously evaluated project called Tutor CoPilot, conducted jointly by Stanford’s NSSA and the online tutoring firm FEVTutor (sadly they’ve since gone bankrupt). To my knowledge, it’s the first and only RCT examining AI-assisted tutoring in real K-12 school districts.

Here’s the study: https://nssa.stanford.edu/studies/tutor-copilot-human-ai-approach-scaling-real-time-expertise

Key findings:

• Immediate session-level learning outcomes improved by 4-9%.
• Remarkably, the tool impacted tutors even more than students. After six weeks, inexperienced tutors reached performance parity with seasoned tutors, and previously low-performing tutors achieved average-level results.

Having contributed directly to the implementation, I observed tutors adapting their interactions based on insights from the AI.  This study did not measure its impact on more distal measures of learning like standardized tests and benchmark assessments, but this type of research is in the works at various organizations.

Given your recent writings on AI and education, I thought you’d find this compelling.