EDUCATION
Smart Grading
Role: Product Designer
Company: Quizlet
Year: 2020
Background
Written questions were graded too strictly with high override rates adding friction to users while studying. There were three types of grading: unsure grading: users can choose whether an answer is correct vs incorrect when the grader is unsure; require one answer only: necessary to input one part of the answer that has multiple answers (separated by slashes, commas, or semicolons); and override: users can override any written question.
Data Collected
Users overrode 42% of incorrect short answers.
Many typos were not captured by the previous grading method
85% threshold numbers should not be treated like letters - no override if a number is present.
SOLUTION
Flexible Grading Options
The long-term vision for written answer grading on Quizlet is to have smart, and appropriate defaults detected based on the user's needs. Break down the components that make up our grading by allowing users to customize (ex: turn on/off typo grading)
Implementing an intelligent answer grader for long/short text written questions will mimic a teacher correcting a student when simple grammar or spelling mistakes are made (which offer partial credit) thus allowing users to be less frustrated and study more often on Quizlet.
Study Experience
A/B tested flexible grading across short/long text written questions. Added a small label to indicate that the question has been smart graded and users have the option to opt-out.
Overall override rate dropped by 8% for Mobile, 40% for Web, with stronger impact seen when isolating only questions eligible for grading service.
Results + Learnings
53% Overrides drop - overall override rates dropped significantly.
+3.5% Increase - number terms mastered.
+4% Increase - user reach end screen (round end).
Increase study session - over the course of the experiment, users initiated an additional study sessions.
Low opt-out rate - Indicates users find grading acceptable.
PERSONALIZATION - Users have varying grading needs based on the subject studied, test format, and learning preferences.
FLEXIBILITY - Providing options to deeply understand which grading options work best for various study needs.
GRADING SETTINGS - To learn more about grading settings, it’s best to keep the options consistent across all study tools.