Shabemaru - AI Conversational Speaking Assessment App
Students have voice conversations with an AI and receive automatic scoring and feedback across five axes: grammar, vocabulary, fluency, pronunciation, and content. Teachers can create assignments and review results.
Duration
Mid-term development
Team
1-2 people
01 // OVERVIEW
Project Overview
An English speaking assessment web app with two roles: teacher and student. Teachers create assignments (specifying the AI's role, CEFR level, number of exchanges, scoring criteria, etc.), and students have a voice conversation with the AI, receiving automatic scoring and AI feedback across five axes: grammar, vocabulary, fluency, pronunciation, and content. Built on a pipeline of Whisper for transcription, GPT-4 for evaluation, and TTS for the AI's voice responses.
02 // KEY FEATURES
Key Features
Real-time voice conversation with AI
Records audio via the browser's MediaRecorder API, transcribes it with OpenAI Whisper, and has GPT-4 generate a response, which is returned as speech via TTS. This loop continues for the configured number of exchanges.
Automatic scoring and AI feedback across five axes
After the conversation ends, scores are calculated across grammar, vocabulary, fluency, pronunciation, and content, and AI feedback plus a transcript of the conversation are presented to the student. Teachers can review individual conversation logs and scores from a list of submissions.
Customizable assignment settings
Teachers can configure the AI's role (e.g., ALT Teacher/Friend), CEFR level (A1-C2), speaking speed, accent, opening question, and scoring criteria for each assignment. Uses a multi-tenant setup that manages users by school ID.
03 // TECHNICAL DEEP-DIVE
Technical Highlights
Solution
Stabilizing the link between mic recording and the OpenAI API
Addressed format compatibility issues between the browser's MediaRecorder API and the Whisper API, as well as handling recording interruptions. Designed the audio-to-text-to-AI-response-to-audio cycle to keep the resulting latency within an acceptable range.
04 // IMPACT & RESULTS
Results
5 axes
Automatic evaluation of grammar, vocabulary, fluency, pronunciation, and content
2 roles
Separate functionality for teachers and students
CEFR levels
Difficulty set across 6 levels from A1 to C2