ve6 — Subtitles, Speaker Detection, and Character Animation for Conversation Videos
A tool that, given an input video, automatically runs transcription, speaker identification, and emotion detection, and outputs a video with bottom subtitles and animated character portraits for each speaker.
Duration
In-house development
Team
1 person
01 // OVERVIEW
Project Overview
A tool that, given an input conversation video, transcribes it with Whisper (via Groq) and uses Gemini to determine each speaker and their emotion, automatically generating a video with bottom subtitles and animated character portraits per speaker. Rendering uses HyperFrames, built on HTML + GSAP, and shares the same timeline.json-centered design philosophy as ve5.
02 // KEY FEATURES
Key Features
Whisper Transcription + Gemini Speaker and Emotion Detection
Runs fast transcription via Groq's Whisper API, while Gemini determines the speaker and emotion (8 types including neutral/smile/laugh/surprised) for each utterance. Accuracy can be improved by supplying a hint for the number of speakers.
HyperFrames Rendering of Character Animation and Subtitles
Each frame is rendered with HyperFrames, running on HTML + GSAP. Character portraits and facial expressions switch per speaker, with subtitles displayed at the bottom. A browser-based editing UI lets users select a project and manually adjust the timeline.
Browser Editing UI and Atomic Saves
A REST API served by FastAPI and a Vite frontend allow dragging regions on the waveform, editing speaker/expression/text, and saving with Ctrl+S. Each save retains 20 generations of backups, and timeline.json is safely rewritten via a tmp -> fsync -> rename sequence.
03 // TECHNICAL DEEP-DIVE
Technical Highlights
Solution
Reconciling Non-Destructive Cuts with Cut Previews
Adopted a non-destructive cut design that presents cut candidates via automatic silence detection and toggles them on or off in timeline.json. Both the cut and uncut versions are kept in the output, so a choice can be made later.
04 // IMPACT & RESULTS
Results
Fully automated
Video in, subtitled + animated video out in one pass
8 emotions
Automatically detects each speaker's emotion
20 generations
timeline.json backups retained