Projects / AI products
Self-hosted speech assessment.
Replaced a licensed third-party pronunciation-scoring service with an in-house one: open-source Whisper running on a GPU server I set up, plus our own scoring algorithm tuned so learners’ scores stayed fair and comparable.
- Stack
- Whisper, PyTorch, Python, FastAPI, NVIDIA GPU, Docker
- Scale
- Every speaking exercise in the app
- Role
- Designed, built, and operated
Flow.
- Learner records
- Upload
- Whisper on GPU
- Align to target sentence
- Score words & fluency
- Feedback in app
How it works.
The language platform scored learners’ pronunciation through a licensed vendor: a per-assessment cost, an external dependency, and a black box we could not tune. The replacement had to score the same exercises at least as well, for less, under our control.
A GPU server runs open-source Whisper for transcription with word-level timestamps. The transcript is aligned to the sentence the learner was asked to say, so the service knows which words were said, which were missed or substituted, and how the timing compares to the reference audio.
Our own scoring algorithm combines word accuracy, timing and fluency, and model confidence into a score on the same scale the old service used, calibrated against a set of recordings that had been scored both ways so results stayed consistent for learners and teachers.
Hard parts.
- Calibrating a new scoring model to an existing scale without a spec for how the old one worked.
- Noisy phone audio and a wide range of accents, which meant scoring what was said rather than what the acoustic model preferred.
- Sizing one GPU box for peak classroom hours while keeping latency acceptable inside a mobile app.
Outcome.
Removed the per-assessment licence and the external dependency. Speaking exercises now score in-house at a fraction of the cost, with an algorithm we can keep improving.
Screens.