Work / AI products
◔
On-device model distillation.
Distilled models small enough to run privately in the browser over WebGPU, giving sub-second inference without sending data to a server.
- Stack
- PyTorch, WebGPU
- Scale
- Sub-second, on-device
- Role
- Research and build
Flow
- Teacher model
- Task dataset
- Distillation
- Quantize
- WebGPU runtime
- Inference in browser
How it works
Some classification and scoring tasks did not need a frontier model, and some users could not send their data to one. A large teacher model labelled a task-specific dataset, and a small student model was trained to match it.
The student was quantized and packaged for a WebGPU runtime, so it loads once and runs entirely in the browser.
The same task interface as the server-side models meant the product could pick on-device or hosted inference per tenant without changing features.
Hard parts
- Browser and GPU support varied widely; graceful fallback to hosted inference was essential.
- Keeping accuracy within an acceptable band of the teacher at a fraction of the size.
Outcome
Sub-second, private inference for the tasks that mattered most to privacy-sensitive customers.