WebGPU composition + browser AI

WebGPU video with browser AI

A browser-native pipeline that decodes video with MediaBunny, composites layers on the GPU with WebGPU, and runs AI on WebGPU inside dedicated workers. Choose RF-DETR for real-time object detection, or SmolVLM to describe the current frame. Decoded VideoFrames are transferred to workers with no canvas readback.

Decode media Video and audio tracks are decoded in the browser with MediaBunny.
Transfer frames Each preview frame clones the decoded VideoFrame and posts it to the worker.
Run AI on WebGPU RF-DETR or SmolVLM in a worker; inference stays off the main thread.
Compose on GPU Video, image overlays, then detection boxes — all in WebGPU passes.
Preview timeline Play, pause, and scrub; render and detection FPS are shown live.

Demo composition

The preview below uses this timeline (1280×720, 30 fps): one video with audio, plus two transparent PNG overlays from 1s to 3s.

export const DEMO_COMPOSITION = new Composition(30, 1280, 720)
  .addLayer(new VideoClip(demoAssetUrl('samples/video.mp4'), 0))
  .addLayer(new AudioClip(demoAssetUrl('samples/video.mp4'), 0))
  .addLayer(new ImageClip(demoAssetUrl('samples/overlay.png'), 1, 3, 0.62, 0.08, 0.32, 0.32, 0.92))
  .addLayer(new ImageClip(demoAssetUrl('samples/overlay-2.png'), 1, 3, 0, 0.08, 0.32, 0.32, 0.92));

AI model

Model

Object detection (RF-DETR · WebGPU)

Model: onnx-community/rfdetr_medium-ONNX via @huggingface/transformers with device: 'webgpu'. Detection runs on the decoded video frame (before image overlays). Only one inference runs at a time; newer frames replace pending work while inference is busy.

Enable detection Threshold 0.50

Render FPS: — · Detection FPS: —

Initializing…

WebGPU video with browser AI

Demo composition

AI model

Object detection (RF-DETR · WebGPU)

Video description (SmolVLM · WebGPU)