xiry.

Development · Tip

Scroll-Synced AI Video Hero Pipeline

AI video generation workflow and WebCodecs playback for scroll-synced landing heroes

From the Xiry library · Updated 11 October 2026

A technical workflow that pairs an AI video generation pipeline with high-performance browser rendering. It uses Flux, Seedance, and Proteus to generate and upscale smooth footage, then decodes and renders it on scroll using WebCodecs and HTML Canvas.

How it works

Step 1: Generate first frame with Flux.1 Kontext Max → upscale with Real-ESRGAN. Step 2: Image-to-video with Seedance 1 Lite for experiments → Seedance 1 Pro for 1080p → Proteus v4 upscale to 1440p @60FPS. Step 3: Demux in-browser with mediabunny (npm i mediabunny) → decode frames with a VideoDecoder instance → paint via Canvas 2D Context API.

What it's for

  • Build smooth scroll-driven video backgrounds for landing pages
  • Generate consistent high-resolution video assets using AI image-to-video tools
  • Decode and control video frames client-side without standard video element playback lag

Key points

  • Step 1: Base frame generation with Flux.1 Kontext Max and Real-ESRGAN upscaling
  • Step 2: Image-to-video animation via Seedance (Lite for testing, Pro for production)
  • Step 3: Upscaling to 1440p at 60 FPS using Proteus v4
  • Step 4: In-browser demuxing using mediabunny
  • Step 5: Frame decoding with the WebCodecs VideoDecoder API and Canvas 2D rendering

Pricing

Free

Details

Type
Tip
Category
Development
Topic
Frontend Animation
Open source
No
Platforms
Web
For
Creative technologists and frontend developers building interactive landing pages
Tags
webcodecs, ai video, seedance, canvas, flux, scroll animation

Why it was saved

  • @konstipaulus, 10 August 2025: Reproducible tutorial combining current AI video tools with WebCodecs implementation.

Related

  • webcodecs-scroll-sync

    Synchronize video playback to page scroll using the WebCodecs API

  • @remotion/webcodecs

    In-browser video and audio conversion library powered by the WebCodecs API.

  • Apple-Style Scroll-Driven Canvas Animation Pipeline

    A workflow for building high-performance scroll-scrubbed canvas animations using WebP frames.

  • Canvas UI

    Open-source html-in-canvas UI component library powered by WebGL and WebGPU

  • COBE

    Lightweight 5KB WebGL globe with CSS anchor positioning support

  • Mediabunny

    Fast, dependency-free JavaScript media toolkit for browser-based video and audio processing

Updated