# Scroll-Synced AI Video Hero Pipeline

> AI video generation workflow and WebCodecs playback for scroll-synced landing heroes

A technical workflow that pairs an AI video generation pipeline with high-performance browser rendering. It uses Flux, Seedance, and Proteus to generate and upscale smooth footage, then decodes and renders it on scroll using WebCodecs and HTML Canvas.

## How it works

Step 1: Generate first frame with Flux.1 Kontext Max → upscale with Real-ESRGAN. Step 2: Image-to-video with Seedance 1 Lite for experiments → Seedance 1 Pro for 1080p → Proteus v4 upscale to 1440p @60FPS. Step 3: Demux in-browser with mediabunny (npm i mediabunny) → decode frames with a VideoDecoder instance → paint via Canvas 2D Context API.

## What it is for

- Build smooth scroll-driven video backgrounds for landing pages
- Generate consistent high-resolution video assets using AI image-to-video tools
- Decode and control video frames client-side without standard video element playback lag

## Key points

- Step 1: Base frame generation with Flux.1 Kontext Max and Real-ESRGAN upscaling
- Step 2: Image-to-video animation via Seedance (Lite for testing, Pro for production)
- Step 3: Upscaling to 1440p at 60 FPS using Proteus v4
- Step 4: In-browser demuxing using mediabunny
- Step 5: Frame decoding with the WebCodecs VideoDecoder API and Canvas 2D rendering

## Pricing

Free

## Details

- Type: Tip
- Category: [Development](https://xiry.app/topics/development.md)
- Open source: no
- Platforms: Web
- For: Creative technologists and frontend developers building interactive landing pages
- Tags: webcodecs, ai video, seedance, canvas, flux, scroll animation

## Why it was saved

- [@konstipaulus, 10 August 2025](https://x.com/konstipaulus/status/1954539977812980030): Reproducible tutorial combining current AI video tools with WebCodecs implementation.

## Related

- [webcodecs-scroll-sync](https://xiry.app/resources/webcodecs-scroll-sync): Synchronize video playback to page scroll using the WebCodecs API (Web app, Open source)
- [@remotion/webcodecs](https://xiry.app/resources/remotion-webcodecs): In-browser video and audio conversion library powered by the WebCodecs API. (Web app, Freemium)
- [Apple-Style Scroll-Driven Canvas Animation Pipeline](https://xiry.app/resources/apple-style-scroll-driven-canvas-animation-pipeline): A workflow for building high-performance scroll-scrubbed canvas animations using WebP frames. (Tip, Free)
- [Canvas UI](https://xiry.app/resources/canvas-ui): Open-source html-in-canvas UI component library powered by WebGL and WebGPU (Web app, Free)
- [COBE](https://xiry.app/resources/cobe): Lightweight 5KB WebGL globe with CSS anchor positioning support (Web app, Open source)
- [Mediabunny](https://xiry.app/resources/mediabunny): Fast, dependency-free JavaScript media toolkit for browser-based video and audio processing (Web app, Open source)

Updated 11 October 2026. Page: https://xiry.app/resources/scroll-synced-ai-video-hero-pipeline

Make your own library: Xiry takes your saved reels, tweets, Reddit bookmarks and Watch Later videos, makes them searchable and pulls the useful stuff out, like this library. It's invite-only for now; join the waitlist at https://xiry.app/#waitlist.