Video editing is one of the highest-friction bottlenecks in modern digital production. Whether creating developer walkthroughs, product launches, tutorials, or social reels, creators and engineers spend hours in timeline-based GUIs performing repetitive tasks: slicing dead air, cutting out filler words, aligning audio tracks, and styling subtitles.
While generative AI models have made strides in generating short synthetic clips, automating the assembly and editing of real footage has remained notoriously difficult.
The naive approach—feeding tens of thousands of video frames into multimodal vision models—burns up to 45 million tokens per project, creates massive latency, and still lacks the precision needed for clean, broadcast-ready cuts.
To solve this, the creators of browser-use have open-sourced video-use (browser-use/video-use)—a lightweight, agentic video editing framework built to turn coding agents (such as Claude Code, Codex, and Hermes) into autonomous video editors.
Here is an architectural breakdown of how video-use works, why its transcript-first strategy outperforms raw video processing, and how you can run it in your terminal.
The Fundamental Flaw: The 45-Million-Token Trap
A standard 1080p 60fps video contains 3,600 frames per minute. If an agent attempts to inspect an entire 10-minute multi-take shoot by passing video frames to a multimodal LLM:
- Token Explosion: 36,000 frames × 1,500 tokens per image = over 50 million tokens.
- Financial Drain: A single edit can easily cost hundreds of dollars in API credits.
- Loss of Temporal Precision: Vision models struggle to pinpoint the exact millisecond when a speaker finishes an "umm" or begins a syllable.
Video-use bypasses this trap with the same design philosophy that made browser-use famous: don't give the agent raw pixels when you can give it structured semantic context.
The Two-Layer Architecture of Video-Use
Video-use operates through a tightly orchestrated, two-layer abstraction model:
┌─────────────────────────────────────────────────────────────┐
│ Raw Video Takes / Clips │
└──────────────────────────────┬──────────────────────────────┘
│
┌──────────────────┴──────────────────┐
▼ ▼
[ Layer 1: Audio Transcript ] [ Layer 2: Visual Composite ]
• ElevenLabs Scribe API • On-demand only
• Word-level timestamps • Filmstrip + Waveform PNGs
• Packed into ~12KB text • Checked only at ambiguous cuts
│ │
└──────────────────┬──────────────────┘
▼
[ AI Coding Agent (Claude Code) ]
(Reasons over speech, plans EDL cuts)
│
▼
[ FFmpeg Engine + Sub-Agent Overlays ]
(30ms audio fades, subtitles, Remotion)
│
▼
[ Self-Evaluation Loop ]
(Catches visual jumps & pops)
│
▼
`edit/final.mp4`
Layer 1: The Speech Surface (~12KB)
Rather than decoding video frames, video-use processes audio using high-precision speech-to-text models (ElevenLabs Scribe).
This outputs:
- Exact millisecond word boundaries.
- Speaker diarization (distinguishing Speaker 0 from Speaker 1).
- Non-verbal audio events like
(laughter), (sigh), and (applause).
All raw takes from a shoot are compiled into a compact markdown file (takes_packed.md) that typically consumes just 12 to 15 kilobytes of text. The agent reads this transcript in milliseconds, easily identifying bad takes, stutters, and dead air.
Layer 2: On-Demand Visual Composites
Visual inspection is only triggered when strictly necessary.
If the agent encounters an ambiguous pause or needs to compare two alternative retakes, it calls a helper script (timeline_view) that renders a composite PNG containing:
- A multi-frame filmstrip.
- The synchronized audio waveform.
- Aligned word boundary markers.
This targeted approach delivers 100% visual precision at less than 1% of the token cost.
Core Automation Capabilities
- Automated Filler Removal: Automatically excises "umm", "uh", false starts, and trailing pauses based on exact word boundaries.
- Audio Micro-Fades: Splicing video footage abruptly often produces annoying digital audio clicks. Video-use automatically applies 30ms audio cross-fades at every cut boundary to guarantee studio-smooth sound.
- Dynamic Overlays & Graphics: Video-use spawns isolated parallel sub-agents to generate programmatic animations and overlays using Remotion, Manim, HyperFrames, or PIL.
- Color Grading & Subtitling: Applies consistent cinematic LUTs and burns customizable, chunked uppercase subtitles directly into the final render.
- Self-Evaluation Quality Gate: Before notifying the user, video-use runs a self-evaluator on the output. It inspects cut boundaries to ensure there are no audio pops, visual jumps, or obscured subtitles. If an issue is flagged, it corrects the EDL and re-renders automatically.
Getting Started
Setup via Agent Shell
You can install and configure video-use inside any shell-capable coding agent (Claude Code, Codex, Antigravity) with a single setup prompt:
Set up https://github.com/browser-use/video-use for me.
Read install.md first to install this repo, wire up ffmpeg, register the skill, and configure my ElevenLabs API key. Then wait for me to drop footage into a folder.
Running Your First Edit
Once installed, navigate to any directory containing your raw video takes:
cd /path/to/my-footage
claude
Inside the session, simply prompt:
Edit these raw clips into a fast-paced launch video with subtitles and clean audio.
The agent will inventory your footage, propose an edit decision list (EDL), wait for your confirmation, and render the final result to edit/final.mp4.
Summary
Video-use demonstrates the incredible leverage of combining specialized transcription APIs, deterministic FFmpeg pipelines, and AI coding agents. By structuring video around text transcripts rather than raw pixels, it transforms video editing from a tedious manual chore into a simple terminal conversation.
