The video-use project is an open-source system designed for automated video editing, leveraging large language models (LLMs) to process raw footage into a final final.mp4 output. It aims to eliminate manual editing tasks such as cutting filler words and dead space, auto color grading, burning subtitles, and generating animation overlays, applicable across diverse content types without predefined presets.

The core methodology revolves around a highly efficient and token-conscious approach where the LLM does not directly process raw video frames. Instead, it interacts with two distinct layers of structured data, minimizing computational and token costs while retaining precise editing capabilities.

1. Layer 1 – Audio Transcript: This layer is always loaded and serves as the LLM's primary reading view. It leverages ElevenLabs Scribe for comprehensive audio analysis of each source video. This analysis provides:
   • Word-level timestamps: Enabling precise cuts based on speech boundaries.
   • Speaker diarization: Identifying different speakers within the audio.
   • Audio events: Detecting and timestamping non-speech events like (laughter), (applause), and (sigh).

All this detailed transcription data from multiple takes is condensed into a compact takes_packed.md file, typically around 12KB, which the LLM consumes for strategic decision-making. The philosophy dictates that audio is primary, and visual cuts derive from speech patterns and silence gaps.

2. Layer 2 – Visual Composite: This layer is generated on demand only at critical decision points, significantly reducing token consumption compared to frame-by-frame analysis. The timeline_view utility produces a visual composite PNG incorporating a filmstrip, waveform, and word labels for specific time ranges. This visual information is requested by the LLM for:
   • Resolving ambiguous pauses.
   • Comparing alternative takes.
   • Performing sanity checks on proposed cut points.

This approach is analogous to how browser-use provides an LLM with a structured Document Object Model (DOM) instead of raw screenshots, efficiently presenting context without overwhelming the model with redundant data.

The system's pipeline operates in a sequence of stages:
$$\text{Transcribe} \longrightarrow \text{Pack} \longrightarrow \text{LLM Reasons} \longrightarrow \text{EDL} \longrightarrow \text{Render} \longrightarrow \text{Self-Eval}$$
After the initial transcription and packing of data, the LLM determines editing reasons and generates an Edit Decision List (EDL). This EDL is then rendered into a video. A crucial aspect of the pipeline is the Self-Evaluation loop, which runs timeline_view on the newly rendered output at every cut boundary. This automated quality assurance step checks for visual inconsistencies (e.g., visual jumps), audio anomalies (e.g., audio pops), and subtitle issues (e.g., hidden subtitles) before presenting the final output. If an issue is detected, the system attempts to fix and re-render (up to three iterations) without user intervention, ensuring production-correctness.

Key design principles underpinning video-use include:

• Text + on-demand visuals: Prioritizing structured text data and generating visuals only when necessary for specific decisions.
• Audio as primary: Focusing on speech boundaries and silence gaps for cutting logic.
• Iterative refinement: An Ask → confirm → execute → self-eval → persist loop ensures strategic approval before execution and continuous quality control.
• Content agnosticism: No assumptions are made about the content type; editing decisions are derived from analyzing the provided footage.
• Rule-based production correctness: While allowing artistic freedom in creative aspects, the system adheres to 12 hard rules to ensure technical correctness of the output.

The system relies on external tools like ffmpeg for video processing, uv (or pip) for Python dependency management, and optionally yt-dlp for downloading online sources. ElevenLabs API is utilized for the high-fidelity transcription service. Animation overlays are generated in parallel by sub-agents using tools such as Manim, Remotion, or PIL.