analyze-video
Extract keyframes from video and describe each shot with structured scene data via local Gemma 4 inference.
A local video analysis pipeline that extracts keyframes (via ffmpeg scene detection and gap-fill sampling), deduplicates frames using perceptual hashing, and describes each shot with a vision model running on Ollama. Emits structured JSON with detailed scene metadata — setting, people, objects, lighting, framing, text, quality flags — and optional audio transcription via whisper.cpp.
- Node.js
- ffmpeg / ffprobe
- Ollama
- Gemma 4
- whisper.cpp (optional)
- Blender Python (side script)
The pipeline solves the problem of manually reviewing long video content by automatically breaking it into meaningful shots and providing rich, queryable descriptions. Two-pass keyframe extraction uses ffmpeg's scene-change filter (configurable threshold, default 0.27) to catch hard cuts, then fills timeline gaps longer than 4 seconds with evenly-spaced samples. This ensures that continuous-take footage — interviews, vlogs, surveillance, tutorials — gets regular sampling even with zero cuts; a 5-minute single-take produces ~75 frames instead of 0–1. Extracted frames are then deduplicated using 8×8 grayscale perceptual hashing with configurable Hamming distance (default threshold 5), reducing token waste on static or near-identical frames.
Each frame is independently described by Gemma 4 (via local Ollama) using a cold-prompt schema that captures 11 fields: a one-sentence summary, setting, people count and array, notable objects (with explicit priority for held/exchanged items), glowing objects, framing/angle, lighting/color palette, visible text, frame quality enum (clean/motion_blur/bokeh/fade), and continuity marker (new_scene/new_shot_same_scene). The schema includes retry logic and validation to handle parsing failures, with num_predict capped to prevent truncation on crowd shots. Vision model budget, temperature, batch size, and all key parameters are configurable via constants at the top of the script or environment variables.
Optional: run Whisper.cpp in parallel (CPU-bound) while Ollama processes frames (GPU-bound), so enabling transcription adds minimal wall time. The output is a shots.json with all metadata plus optional dialogue fields populated from speech recognition segments within each shot's timespan. The project includes utilities for testing dedup thresholds and analyzing full frame dumps, plus a side experiment in make-fliers.py (a deterministic Blender script that recreates Ian Hubert's paper-flier tutorial as idempotent Blender Python).
- Two-pass keyframe extraction: scene-change detection (configurable threshold) plus gap-fill sampling for continuous-take content, ensuring even footage with zero cuts gets ~75 frames per 5 minutes instead of 1–2.
- Perceptual-hash deduplication using 8×8 grayscale pHash and Hamming distance; configurable threshold (default 5) to drop near-identical consecutive frames and reduce Ollama token cost on static/slideshow content.
- Structured shot schema with 11 fields (summary, setting, people/count, objects, glowing objects, framing, lighting, text, quality, continuity); schema validation and retry logic to ensure valid JSON output even on edge cases.
- Graceful degradation: Whisper transcription is opt-in, disabled by default, and fails silently if binary or model is missing — vision-only pipeline always completes with empty dialogue fields.
- Configurable inference pipeline: batch size, temperature, model selection (default Gemma 4), image token budget (70–1120), timeout, and all sampling parameters exposed as top-level constants or env vars; designed for easy tuning on local hardware.