8# VideoDB Skill
9
10**Perception + memory + actions for video, live streams, and desktop sessions.**
11
12Use this skill when you need to:
13
14## 1) Desktop Perception
15- Start/stop a **desktop session** capturing **screen, mic, and system audio**
16- Stream **live context** and store **episodic session memory**
17- Run **real-time alerts/triggers** on what’s spoken and what's happening on screen
18- Produce **session summaries**, a searchable timeline, and **playable evidence links**
19
20## 2) Video ingest + stream
21- Ingest a **file or URL** and return a **playable web stream link**
22- Transcode/normalize: **codec, bitrate, fps, resolution, aspect ratio**
23
24## 3) Understand + index + retrieve (timestamps + evidence)
25- **Understand**: run analyzers over speech, scenes, objects, OCR, brands, activity
26- **Index**: turn analyzer artifacts into semantic, filterable, and aggregatable indexes
27- **Retrieve**: search moments, ask questions, filter exactly, count and group — with **timestamps** and **playable evidence**
28- Auto-create **clips** from results
29
30## 4) Timeline editing + generation
31- Subtitles: **generate**, **translate**, **burn-in**
32- Overlays: **text/image/branding**, motion captions
33- Audio: **background music**, **voiceover**, **dubbing**
34- Programmatic composition and exports via **timeline operations**
35
36## 5) Live streams (RTSP) + monitoring
37- Connect **RTSP/live feeds**
38- Run **real-time visual and spoken understanding** and emit **events/alerts** for monitoring workflows
39
40---
41
42## Common inputs
43- Local **file path**, public **URL**, or **RTSP URL**
44- Desktop capture request: **start / stop / summarize session**
45- Desired operations: get context for understanding, transcode spec, index spec, search query, clip ranges, timeline edits, alert rules
46
47## Common outputs
48- **Stream URL** — make it playable: https://console.videodb.io/player?url={STREAM_URL}
49- Search results with **timestamps** and **evidence links**
50- Generated assets: subtitles, audio, images, clips
51- **Event/alert payloads** for live streams
52- Desktop **session summaries** and memory entries
53
54---
55
56## Canonical prompts (examples)
57- “Start desktop capture and alert when a password field appears.”
58- “Record my session and produce an actionable summary when it ends.”
59- “Ingest this file and return a playable stream link.”
60- “Index this folder and find every scene with people, return timestamps.”
61- “Generate subtitles, burn them in, and add light background music.”
62- “Connect this RTSP URL and alert when a person enters the zone.”
63
64
65## Running Python code
66
67**CRITICAL:** Always cd to the user's project directory before running Python code. This ensures load_dotenv(".env") finds the correct .env file.
68
69```python
70from dotenv import load_dotenv
71load_dotenv(".env")
72
73import videodb
74conn = videodb.connect()
75```
76
77This reads VIDEO_DB_API_KEY from:
781. Environment (if already exported)
792. Project's .env file in current directory
80
81If the key is missing, videodb.connect() raises AuthenticationError automatically.
82
83Do NOT write a script file when a short inline command works.
84
85When writing inline Python (python -c "..."), always use properly formatted code — use semicolons to separate statements and keep it readable. For anything longer than ~3 statements, use a heredoc instead:
86
87```bash
88python << 'EOF'
89from dotenv import load_dotenv
90load_dotenv(".env")
91
92import videodb
93conn = videodb.connect()
94coll = conn.get_collection()
95print(f"Videos: {len(coll.get_videos())}")
96EOF
97```
98
99## Setup
100
101When the user asks to "setup videodb" or similar:
102
103### 1. Install SDK
104
105```bash
106pip install "videodb[capture]>=0.5.0" python-dotenv
107```
108
109If videodb[capture] fails on Linux, install without the capture extra:
110
111```bash
112pip install "videodb>=0.5.0" python-dotenv
113```
114
115The >=0.5.0 pin matters — the understand/index/ask/aggregate APIs do not exist in earlier versions.
116
117### 2. Configure API key
118
119The user must set VIDEO_DB_API_KEY using **either** method:
120
121- **Export in terminal (recommended)**: export VIDEO_DB_API_KEY=your-key
122- **Project .env file**: Save VIDEO_DB_API_KEY=your-key in the project's .env file
123
124Get a free API key at https://console.videodb.io (50 free uploads, no credit card).
125
126**Do NOT** read, write, or handle the API key yourself. Always let the user set it.
127
128## Quick Reference
129
130### Upload media
131
132```python
133# URL
134video = coll.upload(url="https://example.com/video.mp4")
135
136# YouTube
137video = coll.upload(url="https://www.youtube.com/watch?v=VIDEO_ID")
138
139# Local file
140video = coll.upload(file_path="/path/to/video.mp4")
141```
142
143### Understand → index → retrieve (default path)
144
145Three stages. Run analyzers to produce **artifacts**, index each artifact, then retrieve.
146
147```python
148import time
149
150# 1. Understand. Naming each analyzer keeps analyzer.name meaningful downstream.
151understanding = video.understand(
152 analyzers=[
153 {"type": "spoken_words", "name": "transcript"},
154 {"type": "vlm", "name": "scene",
155 "config": {"prompt": "Describe the scene and any on-screen text."}},
156 ],
157 segmentation={"type": "shot", "threshold": 30},
158)
159
160# A run with a failed or skipped analyzer ends partial, which the SDK does not
161# treat as terminal — wait_until_complete() would poll to TimeoutError. Poll the
162# analyzers instead. The analyzers and guard is load-bearing: a refresh can
163# transiently return an empty list, and all([]) is True, which would exit the
164# loop while the run is still going.
165deadline = time.time() + 3600
166while time.time() < deadline:
167 analyzers = understanding.refresh().list_analyzers()
168 if analyzers and all(a.is_complete for a in analyzers):
169 break
170 time.sleep(15)
171
172# 2. Index each artifact that succeeded.
173for analyzer in understanding.list_analyzers():
174 if analyzer.is_successful:
175 video.index(source=analyzer, name=analyzer.name).wait_until_complete()
176
177# 3. Retrieve.
178response = video.search("discussion about pricing")
179for shot in response.shots:
180 print(f"[{shot.start:.1f}s - {shot.end:.1f}s] {shot.text}")
181if response.response_type in ("shots", "deepsearch") and len(response):
182 stream_url = response.compile() # raises SearchError otherwise
183```
184
185Analyzer types: spoken_words (→ artifact transcript), vlm (→ scene), object_detection (→ objects), ocr, brand_detection (→ brands), activity_recognition (→ activity), location_detection (→ location), faces, audio_event_detection. They are plain strings — there is no SDK enum.
186
187See [reference/indexing.md](reference/indexing.md) for segmentation, sampling, field configuration, and cost tuning.
188
189### Retrieval
190
191search(query) is the default — it plans the retrieval and picks the indexes itself. Reach past it when you need something specific:
192
193```python
194# Target a specific index, with a relevance floor
195video.semantic_search("a customer holding the product", index_names=["scene"], score_threshold=0.7)
196
197# Exact filtering, no natural-language interpretation
198video.query(index_name="objects",
199 filter=[{"field": "frames.detections.label", "op": "contains", "value": "car"}])
200
201# Counts and facets — returns the raw server payload, not a SearchResult
202video.aggregate(index_name="objects", group_by="frames.detections.label", metric="count")
203
204# A written answer plus the moments it came from
205answer = video.ask("What did they say about pricing?", include_sources=True)
206```
207
208All five exist on Collection too, fanning out across every indexed video. See [reference/search.md](reference/search.md).
209
210**search() now returns SearchResponse, not SearchResult.** get_shots(), compile(), play(), and iteration all work, but there is no .stream_url on it — use .compile().
211
212### Transcript + subtitle
213
214```python
215# force=True skips the error if the video is already indexed
216video.index_spoken_words(force=True)
217text = video.get_transcript_text()
218stream_url = video.add_subtitle()
219```
220
221index_spoken_words() is the correct call here even on 0.5.0 — add_subtitle() and CaptionAsset(src="auto") read the v1 spoken-word index. A v2 spoken_words artifact does not substitute for it. This is the one place v1 indexing is still the right answer.
222
223### Legacy indexing (existing codebases)
224
225```python
226# v1 API — still supported in 0.5.0, not deprecated. New code should use the v2 path above.
227from videodb import IndexType
228
229video.index_spoken_words(force=True)
230scene_index_id = video.index_scenes(prompt="Describe the visual content.")
231results = video.legacy_search(
232 "person writing on a whiteboard",
233 index_type=IndexType.scene,
234 scene_index_id=scene_index_id,
235)
236```
237
238Recognise this pattern in existing repos and leave it alone unless asked to migrate — it still works. See [reference/migration.md](reference/migration.md) to port it, or [reference/legacy/search.md](reference/legacy/search.md) to maintain it.
239
240### Timeline editing
241
242Use the Editor API to compose videos, images, audio, and text. See [reference/editor.md](reference/editor.md) for full workflow.
243
244```python
245from videodb.editor import Timeline, Track, Clip, VideoAsset, ImageAsset, AudioAsset, Fit
246
247timeline = Timeline(conn)
248timeline.resolution = "1280x720"
249
250video_track = Track()
251video_track.add_clip(0, Clip(asset=VideoAsset(id=video.id, start=10), duration=20))
252
253audio_track = Track()
254audio_track.add_clip(0, Clip(asset=AudioAsset(id=music.id, volume=0.2), duration=20))
255
256timeline.add_track(video_track)
257timeline.add_track(audio_track)
258stream_url = timeline.generate_stream()
259```
260
261### Transcode video (resolution / quality change)
262
263```python
264from videodb import TranscodeMode, VideoConfig, AudioConfig
265
266# Change resolution, quality, or aspect ratio server-side
267job_id = conn.transcode(
268 source="https://example.com/video.mp4",
269 callback_url="https://example.com/webhook",
270 mode=TranscodeMode.economy,
271 video_config=VideoConfig(resolution=720, quality=23, aspect_ratio="16:9"),
272 audio_config=AudioConfig(mute=False),
273)
274```
275
276### Reframe aspect ratio (for social platforms)
277
278**Warning:** reframe() is a slow server-side operation. For long videos it can take
279several minutes and may time out. Best practices:
280- Always limit to a short segment using start/end when possible
281- For full-length videos, use callback_url for async processing
282- Trim the video on a Timeline first, then reframe the shorter result
283
284```python
285from videodb import ReframeMode
286
287# Always prefer reframing a short segment:
288reframed = video.reframe(start=0, end=60, target="vertical", mode=ReframeMode.smart)
289
290# Async reframe for full-length videos (returns None, result via webhook):
291video.reframe(target="vertical", callback_url="https://example.com/webhook")
292
293# Presets: "vertical" (9:16), "square" (1:1), "landscape" (16:9)
294reframed = video.reframe(start=0, end=60, target="square")
295
296# Custom dimensions
297reframed = video.reframe(start=0, end=60, target={"width": 1280, "height": 720})
298```
299
300### Generative media
301
302```python
303image = coll.generate_image(
304 prompt="a sunset over mountains",
305 aspect_ratio="16:9",
306)
307```
308
309### Sandbox Compute (self-hosted / open-weight models)
310
311Run open-weight models (Gemma, Qwen, Whisper, OmniVoice, FLUX, RT-DETR) by creating a sandbox and passing sandbox_id to a supported job. Requires videodb>=0.5.1.
312
313```python
314from videodb import SandboxTier, SandboxModel
315
316# 1. Create a sandbox sized for the largest model, then wait until active.
317sandbox = conn.create_sandbox(
318 tier=SandboxTier.medium,
319 models=[SandboxModel.GEMMA_4_31B.value], # exact ID, NO -FP8 suffix
320)
321sandbox.wait_for_ready(timeout=300, interval=5)
322
323# 2. Understanding: set config.model + config.sandbox_id on the analyzer.
324understanding = video.understand(analyzers=[{
325 "type": "vlm", "name": "scene",
326 "config": {"model": "google/gemma-4-31B-it", "sandbox_id": sandbox.id,
327 "prompt": "Describe the scene."},
328}])
329
330# 2b. Generation: pass model_name + sandbox_id (jobs return GenerationJob → .wait()).
331response = coll.generate_text(prompt="Summarize this.", model_name="Qwen/Qwen3.5-9B",
332 sandbox_id=sandbox.id, max_tokens=300)
333job = coll.generate_image(prompt="a city at sunset", model_name="black-forest-labs/FLUX.1-dev",
334 sandbox_id=sandbox.id)
335image = job.wait(timeout=900, interval=5)
336
337# 3. Stop when done — provisioning/active/alert all count toward the tier limit.
338sandbox.stop(); sandbox.wait_for_stop()
339```
340
341Model IDs must match the catalog exactly (**no -FP8 suffix**) or create_sandbox raises Unsupported sandbox model. See [reference/sandbox.md](reference/sandbox.md) for the full model catalog, tiers, categories, pricing, and pitfalls.
342
343## Error handling
344
345```python
346from videodb.exceptions import AuthenticationError, InvalidRequestError
347
348try:
349 conn = videodb.connect()
350except AuthenticationError:
351 print("Check your VIDEO_DB_API_KEY")
352
353try:
354 video = coll.upload(url="https://example.com/video.mp4")
355except InvalidRequestError as e:
356 print(f"Upload failed: {e}")
357```
358
359### Common pitfalls
360
361| Scenario | Error message | Solution |
362|----------|--------------|----------|
363| Search result has no stream URL | AttributeError: 'SearchResponse' object has no attribute 'stream_url' | search() returns SearchResponse in 0.5.0. Use results.compile() |
364| search(score_threshold=) searches the wrong indexes | no error, unexpected results | score_threshold does not route to legacy. Use semantic_search(score_threshold=), or legacy_search() for v1 indexes |
365| Semantic index on object detection | use_for includes semantic but no scene has embeddable text | Object artifacts have no top-level text. Omit use_for (it degrades automatically) or pass ["query", "aggregate"] |
366| Indexing a field that does not exist | fields.filter names not present in any scene's data | The error lists the available field names — read it. Or check index.field_schema |
367| Search finds no matches | v2 returns an empty SearchResponse; only legacy_search() raises InvalidRequestError: No results found | Check len(response). Wrap only legacy calls in try/except |
368| Indexing an already-indexed video (v1) | Spoken word index for video already exists | Use video.index_spoken_words(force=True) to skip if already indexed |
369| Reframe times out | Blocks indefinitely on long videos | Use start/end to limit segment, or pass callback_url for async |
370| Negative timestamps on Timeline | Silently produces broken stream | Always validate start >= 0 before creating VideoAsset |
371| generate_video() / create_collection() fails | Operation not allowed or maximum limit | Plan-gated features — inform the user about plan limits |
372
373## Additional docs
374
375Reference documentation is in the reference/ directory adjacent to this SKILL.md file. Use the Glob tool to locate it if needed.
376
377- [reference/api-reference.md](reference/api-reference.md) - Complete VideoDB Python SDK API reference
378- [reference/indexing.md](reference/indexing.md) - Understand → index pipeline: analyzers, artifacts, segmentation, field configuration
379- [reference/indexing-reference.md](reference/indexing-reference.md) - Analyzer catalog and Understanding/Index class reference
380- [reference/search.md](reference/search.md) - Retrieval guide: search, ask, semantic_search, query, aggregate
381- [reference/search-reference.md](reference/search-reference.md) - Retrieval signatures, filter syntax, response objects
382- [reference/migration.md](reference/migration.md) - v1 → v2 mapping and SDK 0.5.0 breaking changes. Read when you find v1 code
383- [reference/editor.md](reference/editor.md) - Timeline editing workflow guide (4-layer model, use cases, examples)
384- [reference/editor-reference.md](reference/editor-reference.md) - Editor code reference (constructors, parameters, enums)
385- [reference/streaming.md](reference/streaming.md) - HLS streaming and instant playback
386- [reference/generative.md](reference/generative.md) - AI-powered media generation (images, video, audio)
387- [reference/sandbox.md](reference/sandbox.md) - Sandbox Compute workflow (run open-weight models: Gemma, Qwen, Whisper, OmniVoice, FLUX, RT-DETR)
388- [reference/sandbox-reference.md](reference/sandbox-reference.md) - Sandbox code reference (create/get/list/stop, tiers, model catalog, sandbox-aware generation)
389- [reference/rtstream.md](reference/rtstream.md) - Live stream ingestion workflow (RTSP/RTMP)
390- [reference/rtstream-reference.md](reference/rtstream-reference.md) - RTStream SDK methods and AI pipelines
391- [reference/capture.md](reference/capture.md) - Desktop capture workflow
392- [reference/capture-reference.md](reference/capture-reference.md) - Capture SDK and WebSocket events
393- [reference/use-cases.md](reference/use-cases.md) - Common video processing patterns and examples
394
395Legacy v1 indexing and search. These APIs still work and are not deprecated, but read these only when maintaining existing v1 code:
396
397- [reference/legacy/index.md](reference/legacy/index.md) - v1 scene indexing and frame extraction workflow
398- [reference/legacy/index-reference.md](reference/legacy/index-reference.md) - v1 scene index code reference (SceneCollection/Scene/Frame)
399- [reference/legacy/search.md](reference/legacy/search.md) - v1 spoken-word and scene search
400
401## Screen Recording (Desktop Capture)
402
403Use ws_listener.py to capture WebSocket events during recording sessions. Desktop capture supports **macOS** only.
404
405### Quick Start
406
4071. **Start listener**: python scripts/ws_listener.py --cwd=<PROJECT_ROOT> &
4082. **Get WebSocket ID**: cat /tmp/videodb_ws_id
4093. **Run capture code** (see reference/capture.md for full workflow)
4104. **Events written to**: /tmp/videodb_events.jsonl
411
412### Query Events
413
414```python
415import json
416events = [json.loads(l) for l in open("/tmp/videodb_events.jsonl")]
417
418# Get all transcripts
419transcripts = [e["data"]["text"] for e in events if e.get("channel") == "transcript"]
420
421# Get visual descriptions from last 5 minutes
422import time
423cutoff = time.time() - 300
424recent_visual = [e for e in events
425 if e.get("channel") == "visual_index" and e["unix_ts"] > cutoff]
426```
427
428### Utility Scripts
429
430- [scripts/ws_listener.py](scripts/ws_listener.py) - WebSocket event listener (dumps to JSONL)
431
432For complete capture workflow, see [reference/capture.md](reference/capture.md).
433
434
435**Do not use ffmpeg, moviepy, or local encoding tools** when VideoDB supports the operation. The following are all handled server-side by VideoDB — trimming, combining clips, overlaying audio or music, adding subtitles, text/image overlays, transcoding, resolution changes, aspect-ratio conversion, resizing for platform requirements, transcription, volume control, fade transitions, and media generation. Only fall back to local tools for operations listed under Limitations in reference/editor.md (speed changes, crop/zoom, colour grading, keyframe animation).
436
437### When to use what
438
439| Problem | VideoDB solution |
440|---------|-----------------|
441| Make a video searchable | video.understand(analyzers=[...]) then video.index(source=analyzer) |
442| Find moments by what was said or shown | video.search(query), or semantic_search(index_names=[...]) to target an index |
443| Answer a question about a video | video.ask(question, include_sources=True) |
444| Count or group what appears in a video | video.aggregate(index_name=..., group_by=..., metric="count") |
445| Filter moments on exact field values | video.query(index_name=..., filter={...}) |
446| Platform rejects video aspect ratio or resolution | video.reframe() or conn.transcode() with VideoConfig |
447| Need to resize video for Twitter/Instagram/TikTok | video.reframe(target="vertical") or target="square" |
448| Need to change resolution (e.g. 1080p → 720p) | conn.transcode() with VideoConfig(resolution=720) |
449| Need to overlay audio/music on video | AudioAsset on an Editor Timeline with volume control |
450| Need to add subtitles | video.add_subtitle() or CaptionAsset on Editor Timeline |
451| Need to combine/trim clips | VideoAsset on an Editor Timeline |
452| Need to compose images with voiceover | ImageAsset + AudioAsset on separate Editor tracks |
453| Need to generate voiceover, music, or SFX | coll.generate_voice(), generate_music(), generate_sound_effect() |
454