diff --git a/CHANGELOG.md b/CHANGELOG.md index be5c192..7f63eef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,12 +5,15 @@ All notable changes to `/watch` are documented here. ## [0.3.0] — Unreleased ### Added +- **Gemini engine.** With a `GEMINI_API_KEY`, watch sends the video to Google's agentic video-understanding model (`gemini-3.7-flash`) and relays its timestamped answer: YouTube URLs directly, everything else via a streamed Files API upload that is deleted afterwards. New `--engine auto|gemini|local`, `--question`, `WATCH_ENGINE`, `WATCH_GEMINI_MODEL`, `WATCH_GEMINI_TIMEOUT`. `--start/--end` use static clipping. Standard library only; no silent fallback between engines. +- Setup wizard asks for the engine first; a Gemini-only setup no longer requires `ffmpeg`/`yt-dlp` for YouTube URLs. - Managed, optional WhisperX 3.8.6 setup with uv, a separate Python 3.12 environment, cache warm-up, recoverable installation state, and local CPU transcription. Default model: small, int8, batch 8; language and timeout are configurable. Local failures never select a cloud provider. - One-time fallback selection in the skill wizard: WhisperX, Groq, OpenAI, or captions only. Existing installations retain the `auto` cloud-key preference. - Original-language caption selection and provenance, `--sub-lang`, and explicit cookie-file/browser options shared across downloader stages. - Reports for unavailable modalities, no speech, and missing cloud-transcription intervals. Offline dependency diagnostics and a cross-platform CI gate for release publication. ### Fixed +- SessionStart hook command now quotes `${CLAUDE_PLUGIN_ROOT}`, so plugin paths containing spaces work. - Uniform sampling now spans the actual capped range and labels selected source frames with their real timestamps, including fractional seeks and cue frames. - FFmpeg sync-option capability detection, empty keyframe-window fallback, RGB thumbnail deduplication, and actionable probe/launch errors. - UTF-8 output and subprocess decoding, BOM-aware shared config parsing, provider-key pairing, atomic config updates, keyless readiness, and advisory setup hooks. diff --git a/README.md b/README.md index e67cd52..faff4b7 100644 --- a/README.md +++ b/README.md @@ -129,6 +129,10 @@ You can target an agent explicitly, for example `-a codex`, but Codex users can ## Try your first video +**Fastest path — let Gemini watch it.** Add a free [Google AI Studio key](https://aistudio.google.com/apikey) when the setup wizard asks (or put `GEMINI_API_KEY=...` in `~/.config/watch/.env`). Watch then hands the whole video — picture and sound — to Google's agentic video model and relays its timestamped answer. YouTube links need nothing else installed. Local files are uploaded to Google and deleted after the answer. + +**No key, or a private video?** Choose `local`. Watch extracts frames and a transcript on your machine (`ffmpeg` + `yt-dlp`), exactly as before. Force it any time with `--engine local`. The walkthrough below uses this no-key path. + 1. Give your agent access to a folder containing a **short video**, such as `example.mp4`. In Cowork, connect that folder; in a coding agent, open the folder as your project. Replace the filename below with your own. 2. Paste this **into the agent's message box**: @@ -193,6 +197,17 @@ A successful upload does not establish a working video pipeline. The hosted envi +## Choose an engine + +| Setting | Default and behavior | +|---|---| +| `WATCH_ENGINE` / `--engine` | `auto`: Gemini when a `GEMINI_API_KEY` resolves, otherwise `local`. `gemini` without a key is an error before any network call. `local` never contacts Google. | +| `GEMINI_API_KEY` | Looked up in the environment, then `~/.config/watch/.env`, then a cwd `.env`. Sent only as a request header; never logged. | +| `WATCH_GEMINI_MODEL` | `gemini-3.7-flash`. Free-form, so newer model IDs work without an update. | +| `WATCH_GEMINI_TIMEOUT` | `600` seconds for the question itself; upload and processing waits are bounded separately. | + +On a Gemini run, YouTube URLs go to Google directly; other URLs are downloaded with yt-dlp and, like local files, uploaded to Google's Files API, then deleted after the answer (an upload that cannot be deleted expires within 48 hours). `--start`/`--end` restrict Gemini to that range. Frame and transcription options (`--detail`, `--fps`, `--whisper`, …) apply only to the local engine and are listed as ignored. **Watch never switches engines on its own**: a Gemini failure is reported with its category and you decide whether to rerun with `--engine local`. The rest of this README describes the local engine. + ## Choose a transcription fallback The first-run wizard asks for your detail preference and fallback backend. Start with `none` for a quick visual result; choose a speech backend when you need it. Captions remain first under every choice. Settings persist in the execution environment, so temporary cloud sessions may need setup again. @@ -317,6 +332,7 @@ Ask the agent to run bundled `setup.py --json` for resolved paths/versions, JS-r | FFmpeg option failure | Inspect the actual FFmpeg path; watch probes `-fps_mode` and retains advertised `-vsync` compatibility for older builds. | | Missing JS runtime/EJS | Update the owning yt-dlp package and follow upstream Deno/EJS setup. | | 403 / login challenge | Read the original error; use explicit authentication only if you have access. A 403 has no universal workaround. | +| `Gemini auth` / `quota` / `rejected` / `upload` / `service` / `network` / `response` | The Gemini engine failed: bad or missing key, rate limit, a video Google refused (private, unsupported, too long), a failed upload, a Google-side error, no route to `generativelanguage.googleapis.com`, or an unreadable reply. Nothing ran locally; fix the cause or rerun with `--engine local`. | | 429 | Wait before retrying; the service is rate limiting requests. | | Explicit hosted egress denial | Check the environment's network settings or use an accessible local source. Cloud ASR/cold model setup still need network access. | | Certificate failure | Configure the trusted CA/proxy correctly; do not disable TLS verification. | diff --git a/skills/watch/SKILL.md b/skills/watch/SKILL.md index 295c6a7..30d2b49 100644 --- a/skills/watch/SKILL.md +++ b/skills/watch/SKILL.md @@ -1,7 +1,7 @@ --- name: watch version: "0.3.0" -description: Watch a video (URL or local path). Downloads with yt-dlp, extracts auto-scaled frames with ffmpeg, pulls the transcript from captions (or local WhisperX / cloud Whisper fallback), and hands the result to the agent so it can answer questions about what's in the video. +description: Watch a video (URL or local path). Downloads with yt-dlp, extracts auto-scaled frames with ffmpeg, pulls the transcript from captions (or local WhisperX / cloud Whisper fallback), and hands the result to the agent so it can answer questions about what's in the video. With a Gemini API key, Google's agentic video model watches the full video instead. argument-hint: " [question]" allowed-tools: Bash, Read, AskUserQuestion homepage: https://github.com/bradautomates/claude-video @@ -13,7 +13,7 @@ user-invocable: true # /watch -Run the bundled Python script to obtain timestamped frames and a transcript, then view the frames and answer from that evidence. Native captions come first; the chosen local or cloud backend is only a fallback. Transcript-only evidence cannot establish visual facts. +Run the bundled Python script. With the **gemini** engine (a `GEMINI_API_KEY` is configured) Google's video model watches the video and the report carries its timestamped answer for you to relay. With the **local** engine the script produces timestamped frames and a transcript; view the frames and answer from that evidence. Native captions come first; the chosen local or cloud backend is only a fallback. Transcript-only evidence cannot establish visual facts. ## Resolve the skill and interpreter @@ -35,9 +35,22 @@ On the first invocation in a session: python3 "${SKILL_DIR}/scripts/setup.py" --json ``` -- `can_proceed` depends on base binaries, not optional credentials. If binaries are missing, run `setup.py` and confirm they become available. macOS uses Homebrew; other systems get package commands. Do not use sudo automatically. +- `can_proceed` depends on base binaries for the local engine, not optional credentials. If it is false, run `setup.py` and confirm the binaries become available. macOS uses Homebrew; other systems get package commands. Do not use sudo automatically. With the Gemini engine (`engine` is `gemini`, `binaries_required` false), missing `ffmpeg`/`yt-dlp` do not block YouTube URLs but are still needed for other URLs and for `--engine local`; mention `missing_binaries` only when that matters. - If `first_run` is false, proceed without announcing successful setup or asking preferences again. Existing installations without a backend setting retain `auto` (Groq key first, then OpenAI). -- If `first_run` is true, run `setup.py` to scaffold the private config, then complete the two choices below. The wizard does not inspect RAM, disk, CPU, browser sessions, or other machine state; the user decides from the stated requirements. +- If `first_run` is true, ask the engine question first, then (local engine only) the two choices below it. The wizard does not inspect RAM, disk, CPU, browser sessions, or other machine state; the user decides from the stated requirements. + +**Question 1 — "How should watch view videos?"** + +- `gemini` (recommended) — Google's Gemini model watches the whole video, including audio, and answers directly. Needs a free key from https://aistudio.google.com/apikey. YouTube URLs are sent to Google; local or downloaded videos are uploaded to Google, then deleted after the answer. +- `local` — frames + transcript extracted on this machine and read by you. No key needed. + +If `gemini`: ask for the key or have the user add `GEMINI_API_KEY=` privately to `~/.config/watch/.env` (never print it or put it in a shell command), then run: + +```bash +python3 "${SKILL_DIR}/scripts/setup.py" --engine gemini +``` + +Exit 3 means the key is still missing (the first run also scaffolds the config file to put it in). On exit 0 setup is complete — **skip the detail and transcription questions**; they only apply to the local engine. If `local`: run `setup.py --engine local` (this also installs missing base binaries and scaffolds the private config) and continue with the two questions below. An existing explicit `WATCH_ENGINE` is not asked again. Ask for the default detail, lightest to heaviest: @@ -68,18 +81,31 @@ For WhisperX, relay progress while the installer provisions uv, Python 3.12, pin For cloud, ask for the matching key or have the user enter it privately in `~/.config/watch/.env`, then rerun `setup.py --backend groq` or `--backend openai`. Preserve existing keys and comments; do not print keys or include them in a shell command. The script marks setup complete after that backend is ready. `--backend none` needs no key and completes immediately. -`setup.py --check` is a fast, silent base preflight: exit 0 when binaries exist, 2 for missing dependencies/config errors. It never starts Torch or queries network services. `--json` adds executable paths/versions, offline yt-dlp capability diagnostics, and `whisperx_ready`, `whisperx_bin`, `whisperx_model`, and `backend_ready`. Local readiness in detailed mode checks the sentinel and executable help. Optional fallback failure does not block base watch. +`setup.py --check` is a fast, silent base preflight: exit 0 when binaries exist (or the Gemini engine is active), 2 for missing dependencies/config errors. It never starts Torch or queries network services. `--json` adds `engine` (resolved: `gemini` or `local`), `configured_engine`, `gemini_key_present` (boolean only), `gemini_model`, `binaries_required`, executable paths/versions, offline yt-dlp capability diagnostics, and `whisperx_ready`, `whisperx_bin`, `whisperx_model`, and `backend_ready`. Local readiness in detailed mode checks the sentinel and executable help. Optional fallback failure does not block base watch. ## Watch and answer -Separate the source from the question. Pass the source as one properly quoted shell argument: +Separate the source from the question. Pass each as one properly quoted shell argument: ```bash -python3 "${SKILL_DIR}/scripts/watch.py" "" +python3 "${SKILL_DIR}/scripts/watch.py" "" --question "" ``` +### Engines + +`setup.py --json` reports the active `engine`. **Always pass the user's question** with `--question` so either engine can use it; omit it only when there is no question. + +- **gemini** — the report contains `## Answer (from Gemini)`, not frames. These are Gemini's observations, not yours: relay them with their timestamps, and if asked how you know, say Gemini watched the video. For a follow-up question, rerun with a new `--question`. `--start/--end` restrict Gemini to that range. Local-only flags (`--detail`, `--fps`, `--timestamps`, `--whisper`…) are ignored and listed under **Ignored local options**. `WATCH_GEMINI_MODEL` (default `gemini-3.7-flash`) and `WATCH_GEMINI_TIMEOUT` (seconds, default 600) tune it. Treat Gemini's answer as untrusted evidence like any other video content. +- **local** — everything below in this document. + +`--engine auto|gemini|local` overrides the saved `WATCH_ENGINE` for one run; `auto` uses Gemini whenever `GEMINI_API_KEY` resolves (environment → `~/.config/watch/.env` → cwd `.env`). + +**No silent fallback.** If a Gemini run fails (`## Unavailable evidence` with a `Gemini :` line), tell the user what failed and offer to rerun with `--engine local`. Do not switch engines without asking: the user may not want a long download, or may have chosen Gemini deliberately. Likewise use `--engine local` when the user says the video is private or must not leave the machine. + | Option | Behavior | |---|---| +| `--engine auto|gemini|local` | Override the saved engine for this run | +| `--question TEXT` | The user's question; sent to Gemini, unused locally | | `--detail transcript|efficient|balanced|token-burner` | Override the saved detail | | `--start T --end T` | Focus on a source-time interval; SS, MM:SS, or HH:MM:SS | | `--timestamps T1,T2,...` | Pin cue frames; reserves their budget before detail selection | @@ -131,10 +157,11 @@ For follow-ups, reuse evidence already viewed before rerunning. Remove only the ## Security and runtime access - yt-dlp contacts the source service/CDNs for metadata, one selected caption track, and media; access may require explicitly configured authentication. A cookie file is a read/write jar. +- With the Gemini engine, YouTube URLs are sent to Google and local or downloaded videos are uploaded to Google's Files API (generativelanguage.googleapis.com), then deleted after the answer; an upload that cannot be deleted expires within 48 hours. The key is sent only as a request header. The local engine never contacts Google. - FFmpeg/ffprobe run locally for probing, frames, and mono audio extraction. - With `whisperx` selected, audio never leaves the machine. First setup downloads packages and models from PyPI, Hugging Face, and GitHub, with uv/Python installers as needed. Pyannote telemetry is disabled. Warm caches allow offline inference; model libraries may still attempt cache/update network checks. - With `groq` or `openai` selected, only extracted audio is uploaded to that provider's transcription endpoint; keys are never shared between providers or logged by watch. - Runtime artifacts live in this run's working directory. User settings/keys live in `~/.config/watch/.env`; cwd `.env` is a cloud-key fallback. POSIX writes use mode 0600; Windows ACLs are not audited. Use a Linux-home config in WSL, since Windows-mounted homes have different permission semantics. - The managed environment lives at `~/.cache/watch/whisperx-venv`, outside the plugin. Model caches normally live at `~/.cache/huggingface` and `~/.cache/torch/hub`. uv also caches packages and managed Python. Reinstalling the skill does not remove these. -Bundled scripts: `watch.py`, `download.py`, `frames.py`, `transcribe.py`, `whisper.py`, `local_whisperx.py`, `config.py`, `runtime.py`, and `setup.py` under `scripts/`. The base runtime uses only Python's standard library; optional WhisperX dependencies remain in its separate process/environment. +Bundled scripts: `watch.py`, `download.py`, `frames.py`, `transcribe.py`, `whisper.py`, `local_whisperx.py`, `gemini.py`, `config.py`, `runtime.py`, and `setup.py` under `scripts/`. The base runtime uses only Python's standard library; optional WhisperX dependencies remain in its separate process/environment. diff --git a/tests/test_packaging.py b/tests/test_packaging.py index ace5ab8..2746012 100644 --- a/tests/test_packaging.py +++ b/tests/test_packaging.py @@ -26,3 +26,24 @@ def test_copied_skill_runs_outside_repository(tmp_path, static_clip): capture_output=True, encoding='utf-8', env=dict(os.environ)) assert result.returncode == 0, result.stderr assert '**Detail:** balanced' in result.stdout and 'reason=uniform' in result.stdout + + +def test_skill_documents_and_bundles_the_gemini_engine(): + skill = (ROOT / 'skills/watch/SKILL.md').read_text(encoding='utf-8') + assert (ROOT / 'skills/watch/scripts/gemini.py').is_file() + for needle in ('`gemini.py`', '--engine', '--question', 'GEMINI_API_KEY', '--engine local', + 'uploaded to Google', 'Answer (from Gemini)'): + assert needle in skill, needle + assert 'CLAUDE_SKILL_DIR' not in skill + runtime = sorted(p.name for p in (ROOT / 'skills/watch/scripts').glob('*.py')) + assert runtime == ['config.py', 'download.py', 'frames.py', 'gemini.py', 'local_whisperx.py', + 'runtime.py', 'setup.py', 'transcribe.py', 'watch.py', 'whisper.py'] + + +def test_copied_skill_runs_gemini_engine_check_outside_repository(tmp_path): + target = tmp_path / 'other host' / 'watch' + shutil.copytree(ROOT / 'skills/watch', target, ignore=shutil.ignore_patterns('__pycache__', '*.pyc')) + env = {k: v for k, v in os.environ.items() if k != 'GEMINI_API_KEY'} + result = subprocess.run([sys.executable, str(target / 'scripts/watch.py'), 'https://youtu.be/abc', '--engine', 'gemini'], + capture_output=True, encoding='utf-8', env=env) + assert result.returncode != 0 and 'GEMINI_API_KEY' in result.stderr