Captions That Dodge Your Hands

---
name: captions-dodge-hands
description: Make a beat-synced vertical reel where the captions are physical objects that react to the speaker's real hands - a dull default caption bar shatters into word chips on the beat drop, the chips float around the speaker's head, flee their tracked hands, ricochet off things in the room, form a halo for one beat, get grabbed, held and tossed - and every dodge plays a note in the song's key, so the hands play a melody. Hand and face tracking, precomputed physics, a hand-tracking overlay, bottom captions, a comment-to-get ending. Use when asked for "captions that dodge my hands", "words that react to me", "interactive captions", or when given the captions-dodge-hands prompt.
---

# Captions that dodge your hands

The idea: open on the plain auto-caption bar everyone scrolls past, then break it. On the beat drop the bar shatters into word chips that live around the speaker's head. The chips flee the speaker's tracked hands, bounce off the room, and each dodge plays a soft note in the song's key, so the gestures turn into a melody. The spoken line still appears at the bottom in a normal caption: the toys never carry the reading.

Work through the steps in order. Show the user the storyboard (step 2) and wait for their OK before you generate anything.

## 0. Tools

**You need a code environment** (for example Claude Code, or Claude with code execution) with Node 20+, ffmpeg and Python 3.10+ with numpy, scipy, soundfile, mediapipe, torch and torchaudio. The chips, the overlay and the final render are a [Remotion](https://www.remotion.dev) project that you write (React, rendered to MP4). Renders you make with this skill are covered by HYVIRAL's Remotion license, so you don't need a Remotion license of your own.

Every library named here allows this use: MediaPipe (Apache-2.0) for hands, face and the matte, torchaudio's wav2vec2 base model for word timing, and torchaudio's Hybrid Demucs for music stems.

**For the takes, the music and the sound effects, use whatever the user prefers.** If the user hasn't said, ask which tools they want, or use what is already set up. Two common routes:

- **Your own providers.** The takes can be the user's own phone videos on a tripod (the best case: real hands track best), or an image-to-video model that also makes speech (for example Veo, Kling or Runway), started from a photo of the presenter. Music can come from a music model (for example Lyria) or a licensed library track. Effects can come from a sound library or a generator (for example ElevenLabs Sound Effects). The chips' notes are synthesized in code. No AI media at all is fine too.
- **HYVIRAL (optional).** If the user has it connected (hyviral.com), one connection covers the media steps:

| You need | HYVIRAL tool |
|----------|--------------|
| AI presenter takes (image to video, with speech) | `generate_video` |
| A music track | `generate_music` |
| Sound placement, review, final render | `video_audio`, `video_review`, `video_export` |
| Long jobs | `get_job_status` (poll until delivered) |

With HYVIRAL connected, start with `get_connection_info`. To connect it: in Claude.ai or Claude Desktop, Customize → Connectors → add a custom connector with the URL `https://api.hyviral.com/mcp`; in Claude Code, `claude mcp add --transport http hyviral https://api.hyviral.com/mcp`, then `/mcp` to sign in.

You can mix routes. Nothing below depends on any one provider.

## 1. The line and the gestures

Write the words first, then give every line a hand move that proves it. A shape that works, at about 100 BPM (a bar is 2.4 seconds, about 23 seconds in all):

| Moment | Hand move | Line (example) |
|--------|-----------|----------------|
| Opening, before the drop | none; a dull caption bar at the bottom; one word in it twitches when it is said | "Ever notice how captions just sit there?" |
| The drop | a clap, then both hands burst apart: the bar shatters into chips on this exact moment | "Mine don't." |
| Bar 3 | a slow sweep of an open hand across the frame: the chips flee it and spring back | "When my hand comes through, they get out of the way." |
| Bar 4 | one finger flicks a chip: it ricochets off something in the room on the next downbeat | "Flick one, and it bounces right back." |
| Bar 5 | both hands conduct; on beat 4 the line rises into a halo above the head for one beat | "Every word, every single time." |
| Bar 6 | a hand closes on one word: the music stops dead for a beat | "Except this one." |
| Bar 7 | the fist toward the lens, then a toss into something in the room, where the word tumbles | "This one's mine." / "Now Claude can make your captions do this." |
| Ending | a slow wave; the chips orbit the head and dodge the waving hand | "Comment [KEYWORD] and I'll send you the skill." |

Pick a place with something to bounce off and something to throw into (a laundromat's washer doors are perfect; a window, a fridge or a fishbowl work too). Keep every gap between spoken lines at 1 second or less.

## 2. Storyboard (before generating anything)

Build one page: per moment, a concept frame with the chips drawn around the real hand positions, the line, the gesture, every sound, and which part of the music plays. List the exact prompt for every generated asset. Show it to the user and wait for their OK.

## 3. The takes

- **The camera never moves.** A phone on a tripod, chest height, the same framing, the person on one spot with room around the head and both hands in frame.
- **Hands that must be tracked move slowly and deliberately, palms toward the lens, fingers spread.** Fast moves blur and lose their landmarks. A closing fist needs its own sentence in the direction.
- **With an AI presenter:** start every take from the same approved photo, and keep one prompt template: the voice, "a vertical phone on a tripod, completely still", the line in quotes once, each gesture with when it happens, "she says only this line and nothing else", and an audio sentence (only her voice, no music, no clicks). Ask for open hands that never point or show single fingers, except a deliberate single index finger for the flick. Two traps:
  - A keyword in capitals gets spelled out letter by letter. Write it in lowercase and add "said once as one ordinary word, never spelled out".
  - Takes often add words or drift at the very end. Cut at the last scripted word.
- **Word timings:** force-align the known line on each take (torchaudio `WAV2VEC2_ASR_BASE_960H` plus `forced_align`). That model is trained on English speech, so it fits English lines only. For another language, use a wav2vec2 alignment model for that language whose license allows your use, or your speech-to-text tool's word stamps, and check the first word of each line by ear. Not torchaudio's `MMS_FA`: its license is non-commercial. If a word cannot be heard in the final mix, leave it out of every caption too.

## 4. Tracking

- **Hands:** MediaPipe HandLandmarker in VIDEO mode, two hands, 21 landmarks each, every frame. Label the hands by screen side (the one on the picture's left is "L"), so a hand keeps its name between frames.
- **Face:** a face box per frame (MediaPipe face detection). The chips' homes are anchored to it, so the swarm follows the person between shots.
- **The key moments come from the landmarks, not by eye:** the clap where the two hand centers meet, the burst where they fly apart, the sweep where a palm's x moves fast, the flick at the index tip's speed peak, the grab where the fingertips close on the palm, the toss at the wrist's speed peak.
- **Place the music by the gestures:** slide the track so its downbeat lands on the burst, and put the other moves on beats by choosing where to cut between takes.

## 5. Physics (precomputed in Python)

Remotion renders frames out of order, so springs and collisions can't run there. Simulate every chip in Python, four substeps per frame, and write one JSON row per frame: id, x, y, rotation, scale, opacity, mode.

- **Home:** each chip has a home around the head, in face widths from the face center, never on the face. A spring pulls it home (stiffness about 60, damping about 10.5, in px/s²).
- **Hands push:** for each hand, the nearest of its 21 landmarks repels the chip within a radius of about a quarter of the frame width, with a force falling off as (1 - d/R)².
- **Never on the face:** an ellipse around the face box pushes chips out.
- **Chips keep apart:** a soft push between chips, measured on a squashed ellipse because chips are wide.
- **Bounds:** keep chips out of the top 345 px (the app's buttons) and the caption band at the bottom, with a soft bounce. Near the end, lower the bound so no chip covers the ending card.
- **Colliders:** a circle for whatever they ricochet off (a washer door glass), with a bounce.
- **Scripted moves on top:** the shatter (chips launch upward from their word's place in the bar); the flick (one chip flies on a curve over the head and hits the collider exactly on the downbeat, then returns to the physics); the halo (the line's words hold an arc above the head for one beat, then burst out and take the homes of the oldest chips, which pop); the grab (one word flies to the raised hand and locks to the fist); the toss (an arc into the target, shrinking); the tumble (the word circles inside the target, once per beat).
- **Events:** log every hand dodge (a chip entering a hand's field), every bounce and every first landing, with the chip's height. They become the notes.

## 6. The picture (Remotion)

Stack: the raw takes cut by the plan → chips → the hand overlay → ripples → captions → ending → film grain.

- **The dull bar:** white Arial on a black box at the bottom, no animation, the way auto-captions look. When its most boring word is said ("sit"), that word hops out of the line for a moment and snaps back. On the burst, crack lines draw across it for two frames, a short flash, and the chips fly out.
- **Chips:** solid rounded pills in two or three bold colors, white heavy sans-serif type at about 56 px on a 1080 px frame, a soft shadow, and a faint glow on a chip that is being thrown. A thrown chip leaves two or three faint, blurred ghosts.
- **A grabbed word sits on top of the hand,** never behind the fingers: it reads as held.
- **The tumble** sits behind a glass highlight (a circle with a diagonal sheen) and stays in the one object it was thrown into. Never show it in a different object in a later shot.
- **Hand-tracking overlay:** only while the hands prove the point (the sweep). Gradient bones with a soft glow, white dots at the fingertips, four corner brackets around each hand, a dashed ring for the push radius, and a small pill: "HANDS TRACKED · 30 FPS". Size the pill to its text. Make it look designed, never like a debug view.
- **Ripples:** two or three rings expanding from every ricochet, with a small "TINK" on the main one.
- **The stop:** for the one beat after the grab, desaturate slightly and darken the edges.
- **Captions:** every spoken line in a dark pill at the bottom, the spoken word highlighted, the bottom edge no lower than y 1480 of 1920.
- **Ending:** "COMMENT", the keyword in a bold chip, and "and I'll send you the skill", landing on the spoken words, the chips orbiting above.
- **Camera:** a slow push during the dull opening, a big punch on the drop, a small punch on each cut, a slow push-in while the word is held, short shakes on the drop and the toss.

## 7. Music and sound

- **Music:** a playful funk or similar at about 100 BPM. Measure the tempo, then **check the bar line**: beat trackers often get the downbeat wrong by one or two beats. The beat phase where the chords change (spectral flux of the non-drum stem) is the bar line. Measure the key from the chroma of the bass and chord stems.
- **Arrange it from stems** (Hybrid Demucs): before the drop, the track as if from a small radio across the room (band-limited, mono, quiet). The full band on the burst. The bass steps out for two beats under the sweep so the chips' riff is heard. A dead stop for one beat on the grab, with only the room tone running, then the drums alone back in with a fill. The full groove again into the ending.
- **The chips are an instrument:** turn each event into one soft marimba note of the song's pentatonic scale (two octaves), pitched by the chip's height (higher on screen, higher note), quantized to the nearest 16th note of the track, at most two notes per 16th. Make the shatter's landings an ascending run. Synthesize the marimba (a sine, a quiet 4x partial that dies fast, a click of attack) and keep it below 3 kHz.
- **One quiet sound per move:** the clap, a soft glass shatter on the drop with a low thump, a whoosh on the sweep, a glass "tink" on the ricochet, a click as the halo locks, a cloth snatch and a short record-stop on the grab, a toss whoosh, a door thunk on the beat and muffled clunks per beat inside. A room-tone bed under everything keeps the cuts seamless.
- **Voice first:** music about 16 LU under the voice while they talk, the loudest effect about 6.5 to 10 LU under, effects warm and low-passed. Master at -14 LUFS, true peak at -1.5 dBTP or lower.

## 8. QA gates (all must pass)

- A transcript of the final mix matches the script, and every word shown in a caption is actually heard.
- A contact sheet at 2 fps with the chips drawn on: no chip on the face, no chip passing through a hand, none over the ending card, no label spilling out of its pill.
- Render twice and compare the two files frame by frame (PSNR). A single outlier frame means a flicker.
- Loudness and true peak are on target. No gap between spoken lines is longer than 1 second.
- A second opinion is useful: have another model review the finished video, and treat its notes as a list to check, not as orders.

## Appendix: the chip step (Python)

```python
import numpy as np

def step(chip, home, hands, face, fw, fh, dt, R=270.0):
    """One substep for one free chip. chip: {'p': np.array([x, y]), 'v': np.array([vx, vy])};
    hands: list of (21, 2) landmark arrays in frame pixels; face: center; fw, fh: face box size."""
    acc = (home - chip['p']) * 62 - chip['v'] * 10.5                   # spring home, damped
    for h in hands:                                                    # every hand pushes
        d = np.linalg.norm(h - chip['p'], axis=1); j = int(np.argmin(d))
        if d[j] < R:
            acc += (chip['p'] - h[j]) / max(d[j], 1e-3) * 11000 * (1 - d[j] / R) ** 2
    e = (chip['p'] - face) / np.array([fw * 0.95, fh * 1.05]); r = float(np.linalg.norm(e))
    if r < 1.0:                                                        # never on the face
        acc += e / max(r, 1e-3) * 9000 * (1 - r)
    chip['v'] = chip['v'] + acc * dt
    chip['p'] = chip['p'] + chip['v'] * dt
    return chip
```

# The prompt

First give your agent the captions-dodge-hands skill (paste the skill text and ask it to save it as a skill). Then replace everything in [BRACKETS] below and paste it in.

**Tip:** use any AI tools you like for the takes, the music and the effects, or none at all (your own phone videos track best). If you'd rather not set up separate AI accounts and API keys, HYVIRAL (optional) is one way to get video and music generation behind one connection (hyviral.com; in Claude.ai → Customize → Connectors → add `https://api.hyviral.com/mcp`).

---

Use the captions-dodge-hands skill to make a vertical video for [PLATFORM: Instagram Reels / TikTok / YouTube Shorts].

- **The takes:** [FOOTAGE - e.g. "my own phone videos, attached, filmed on a tripod", or "an AI presenter made from this photo of me"]
- **Where it is filmed:** [PLACE - e.g. a laundromat with washer doors to bounce off and throw into, my kitchen with a fishbowl, my office window]
- **Who is talking:** [PRESENTER - e.g. me, a woman in her twenties, playful, relaxed American accent]
- **What I say:** [LINE - e.g. "use the skill's example lines", or my own script with one hand move per line]
- **The word I grab:** [GRAB WORD - e.g. ONE, from "Except this one"]
- **Length:** [LENGTH - e.g. about 23 seconds]
- **Language:** [LANGUAGE - e.g. English]
- **Music:** [MUSIC - e.g. playful funk at about 100 BPM with slap bass and clavinet, no vocals; or a track I own]
- **Look:** [STYLE - e.g. violet, coral and cyan chips with white heavy type, a gradient hand overlay]
- **AI tools:** [AI TOOLS - e.g. "whatever you already have set up", "HYVIRAL", or "Veo for the takes and Lyria for the music"]
- **Ending:** [CTA - e.g. "Comment DODGE and I'll send you the skill", or "Follow for more"]
- **Avoid:** [AVOID - e.g. no chips over my face, no brand logos]

Work in this order:

1. Write the line with one hand move per line on the music grid, and pick what the chips bounce off and what I throw the word into.
2. Build a storyboard page (per moment: a concept frame with the chips around my real hands, the line, the gesture, every sound, the music section, and the exact prompt for every generated asset). Show it to me and wait for my OK.
3. Make or check the takes (camera completely still, slow open-handed gestures), then track my hands and face on every frame and find the clap, the burst, the sweep, the flick, the grab and the toss from the landmarks.
4. Place the music so its downbeat lands on the burst, precompute the chip physics in Python, and turn every dodge, bounce and landing into a note in the song's key.
5. Build the dull bar, the shatter, the chips, the hand overlay, the ripples, the captions and the ending in Remotion, with the grabbed word on top of my hand, and mix voice first.
6. Run every QA gate in the skill (transcript, contact sheet, two identical renders, loudness, no gap over 1 second) and give me the MP4, a cover frame, and a ready-to-paste caption with hashtags.