---
name: captions-dodge-hands
description: Make a beat-synced vertical reel where the captions are physical objects that react to the speaker's real hands - a dull default caption bar shatters into word chips on the beat drop, the chips float around the speaker's head, flee their tracked hands, ricochet off things in the room, form a halo for one beat, get grabbed, held and tossed - and every dodge plays a note in the song's key, so the hands play a melody. Hand and face tracking, precomputed physics, a hand-tracking overlay, bottom captions, a comment-to-get ending. Use when asked for "captions that dodge my hands", "words that react to me", "interactive captions", or when given the captions-dodge-hands prompt.
---

# Captions that dodge your hands

The idea: open on the plain auto-caption bar everyone scrolls past, then break it. On the beat drop the bar shatters
into word chips that live around the speaker's head. The chips flee the speaker's tracked hands, bounce off the
room, and each dodge plays a soft note in the song's key, so the gestures turn into a melody. The spoken line still
appears at the bottom in a normal caption: the toys never carry the reading.

Work through the steps in order. Show the user the storyboard (step 2) and wait for their OK before you generate
anything.

## 0. Tools

**You need a code environment** (for example Claude Code, or Claude with code execution) with Node 20+, ffmpeg and
Python 3.10+ with numpy, scipy, soundfile, mediapipe, torch and torchaudio. The chips, the overlay and the final
render are a [Remotion](https://www.remotion.dev) project that you write (React, rendered to MP4). Renders you make
with this skill are covered by HYVIRAL's Remotion license, so you don't need a Remotion license of your own.

Every library named here allows this use: MediaPipe (Apache-2.0) for hands, face and the matte, torchaudio's wav2vec2
base model for word timing, and torchaudio's Hybrid Demucs for music stems.

**For the takes, the music and the sound effects, use whatever the user prefers.** If the user hasn't said, ask which
tools they want, or use what is already set up. Two common routes:

- **Your own providers.** The takes can be the user's own phone videos on a tripod (the best case: real hands track
  best), or an image-to-video model that also makes speech (for example Veo, Kling or Runway), started from a photo
  of the presenter. Music can come from a music model (for example Lyria) or a licensed library track. Effects can
  come from a sound library or a generator (for example ElevenLabs Sound Effects). The chips' notes are synthesized
  in code. No AI media at all is fine too.
- **HYVIRAL (optional).** If the user has it connected (hyviral.com), one connection covers the media steps:

| You need | HYVIRAL tool |
|----------|--------------|
| AI presenter takes (image to video, with speech) | `generate_video` |
| A music track | `generate_music` |
| Sound placement, review, final render | `video_audio`, `video_review`, `video_export` |
| Long jobs | `get_job_status` (poll until delivered) |

With HYVIRAL connected, start with `get_connection_info`. To connect it: in Claude.ai or Claude Desktop, Customize →
Connectors → add a custom connector with the URL `https://api.hyviral.com/mcp`; in Claude Code,
`claude mcp add --transport http hyviral https://api.hyviral.com/mcp`, then `/mcp` to sign in.

You can mix routes. Nothing below depends on any one provider.

## 1. The line and the gestures

Write the words first, then give every line a hand move that proves it. A shape that works, at about 100 BPM
(a bar is 2.4 seconds, about 23 seconds in all):

| Moment | Hand move | Line (example) |
|--------|-----------|----------------|
| Opening, before the drop | none; a dull caption bar at the bottom; one word in it twitches when it is said | "Ever notice how captions just sit there?" |
| The drop | a clap, then both hands burst apart: the bar shatters into chips on this exact moment | "Mine don't." |
| Bar 3 | a slow sweep of an open hand across the frame: the chips flee it and spring back | "When my hand comes through, they get out of the way." |
| Bar 4 | one finger flicks a chip: it ricochets off something in the room on the next downbeat | "Flick one, and it bounces right back." |
| Bar 5 | both hands conduct; on beat 4 the line rises into a halo above the head for one beat | "Every word, every single time." |
| Bar 6 | a hand closes on one word: the music stops dead for a beat | "Except this one." |
| Bar 7 | the fist toward the lens, then a toss into something in the room, where the word tumbles | "This one's mine." / "Now Claude can make your captions do this." |
| Ending | a slow wave; the chips orbit the head and dodge the waving hand | "Comment [KEYWORD] and I'll send you the skill." |

Pick a place with something to bounce off and something to throw into (a laundromat's washer doors are perfect; a
window, a fridge or a fishbowl work too). Keep every gap between spoken lines at 1 second or less.

## 2. Storyboard (before generating anything)

Build one page: per moment, a concept frame with the chips drawn around the real hand positions, the line, the
gesture, every sound, and which part of the music plays. List the exact prompt for every generated asset. Show it to
the user and wait for their OK.

## 3. The takes

- **The camera never moves.** A phone on a tripod, chest height, the same framing, the person on one spot with room
  around the head and both hands in frame.
- **Hands that must be tracked move slowly and deliberately, palms toward the lens, fingers spread.** Fast moves blur
  and lose their landmarks. A closing fist needs its own sentence in the direction.
- **With an AI presenter:** start every take from the same approved photo, and keep one prompt template: the voice,
  "a vertical phone on a tripod, completely still", the line in quotes once, each gesture with when it happens, "she
  says only this line and nothing else", and an audio sentence (only her voice, no music, no clicks). Ask for open
  hands that never point or show single fingers, except a deliberate single index finger for the flick. Two traps:
  - A keyword in capitals gets spelled out letter by letter. Write it in lowercase and add "said once as one ordinary
    word, never spelled out".
  - Takes often add words or drift at the very end. Cut at the last scripted word.
- **Word timings:** force-align the known line on each take (torchaudio `WAV2VEC2_ASR_BASE_960H` plus
  `forced_align`). That model is trained on English speech, so it fits English lines only. For another
  language, use a wav2vec2 alignment model for that language whose license allows your use, or your speech-to-text
  tool's word stamps, and check the first word of each line by ear. Not torchaudio's `MMS_FA`: its license is
  non-commercial. If a word cannot be heard in the final mix, leave it out of every caption too.

## 4. Tracking

- **Hands:** MediaPipe HandLandmarker in VIDEO mode, two hands, 21 landmarks each, every frame. Label the hands by
  screen side (the one on the picture's left is "L"), so a hand keeps its name between frames.
- **Face:** a face box per frame (MediaPipe face detection). The chips' homes are anchored to it, so the swarm
  follows the person between shots.
- **The key moments come from the landmarks, not by eye:** the clap where the two hand centers meet, the burst where
  they fly apart, the sweep where a palm's x moves fast, the flick at the index tip's speed peak, the grab where the
  fingertips close on the palm, the toss at the wrist's speed peak.
- **Place the music by the gestures:** slide the track so its downbeat lands on the burst, and put the other moves on
  beats by choosing where to cut between takes.

## 5. Physics (precomputed in Python)

Remotion renders frames out of order, so springs and collisions can't run there. Simulate every chip in Python, four
substeps per frame, and write one JSON row per frame: id, x, y, rotation, scale, opacity, mode.

- **Home:** each chip has a home around the head, in face widths from the face center, never on the face. A spring
  pulls it home (stiffness about 60, damping about 10.5, in px/s²).
- **Hands push:** for each hand, the nearest of its 21 landmarks repels the chip within a radius of about a quarter of
  the frame width, with a force falling off as (1 - d/R)².
- **Never on the face:** an ellipse around the face box pushes chips out.
- **Chips keep apart:** a soft push between chips, measured on a squashed ellipse because chips are wide.
- **Bounds:** keep chips out of the top 345 px (the app's buttons) and the caption band at the bottom, with a soft
  bounce. Near the end, lower the bound so no chip covers the ending card.
- **Colliders:** a circle for whatever they ricochet off (a washer door glass), with a bounce.
- **Scripted moves on top:** the shatter (chips launch upward from their word's place in the bar); the flick (one chip
  flies on a curve over the head and hits the collider exactly on the downbeat, then returns to the physics); the
  halo (the line's words hold an arc above the head for one beat, then burst out and take the homes of the oldest
  chips, which pop); the grab (one word flies to the raised hand and locks to the fist); the toss (an arc into the
  target, shrinking); the tumble (the word circles inside the target, once per beat).
- **Events:** log every hand dodge (a chip entering a hand's field), every bounce and every first landing, with the
  chip's height. They become the notes.

## 6. The picture (Remotion)

Stack: the raw takes cut by the plan → chips → the hand overlay → ripples → captions → ending → film grain.

- **The dull bar:** white Arial on a black box at the bottom, no animation, the way auto-captions look. When its most
  boring word is said ("sit"), that word hops out of the line for a moment and snaps back. On the burst, crack lines
  draw across it for two frames, a short flash, and the chips fly out.
- **Chips:** solid rounded pills in two or three bold colors, white heavy sans-serif type at about 56 px on a
  1080 px frame, a soft shadow, and a faint glow on a chip that is being thrown. A thrown chip leaves two or three
  faint, blurred ghosts.
- **A grabbed word sits on top of the hand,** never behind the fingers: it reads as held.
- **The tumble** sits behind a glass highlight (a circle with a diagonal sheen) and stays in the one object it was
  thrown into. Never show it in a different object in a later shot.
- **Hand-tracking overlay:** only while the hands prove the point (the sweep). Gradient bones with a soft glow, white
  dots at the fingertips, four corner brackets around each hand, a dashed ring for the push radius, and a small pill:
  "HANDS TRACKED · 30 FPS". Size the pill to its text. Make it look designed, never like a debug view.
- **Ripples:** two or three rings expanding from every ricochet, with a small "TINK" on the main one.
- **The stop:** for the one beat after the grab, desaturate slightly and darken the edges.
- **Captions:** every spoken line in a dark pill at the bottom, the spoken word highlighted, the bottom edge no lower
  than y 1480 of 1920.
- **Ending:** "COMMENT", the keyword in a bold chip, and "and I'll send you the skill", landing on the spoken words,
  the chips orbiting above.
- **Camera:** a slow push during the dull opening, a big punch on the drop, a small punch on each cut, a slow push-in
  while the word is held, short shakes on the drop and the toss.

## 7. Music and sound

- **Music:** a playful funk or similar at about 100 BPM. Measure the tempo, then **check the bar line**: beat trackers
  often get the downbeat wrong by one or two beats. The beat phase where the chords change (spectral flux of the
  non-drum stem) is the bar line. Measure the key from the chroma of the bass and chord stems.
- **Arrange it from stems** (Hybrid Demucs): before the drop, the track as if from a small radio across the room
  (band-limited, mono, quiet). The full band on the burst. The bass steps out for two beats under the sweep so the
  chips' riff is heard. A dead stop for one beat on the grab, with only the room tone running, then the drums alone
  back in with a fill. The full groove again into the ending.
- **The chips are an instrument:** turn each event into one soft marimba note of the song's pentatonic scale (two
  octaves), pitched by the chip's height (higher on screen, higher note), quantized to the nearest 16th note of the
  track, at most two notes per 16th. Make the shatter's landings an ascending run. Synthesize the marimba (a sine,
  a quiet 4x partial that dies fast, a click of attack) and keep it below 3 kHz.
- **One quiet sound per move:** the clap, a soft glass shatter on the drop with a low thump, a whoosh on the sweep, a
  glass "tink" on the ricochet, a click as the halo locks, a cloth snatch and a short record-stop on the grab, a toss
  whoosh, a door thunk on the beat and muffled clunks per beat inside. A room-tone bed under everything keeps the cuts
  seamless.
- **Voice first:** music about 16 LU under the voice while they talk, the loudest effect about 6.5 to 10 LU under,
  effects warm and low-passed. Master at -14 LUFS, true peak at -1.5 dBTP or lower.

## 8. QA gates (all must pass)

- A transcript of the final mix matches the script, and every word shown in a caption is actually heard.
- A contact sheet at 2 fps with the chips drawn on: no chip on the face, no chip passing through a hand, none over the
  ending card, no label spilling out of its pill.
- Render twice and compare the two files frame by frame (PSNR). A single outlier frame means a flicker.
- Loudness and true peak are on target. No gap between spoken lines is longer than 1 second.
- A second opinion is useful: have another model review the finished video, and treat its notes as a list to check,
  not as orders.

## Appendix: the chip step (Python)

```python
import numpy as np

def step(chip, home, hands, face, fw, fh, dt, R=270.0):
    """One substep for one free chip. chip: {'p': np.array([x, y]), 'v': np.array([vx, vy])};
    hands: list of (21, 2) landmark arrays in frame pixels; face: center; fw, fh: face box size."""
    acc = (home - chip['p']) * 62 - chip['v'] * 10.5                   # spring home, damped
    for h in hands:                                                    # every hand pushes
        d = np.linalg.norm(h - chip['p'], axis=1); j = int(np.argmin(d))
        if d[j] < R:
            acc += (chip['p'] - h[j]) / max(d[j], 1e-3) * 11000 * (1 - d[j] / R) ** 2
    e = (chip['p'] - face) / np.array([fw * 0.95, fh * 1.05]); r = float(np.linalg.norm(e))
    if r < 1.0:                                                        # never on the face
        acc += e / max(r, 1e-3) * 9000 * (1 - r)
    chip['v'] = chip['v'] + acc * dt
    chip['p'] = chip['p'] + chip['v'] * dt
    return chip
```