# video-to-game

Adopt this skill: https://proskills.md/skills/development/lovisdotio-video-to-game-skills-video-to-game

Page: https://proskills.md/skills/development/lovisdotio-video-to-game-skills-video-to-game
This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required.

## File: skills/video-to-game/SKILL.md
size: 6199
sha: 275d1ce407303bb7ff927093bebcd74efecf0373
upstream: https://github.com/lovisdotio/video-to-game/blob/main/skills/video-to-game/SKILL.md

```text
---
name: video-to-game
description: >-
  Turns a gameplay video (generated by a video model such as MiniMax H3 Max or Seedance, or any clip) into a real,
  playable WebGL game in Three.js whose every image, 3D model, sound and music track is generated with fal, then
  proves it: side-by-side frames of the video and the game, and a scenario that plays the game with the keyboard
  and films its best seconds. Ships scripts to study the video (frames, palette, cuts), run and log fal models,
  record a playtest at the real frame rate, compare game and video, and find the liveliest window of a take.
  Use when the user wants a video, an AI gameplay clip or a game concept "turned into a game", "made playable",
  "recreated in the browser" or "rebuilt in Three.js/WebGL". Not for game engines (Unity, Unreal, Godot) or for
  copying existing commercial games.
license: MIT
metadata:
  hermes:
    tags:
      - threejs
      - webgl
      - game-development
      - video-to-game
      - fal
      - generative-assets
    category: development
---

# Video to Game

The video is the game design document, and you cannot watch it. A video model is good at what you are bad at:
the look of a game, its camera, its one memorable mechanic. Your job is to reverse-engineer that clip into a
game someone can play, with assets generated to match it, and to prove it with pictures, not adjectives.

## Workflow

1. **Generate or take the video.** One original concept, one mechanic you can see, a third-person or side camera,
   no HUD text. `prompts/gameplay-video-prompts.md` (repo root) has the recipe and 31 prompts that worked.
2. **Study it.** `python3 scripts/study_video.py gameplay.mp4 -o study/` saves a frame every 0.25 s, a contact sheet,
   the palette and the cuts. Open the sheet, then the frames. Write the spec before any code:
   camera (distance, height, lens), hero, the one core mechanic, world layout, palette, light, materials, mood,
   and the key moment the video sells. See `references/study-the-video.md`.
3. **Generate every asset with fal**, one job at a time, from the video's own frames as references:
   concept sheet, background removal, image to 3D, rigging, textures, parallax layers, sound effects, music.
   `python3 scripts/fal_asset.py <endpoint> --arg k=v --file k=path -o assets/` runs a model and logs it.
   Nothing drawn by hand, nothing downloaded elsewhere. See `references/fal-assets.md`.
4. **Build the game** from `templates/game.html`: Three.js only, one import map, a real loop (controls, goal,
   progression, feedback, end, restart), keyboard and gamepad, `?autostart` and `?seed=N`, `window.__game` state.
   See `references/game-loop.md`.
5. **Match the look.** Palette, fog, light, grade, bloom, vignette and grain are part of the spec from day one,
   not polish. Compare a game frame with a video frame at fixed spots after every change:
   `python3 scripts/side_by_side.py gameplay.mp4 shot.png -o pairs/ --times 2,5,8`.
   See `references/look-and-post.md`.
6. **Play it yourself.** Write `play.mjs` from `templates/play.mjs`: keyboard only, reading `window.__game`.
   `node scripts/record_play.mjs "http://127.0.0.1:8000/?autostart" play.mjs -o take.mp4` films it at its real
   frame rate; `python3 scripts/best_window.py take.mp4 --length 10` finds the liveliest stretch; look at those
   frames. Fix what a player would hit: falls, walls, dead seconds, unreadable moments. See `references/playtest.md`.
7. **Deliver:** `index.html` + `assets/` + `README.md` (concept, controls, every fal asset with its endpoint),
   the side-by-side pairs, the best take, and an honest list of what differs from the video.

Scripts need Python 3.9+, ffmpeg, Node 18+ with `playwright-core` for the recorder, and `pip install fal-client`
with `FAL_KEY` set for `fal_asset.py`. Run them from this skill's directory or with full paths.

## Hard rules

- **The game looks like its video.** Same camera, palette, light and materials. Every look decision is checked
  on a side-by-side pair, at the same three spots each time. "Looks right" without a pair image is not a check.
- **The video's mechanic is the game.** If the clip shows a girl turning the world upside down, the game is about
  turning the world upside down, not a generic runner with her model in it.
- **Every asset comes from fal**, generated from the video's frames as references. Log each call (file, endpoint).
  One job at a time: look at each result before asking for the next.
- **No text on screen** beyond a minimal title and one controls line. The world explains itself.
- **Original worlds only.** No characters, logos, names or level designs from existing games or films.
- **Post-processing is in the spec**: tone mapping, one grade, bloom, fog for depth, vignette, grain. Keep 60 fps
  on a laptop; 30 fps is acceptable only for a much better image.
- **Calm the world, let the game read.** One harmonized palette, no acid colors; dense but grouped; hero and
  interactive things stand out on a quiet background.
- **Play before you ship.** A recorded take with zero console errors, no fall, no dead second, the key moment
  of the video visible.

## Reference routing

| File | Read when |
|---|---|
| `references/study-the-video.md` | starting: frames, palette, camera, mechanic, the spec to write |
| `references/fal-assets.md` | generating assets: which model for what, the character pipeline, prompts that work, costs |
| `references/game-loop.md` | building: structure, input, camera, state contract, multiplayer, performance |
| `references/look-and-post.md` | matching the video's look: grade, fog, bloom, density, hierarchy, the side-by-side routine |
| `references/playtest.md` | playing your own game: scenarios, recording, best window, what to fix |

Templates: `templates/game.html` (Three.js skeleton with the post stack, input, state contract),
`templates/play.mjs` (a keyboard playtest scenario).

## Output expectations

- Say what was checked and how: which pairs, which take, fps, console errors.
- List the fal calls: count per endpoint and total cost.
- List honestly what differs from the video (a simplified mechanic, a missing effect), never "identical".
```

## File: skills/video-to-game/scripts/best_window.py
size: 2652
sha: 3cbad03d2266ab4c7921803c8c8229434d33ee1f
upstream: https://github.com/lovisdotio/video-to-game/blob/main/skills/video-to-game/scripts/best_window.py

```text
#!/usr/bin/env python3
"""Find the liveliest stretch of a recorded take, to show the game at its best.

Scores every frame by how much the picture changes (ffmpeg scene score), then ranks windows of --length
seconds. A high score means action on screen; a flat one means the player stood still or a menu was up.
Always look at the frames of the winning window before using it: motion is not the same as beauty.

Usage:
  python3 best_window.py take.mp4 --length 3
  python3 best_window.py take.mp4 --length 10 --top 3 --cut best.mp4

Needs ffmpeg on PATH. Standard library only.
"""
import argparse
import re
import subprocess

def scores(path):
    err = subprocess.run(['ffmpeg', '-v', 'info', '-i', path, '-vf', "select='gte(scene,0)',metadata=print",
                          '-an', '-f', 'null', '-'], capture_output=True, text=True).stderr
    t = [float(x) for x in re.findall(r'pts_time:([\d.]+)', err)]
    s = [float(x) for x in re.findall(r'lavfi\.scene_score=([\d.]+)', err)]
    return list(zip(t, s))

def main():
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument('take')
    ap.add_argument('--length', type=float, default=3.0)
    ap.add_argument('--top', type=int, default=3)
    ap.add_argument('--skip', type=float, default=1.0, help='ignore the first seconds (loading, title)')
    ap.add_argument('--cut', help='also cut the best window to this file')
    a = ap.parse_args()

    pts = [(t, s) for t, s in scores(a.take) if t >= a.skip]
    if not pts:
        raise SystemExit('no frames scored')
    end = pts[-1][0]
    ranked = []
    step = 0.25
    start = pts[0][0]
    while start + a.length <= end:
        win = [s for t, s in pts if start <= t < start + a.length]
        # motion everywhere beats one big spike: reward the median as much as the mean
        if win:
            med = sorted(win)[len(win) // 2]
            ranked.append((sum(win) / len(win) + med, round(start, 2)))
        start += step
    ranked.sort(reverse=True)
    picked = []
    for score, st in ranked:
        if all(abs(st - p) >= a.length for _, p in picked):
            picked.append((score, st))
        if len(picked) == a.top:
            break
    for score, st in picked:
        print(f'{st:6.2f} s -> {st + a.length:6.2f} s   score {score:.4f}')
    if a.cut and picked:
        st = picked[0][1]
        subprocess.run(['ffmpeg', '-v', 'error', '-y', '-ss', str(st), '-i', a.take, '-t', str(a.length),
                        '-c:v', 'libx264', '-crf', '18', '-an', a.cut], check=True)
        print(a.cut)

if __name__ == '__main__':
    main()
```

## File: skills/video-to-game/scripts/fal_asset.py
size: 4579
sha: 5c79a3e4bf15d6a51ee71a08f1d4334de8aee962
upstream: https://github.com/lovisdotio/video-to-game/blob/main/skills/video-to-game/scripts/fal_asset.py

```text
#!/usr/bin/env python3
"""Run one fal model, download what it returns, and log the call.

Every asset of the game goes through this script, so the log (fal_log.json next to the outputs) lists each
file with the endpoint that made it. Local files passed with --file are uploaded to the fal CDN first.

Usage:
  python3 fal_asset.py minimax/h3-max/text-to-video -o video/ --arg prompt="..." --arg duration=10 \\
      --arg prompt_expansion_mode=disabled --arg resolution=1080P
  python3 fal_asset.py fal-ai/nano-banana-pro/edit -o assets/ --name hero_concept \\
      --arg prompt="the hero alone, A-pose, plain light grey background" --file image_urls=study/f_012.jpg
  python3 fal_asset.py fal-ai/birefnet/v2 -o assets/ --name hero_cutout --file image_url=assets/hero_concept.png
  python3 fal_asset.py meshy/v7.1/image-to-3d -o assets/ --name hero --file image_url=assets/hero_cutout.png
  python3 fal_asset.py fal-ai/elevenlabs/sound-effects/v2 -o assets/ --name jump --arg text="soft cloth whoosh"

--arg key=value   JSON value if it parses (numbers, true, lists), else a string. Repeat for each field.
--file key=path   uploads the file; repeat the same key to build a list (image_urls=a.png --file image_urls=b.png).

Needs FAL_KEY in the environment and `pip install fal-client`. One job at a time: wait for it, look at it,
then decide the next one.
"""
import argparse
import json
import mimetypes
import os
import sys
import time
import urllib.request

LIST_KEYS = ('image_urls', 'animation_action_ids', 'reference_image_urls')

def parse_value(v):
    try:
        return json.loads(v)
    except ValueError:
        return v

def urls_in(obj):
    """Every downloadable file URL in a fal result, with the key path it came from."""
    out = []
    if isinstance(obj, dict):
        if isinstance(obj.get('url'), str) and obj['url'].startswith('http'):
            out.append(('', obj['url'], obj.get('file_name') or obj.get('content_type')))
        for k, v in obj.items():
            if k != 'url':
                out += [(f'{k}.{p}'.strip('.'), u, n) for p, u, n in urls_in(v)]
    elif isinstance(obj, list):
        for i, v in enumerate(obj):
            out += [(f'{i}.{p}'.strip('.'), u, n) for p, u, n in urls_in(v)]
    return out

def ext_for(url, hint):
    path = url.split('?')[0]
    ext = os.path.splitext(path)[1]
    if ext and len(ext) <= 5:
        return ext
    return mimetypes.guess_extension(hint or '') or '.bin'

def main():
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument('endpoint')
    ap.add_argument('-o', '--out', default='assets')
    ap.add_argument('--name', help='base name of the saved files (default: endpoint + time)')
    ap.add_argument('--arg', action='append', default=[], metavar='KEY=VALUE')
    ap.add_argument('--file', action='append', default=[], metavar='KEY=PATH')
    a = ap.parse_args()

    if not os.environ.get('FAL_KEY'):
        sys.exit('FAL_KEY is not set')
    import fal_client  # pip install fal-client

    args = {}
    for kv in a.arg:
        k, v = kv.split('=', 1)
        args[k] = parse_value(v)
    for kv in a.file:
        k, p = kv.split('=', 1)
        url = fal_client.upload_file(p)
        if k in args or k in LIST_KEYS:
            args.setdefault(k, [])
            args[k] = (args[k] if isinstance(args[k], list) else [args[k]]) + [url]
        else:
            args[k] = url

    os.makedirs(a.out, exist_ok=True)
    name = a.name or a.endpoint.replace('/', '_') + time.strftime('_%H%M%S')
    t0 = time.time()
    result = fal_client.subscribe(a.endpoint, arguments=args, with_logs=False)
    files = []
    found = urls_in(result)
    for i, (key, url, hint) in enumerate(found):
        dest = os.path.join(a.out, name + ('' if len(found) == 1 else f'_{i}') + ext_for(url, hint))
        urllib.request.urlretrieve(url, dest)
        files.append({'file': dest, 'key': key, 'url': url})

    log_path = os.path.join(a.out, 'fal_log.json')
    log = json.load(open(log_path)) if os.path.exists(log_path) else []
    log.append({'endpoint': a.endpoint, 'name': name, 'seconds': round(time.time() - t0, 1),
                'arguments': {k: v for k, v in args.items() if k != 'prompt'}, 'prompt': args.get('prompt'),
                'files': [f['file'] for f in files]})
    with open(log_path, 'w') as f:
        json.dump(log, f, indent=1)
    for f in files:
        print(f['file'])
    if not files:
        print(json.dumps(result, indent=1)[:2000])

if __name__ == '__main__':
    main()
```

## File: skills/video-to-game/scripts/record_play.mjs
size: 3881
sha: c5a573850e67282ce0137cf9a63be02716ca6baa
upstream: https://github.com/lovisdotio/video-to-game/blob/main/skills/video-to-game/scripts/record_play.mjs

```text
#!/usr/bin/env node
// Film the game while a scenario plays it with the keyboard, at its real frame rate.
//
//   node record_play.mjs http://127.0.0.1:8000/?autostart play.mjs -o take.mp4 [--seconds 18] [--size 1920x1080]
//
// play.mjs exports `async function play(page, { seconds, t0 })` and only presses keys (page.keyboard) and
// reads state the game exposes (window.__game). The take is encoded with each frame's real duration, so a
// slow moment stays slow instead of being stretched. Needs Node 18+, ffmpeg, and `npm i playwright-core`
// (or playwright). Uses the system Chrome when found, for real GPU WebGL.
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { pathToFileURL } from 'node:url';
import { execFileSync } from 'node:child_process';

const argv = process.argv.slice(2);
const opt = (k, d) => { const i = argv.indexOf(k); return i >= 0 ? argv.splice(i, 2)[1] : d; };
const out = opt('-o', 'take.mp4'), seconds = +opt('--seconds', 18), [W, H] = opt('--size', '1920x1080').split('x').map(Number);
const [url, playFile] = argv;
if (!url || !playFile) { console.error('usage: node record_play.mjs URL play.mjs -o take.mp4 [--seconds 18]'); process.exit(1); }

let chromium;
try { ({ chromium } = await import('playwright-core')); } catch { ({ chromium } = await import('playwright')); }
const chrome = ['/Applications/Google Chrome.app/Contents/MacOS/Google Chrome', '/usr/bin/google-chrome', '/usr/bin/chromium']
  .find(p => fs.existsSync(p));
const browser = await chromium.launch({
  executablePath: chrome, headless: true,
  args: ['--enable-gpu', '--ignore-gpu-blocklist', '--use-angle=' + (process.platform === 'darwin' ? 'metal' : 'default'),
    '--autoplay-policy=no-user-gesture-required', '--mute-audio', '--disable-background-timer-throttling'],
});
const ctx = await browser.newContext({ viewport: { width: W, height: H } });
const page = await ctx.newPage();
const errors = [];
page.on('pageerror', e => errors.push(String(e)));
page.on('console', m => m.type() === 'error' && errors.push(m.text()));
await page.goto(url, { waitUntil: 'load', timeout: 90000 });
await page.waitForTimeout(2500);
await page.mouse.click(W / 2, H / 2).catch(() => {});

const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'take-'));
const cdp = await ctx.newCDPSession(page);
const times = []; let n = 0;
cdp.on('Page.screencastFrame', f => {
  fs.writeFileSync(path.join(dir, String(n++).padStart(5, '0') + '.jpg'), Buffer.from(f.data, 'base64'));
  times.push(f.metadata.timestamp);
  cdp.send('Page.screencastFrameAck', { sessionId: f.sessionId }).catch(() => {});
});
await cdp.send('Page.startScreencast', { format: 'jpeg', quality: 90, maxWidth: W, maxHeight: H, everyNthFrame: 1 });
const { play } = await import(pathToFileURL(path.resolve(playFile)).href);
const t0 = Date.now();
await Promise.race([play(page, { seconds, t0 }).catch(e => errors.push('play: ' + e)), page.waitForTimeout((seconds + 15) * 1000)]);
await cdp.send('Page.stopScreencast');
await browser.close();

if (n < 2) { console.error('no frames captured', errors.slice(0, 3)); process.exit(1); }
const list = times.map((t, i) => `file '${String(i).padStart(5, '0')}.jpg'\nduration ${((times[i + 1] ?? t + 1 / 60) - t).toFixed(5)}`);
fs.writeFileSync(path.join(dir, 'list.txt'), list.join('\n') + `\nfile '${String(n - 1).padStart(5, '0')}.jpg'\n`);
execFileSync('ffmpeg', ['-v', 'error', '-y', '-f', 'concat', '-safe', '0', '-i', 'list.txt',
  '-vf', `scale=${W}:${H}:flags=lanczos,fps=60,format=yuv420p`, '-c:v', 'libx264', '-crf', '20', path.resolve(out)], { cwd: dir });
fs.rmSync(dir, { recursive: true, force: true });
const dur = times.at(-1) - times[0];
console.log(`${out}: ${n} frames, ${dur.toFixed(1)} s, ${(n / dur).toFixed(1)} fps, ${errors.length} console errors`);
for (const e of errors.slice(0, 5)) console.log('  ' + e.slice(0, 200));
```

## File: skills/video-to-game/scripts/side_by_side.py
size: 2331
sha: 386b99b26cfcd400a994a94252e319743d0e854e
upstream: https://github.com/lovisdotio/video-to-game/blob/main/skills/video-to-game/scripts/side_by_side.py

```text
#!/usr/bin/env python3
"""Put the source video and the game next to each other, so "it looks like the video" becomes a picture.

Writes one image per time: source on the left, game on the right, same height. Use it on a recorded take
(record_play.mjs) or on screenshots of the game at fixed spots.

Usage:
  python3 side_by_side.py gameplay.mp4 take.mp4 -o pairs/ --times 1,3,5,8
  python3 side_by_side.py gameplay.mp4 shot.png -o pairs/ --times 4        # a still of the game
  python3 side_by_side.py gameplay.mp4 take.mp4 -o pairs/ --every 1 --game-offset 2.5

Needs ffmpeg on PATH. Standard library only.
"""
import argparse
import os
import subprocess

def main():
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument('source')
    ap.add_argument('game', help='a recorded take (.mp4) or a still (.png/.jpg)')
    ap.add_argument('-o', '--out', default='pairs')
    ap.add_argument('--times', help='comma separated times in the source, in seconds')
    ap.add_argument('--every', type=float, help='or one pair every N seconds')
    ap.add_argument('--game-offset', type=float, default=0, help='game time = source time + offset')
    ap.add_argument('--height', type=int, default=540)
    a = ap.parse_args()

    dur = float(subprocess.run(['ffprobe', '-v', 'error', '-show_entries', 'format=duration', '-of', 'csv=p=0', a.source],
                               capture_output=True, text=True, check=True).stdout)
    if a.times:
        times = [float(t) for t in a.times.split(',')]
    else:
        step = a.every or 1.0
        times = [round(i * step, 2) for i in range(int(dur / step))]
    still = a.game.lower().endswith(('.png', '.jpg', '.jpeg', '.webp'))
    os.makedirs(a.out, exist_ok=True)
    for t in times:
        dest = os.path.join(a.out, f'pair_{t:06.2f}.jpg')
        game_in = ['-i', a.game] if still else ['-ss', str(max(0, t + a.game_offset)), '-i', a.game]
        subprocess.run(['ffmpeg', '-v', 'error', '-y', '-ss', str(t), '-i', a.source, *game_in, '-filter_complex',
                        f'[0:v]scale=-2:{a.height},setsar=1[s];[1:v]scale=-2:{a.height},setsar=1[g];[s][g]hstack=2',
                        '-frames:v', '1', '-q:v', '3', dest], check=True)
        print(dest)

if __name__ == '__main__':
    main()
```

## File: skills/video-to-game/scripts/study_video.py
size: 4120
sha: 1570ec70d3e29e2f188995e0fae2cb91028f5f48
upstream: https://github.com/lovisdotio/video-to-game/blob/main/skills/video-to-game/scripts/study_video.py

```text
#!/usr/bin/env python3
"""Turn a gameplay video into things an agent can look at.

Saves a frame every --every seconds, a contact sheet, the dominant palette (hex), the cut times, and a
study.json summary. Open the sheet first, then the frames, then write the game spec.

Usage:
  python3 study_video.py gameplay.mp4 -o study/
  python3 study_video.py gameplay.mp4 -o study/ --every 0.5 --colors 12

Needs ffmpeg and ffprobe on PATH. Standard library only.
"""
import argparse
import json
import os
import re
import subprocess
from collections import Counter

def probe(path):
    out = subprocess.run(['ffprobe', '-v', 'error', '-select_streams', 'v:0', '-show_entries',
                          'stream=width,height,r_frame_rate:format=duration', '-of', 'json', path],
                         capture_output=True, text=True, check=True).stdout
    d = json.loads(out)
    s = d['streams'][0]
    num, den = s['r_frame_rate'].split('/')
    return {'width': s['width'], 'height': s['height'], 'fps': round(int(num) / int(den), 3),
            'duration': round(float(d['format']['duration']), 3)}

def frames(path, out, every):
    os.makedirs(out, exist_ok=True)
    subprocess.run(['ffmpeg', '-v', 'error', '-y', '-i', path, '-vf', f'fps=1/{every},scale=960:-2',
                    '-q:v', '3', os.path.join(out, 'f_%03d.jpg')], check=True)
    names = sorted(f for f in os.listdir(out) if f.startswith('f_'))
    return [(round(i * every, 2), n) for i, n in enumerate(names)]

def sheet(path, out, n):
    cols = 6
    rows = max(1, -(-n // cols))
    every = probe(path)['duration'] / max(1, cols * rows)
    subprocess.run(['ffmpeg', '-v', 'error', '-y', '-i', path, '-vf',
                    f'fps=1/{every:.4f},scale=480:-2,tile={cols}x{rows}:padding=6:color=white',
                    '-frames:v', '1', os.path.join(out, 'sheet.jpg')], check=True)

def palette(path, k):
    """Dominant colors from 16 small frames, quantized to 4 bits per channel."""
    raw = subprocess.run(['ffmpeg', '-v', 'error', '-i', path, '-vf', 'fps=16/10,scale=64:36', '-frames:v', '16',
                          '-f', 'rawvideo', '-pix_fmt', 'rgb24', '-'], capture_output=True, check=True).stdout
    bins = Counter()
    sums = {}
    for i in range(0, len(raw) - 2, 3):
        r, g, b = raw[i], raw[i + 1], raw[i + 2]
        key = (r >> 4, g >> 4, b >> 4)
        bins[key] += 1
        s = sums.setdefault(key, [0, 0, 0])
        s[0] += r; s[1] += g; s[2] += b
    total = sum(bins.values())
    out = []
    for key, n in bins.most_common(k):
        s = sums[key]
        out.append({'hex': '#%02x%02x%02x' % (s[0] // n, s[1] // n, s[2] // n), 'share': round(n / total, 3)})
    return out

def cuts(path, threshold):
    err = subprocess.run(['ffmpeg', '-v', 'info', '-i', path, '-vf', f"select='gt(scene,{threshold})',showinfo",
                          '-f', 'null', '-'], capture_output=True, text=True).stderr
    return [round(float(t), 2) for t in re.findall(r'pts_time:([\d.]+)', err)]

def main():
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument('video')
    ap.add_argument('-o', '--out', default='study')
    ap.add_argument('--every', type=float, default=0.25, help='seconds between saved frames (default 0.25)')
    ap.add_argument('--colors', type=int, default=10)
    ap.add_argument('--cut-threshold', type=float, default=0.35)
    a = ap.parse_args()

    info = probe(a.video)
    fr = frames(a.video, a.out, a.every)
    sheet(a.video, a.out, 24)
    info.update(palette=palette(a.video, a.colors), cuts=cuts(a.video, a.cut_threshold),
                frames=[{'t': t, 'file': n} for t, n in fr], sheet='sheet.jpg')
    with open(os.path.join(a.out, 'study.json'), 'w') as f:
        json.dump(info, f, indent=1)
    print(f"{info['width']}x{info['height']} {info['fps']} fps {info['duration']} s, {len(fr)} frames, "
          f"{len(info['cuts'])} cuts, palette {' '.join(c['hex'] for c in info['palette'])}")
    print(f'open {os.path.join(a.out, "sheet.jpg")} first')

if __name__ == '__main__':
    main()
```

## File: skills/video-to-game/references/fal-assets.md
size: 4417
sha: 1219b92e74f68db1857e5523c9db5a0a6643b7ec
upstream: https://github.com/lovisdotio/video-to-game/blob/main/skills/video-to-game/references/fal-assets.md

```text
# Generate every asset with fal

Rule of the project: every image, 3D model, sound and music track in the game is generated on fal, from the
video's own frames as references, so the assets share the video's style. Nothing is drawn by hand or downloaded.

Run each call with `scripts/fal_asset.py`, which uploads local files, downloads the outputs and appends to
`fal_log.json`. One job at a time: open the result, judge it against the video, then ask for the next.
Check endpoint ids and parameters in the live catalog (fal.ai/models) before a first call; do not guess them.

## What the 31 games of the gallery used

1,013 calls in total, about 6 USD per game (median 6.13, from 0.79 to 15.48).

| Job | Endpoint (calls) |
|---|---|
| Concepts, sprites, textures, sky, parallax layers | `fal-ai/nano-banana-pro/edit` (388), `fal-ai/nano-banana-pro` (73) |
| Background removal | `fal-ai/birefnet/v2` (74), `fal-ai/bria/background/remove` (12) |
| Image to 3D (GLB) | `meshy/v7.1/image-to-3d` (81), `tripo3d/h3.1/image-to-3d` (40), Trellis 2 (23), `fal-ai/hunyuan-3d/v3.1/pro/image-to-3d` (17), Rodin (15) |
| Rigging and animations | `fal-ai/meshy/rigging/multi-animation` (24) |
| Sound effects | `fal-ai/elevenlabs/sound-effects/v2` (165), Stable Audio 3 SFX (41), Sonilo SFX (26) |
| Music | `elevenlabs/music/v2.5` (22), `google/lyria-3.5` (8) |
| The gameplay video itself | `minimax/h3-max/text-to-video` (30), `bytedance/seedance-2.5/text-to-video` (1) |

## The character pipeline

1. **Concept from the video.** `fal-ai/nano-banana-pro/edit` with 1 to 3 frames where the hero is clearest as
   `image_urls`, and a prompt like: "the same character alone, full body, A-pose, arms slightly away from the body,
   neutral expression, plain light grey background, even studio light, same art style and colors as the reference".
   A-pose (not T-pose) rigs better and keeps sleeves and capes readable.
2. **Optional turnaround.** Same model, "front, side and back views of the same character on one sheet", for
   multi-view 3D models and to check the design from behind (the camera mostly sees the back).
3. **Cut it out.** `fal-ai/birefnet/v2` on the concept. A clean alpha gives a clean mesh.
4. **To 3D.** `meshy/v7.1/image-to-3d` (`image_url`, options for A/T pose and PBR) or `tripo3d/h3.1/image-to-3d`
   for characters; Hunyuan 3D Pro (`input_image_url`, plus optional back and side views) for detailed props.
   Normalize every GLB on load: scale to a known height, put the feet at y = 0, face -Z.
5. **Rig and animate.** `fal-ai/meshy/rigging/multi-animation` with `model_url` and `animation_action_ids`
   (run, jump, idle, glide) and `height_meters`. Keep the clips in one GLB and blend them with an AnimationMixer.

## Environments and textures

- **Wide layers** (far mountains, cloud banks, city skyline): one wide image per depth, then background removal,
  on planes at different distances for parallax. Ask for "isolated on a flat white background" so the cutout is clean.
- **Tileable materials** (grass, stone, roof tiles, sand): "seamless tileable texture, top-down, flat even light,
  no shadows, no perspective". Check the tiling by repeating it 3x3 before using it.
- **Sprites and effects** (petals, sparks, leaves, puffs): an atlas of 4x4 on a flat color, then background removal.
- **Props** (houses, trees, lanterns): concept on grey, cutout, image to 3D. Reuse one model many times with
  scale, rotation and tint variations instead of generating hundreds.
- Keep the video frame in `image_urls` for everything, so a texture made in isolation still belongs to the world.

## Sound

- One sound per player action and per world reaction: jump, land, glide loop, collect, the key mechanic, ambient bed.
  `fal-ai/elevenlabs/sound-effects/v2` takes `text`, `duration_seconds`, `loop` (true for wind, engines, rain).
- Music: `elevenlabs/music/v2.5` with `force_instrumental: true`, or a `composition_plan` of sections with
  durations when the music must build at a precise moment. Normalize loudness so effects sit above the music.

## Cost habits

- Log every call (the script does). Report calls per endpoint and the total at the end.
- Generate 2 or 3 variations only for the hero and the key mechanic's assets; one is enough for background props.
- Shrink textures before shipping (WebP, 1K for props, 2K for terrain): a game with 8 characters and 80 props
  still loads in seconds.
```

## File: skills/video-to-game/references/game-loop.md
size: 2483
sha: 95d726b6041171be71c4249240f88d680e36413c
upstream: https://github.com/lovisdotio/video-to-game/blob/main/skills/video-to-game/references/game-loop.md

```text
# Build the game

Start from `templates/game.html`. One `index.html`, an `assets/` folder, Three.js from a pinned import map
(`three@0.160.0` on jsDelivr) and its examples (GLTFLoader, EffectComposer). No other library, no engine,
no build step: the file opens from any static server.

## A real loop, not a scene

- **Controls** in one line: arrows or WASD to move, Space for the main action (hold for the long version:
  glide, pour, charge), one more key at most. Gamepad too: left stick, A for the action, Start to begin.
- **A goal** the player understands without text: reach the lighthouse, wake every tree, rescue the villagers.
- **Progression**: the world changes as the player succeeds (color spreads, a new island, faster wind).
- **Feedback** on every action: particles, a camera nudge, a sound, the world reacting.
- **An end** and a restart (R or Start). A short calm ending shot beats a score screen.
- **Multiplayer** when the video shows it: local co-op on one keyboard (WASD + arrows) or AI teammates that
  play well enough to be shown in a trailer.

## The contract with the playtest

- `?autostart` skips the title, `?seed=N` fixes the level, and optional debug parameters can place the player
  (`?x=120`) or the camera (`?camd=24`) so a scenario reaches the key moment fast. The rendered game must stay the
  real one.
- `window.__game.state` returns read-only numbers a player would see on screen: phase, position, speed, grounded,
  the next obstacle or target. Scenarios read it to time their keys; they never write into the game.

## Camera

Reproduce the video's framing first, then make it playable: smooth follow with damping (`1 - exp(-k dt)`),
a little look-ahead in the direction of motion, a slight lag on height so jumps read, and no clipping into the
world. The hero stays big enough to read, near the center, as in the video.

## Performance

- 60 fps on a laptop is the target. Instancing for anything repeated (trees, rocks, grass tufts, birds),
  merged static geometry, shadows from one light with a tight frustum that follows the player.
- Texture sizes from the asset step (1K props, 2K terrain), mipmaps on, anisotropy 4 to 8.
- Measure: log `renderer.info.render.calls` and the frame time; drop the most expensive effect before cutting beauty.

## Text

The only text: a minimal title, one controls line, maybe an ending line. No tutorials, no HUD chrome, no fake
UI copied from the video. Counters, if any, are small icons in a corner.
```

## File: skills/video-to-game/references/look-and-post.md
size: 2563
sha: 4d39851da74366474051ff5e62b10a9ce5258bb7
upstream: https://github.com/lovisdotio/video-to-game/blob/main/skills/video-to-game/references/look-and-post.md

```text
# Match the look

The first versions of these games were playable and wrong: the right mechanic in a flat, colorful, generic
world that looked nothing like its video. What fixed them was treating the look as a spec with measurements,
and comparing pictures after every change.

## The side-by-side routine

1. Pick three spots that cover the game: the start, the key moment, and somewhere far from the start
   (expose a `?x=` or a `flyTo` for it).
2. Screenshot the game at those spots, and pick the video frames that show the same kind of moment.
3. `python3 scripts/side_by_side.py gameplay.mp4 shot.png -o pairs/ --times 4` and look at each pair.
4. Write what differs (too saturated, sky too clean, fog too thick, hero too small), fix one thing, repeat at the
   same three spots. Always the same spots, or you are comparing luck.

## Post-processing is part of the build

Set it up with the first scene, not at the end:

- **Tone mapping** ACES Filmic, exposure tuned on a pair.
- **One grade** for the whole game, like a LUT: lift, gain, a hue shift in the shadows. Change the grade, not
  forty materials.
- **Fog / aerial perspective**: far things melt into the haze color from the palette. It gives depth and hides
  the edge of the world.
- **Bloom** only on what glows in the video (sun, lanterns, magic). Threshold high.
- **Vignette and grain** at low strength; they glue generated textures together.
- **Depth of field / tilt-shift** only if the video has it, and keep the horizon readable.

## Calm the world

The usual failure after "make it richer" is noise: random props everywhere, saturated greens, detail at every
pixel. Then nothing reads.

- **One harmonized palette.** Olive and sage instead of acid green, slate blues, earthy ochres: whatever the video
  uses, pulled together by the grade.
- **Group, do not scatter.** Trees in groves, rocks in clusters, open meadows and plazas between them, gaps of sky.
- **Hierarchy.** Landmark buildings > houses > props > ground detail. Interactive things (towers, rings, targets,
  pickups) get the accent color and stand out on a calm background.
- **Painted foliage stays calm**: leaf cards with soft shading, not lumpy green balls.
- **Density is welcome** (more than the video), as long as the frame still reads at a glance.

## Checks before calling it done

- A pair at each of the three spots where the game could pass for a frame of the video at thumbnail size.
- The hero is readable at the size the video shows it.
- 60 fps (or 30 fps with a clearly better image), zero console errors.
```

## File: skills/video-to-game/references/playtest.md
size: 2314
sha: 7de85e95499f8461b0296748dafd9c6751f1f6e7
upstream: https://github.com/lovisdotio/video-to-game/blob/main/skills/video-to-game/references/playtest.md

````text
# Play your own game

A game nobody played is a scene. The agent plays it, films it, watches the film and fixes what a player would hit.
The same takes become the trailer.

## Write a scenario

`templates/play.mjs` exports `async function play(page, { seconds, t0 })`:

- **Keyboard and mouse only** (`page.keyboard.down/up/press`), like a player.
- **Read what a player sees** through `window.__game.state` (position, speed, grounded, next gap or target)
  to time jumps and turns. Never write into the game, never teleport the hero.
- **Play well**: keep moving forward, sprint and boost when the game has them, jump at the right moment, never
  fall, never run into a wall, never walk backwards, no menu in the middle.
- **Show every mechanic**, and bring the video's key moment to the center of the frame early in the take.
- Use the game's own URL parameters (`?autostart`, `?seed=7`, `?x=120`) to start close to the action. A built-in
  autopilot is fine only if it plays naturally; otherwise it fights your keys.

## Record and pick

```bash
python3 -m http.server 8000 &                                   # from the game folder
node scripts/record_play.mjs "http://127.0.0.1:8000/?autostart&seed=7" play.mjs -o take.mp4 --seconds 18
python3 scripts/best_window.py take.mp4 --length 10 --top 3     # liveliest windows
python3 scripts/best_window.py take.mp4 --length 3 --cut best.mp4
```

The recorder captures every frame with its real timestamp and encodes it at that pace, so the take shows the true
frame rate (it prints it, with the console errors). `best_window.py` ranks windows by how much the picture changes;
then extract a frame every 0.25 s of the winner and look at every one. Reject a window if a single frame shows a
title or tip text, a cut-off or tiny hero, a shaking camera, a fall, a step back or a glitch.

## What to fix after watching

- A dead second: the level is too empty there, or the scenario waits too long.
- The key moment happens off-center or too late: move the trigger, adjust the camera, start closer.
- A fall or a wall: the level design punishes the obvious path; fix the level before the scenario.
- A frame drop: find the effect or the draw calls behind it.

Up to five takes per game is normal. Keep the best one, and write in the README which take and window it is.
````

## File: skills/video-to-game/references/study-the-video.md
size: 2321
sha: 9660c9d1e113900d051a0f47a2e6bee895cd689b
upstream: https://github.com/lovisdotio/video-to-game/blob/main/skills/video-to-game/references/study-the-video.md

````text
# Study the video

You will build from frames, so make them first and look at all of them.

```bash
python3 scripts/study_video.py gameplay.mp4 -o study/            # frames every 0.25 s, sheet.jpg, palette, cuts
python3 scripts/study_video.py gameplay.mp4 -o study/ --every 1  # a lighter pass for a 15 s clip
```

Open `study/sheet.jpg`, then every frame around the moments that matter. Read the prompt that generated the
video if there is one: it names the mechanic and the world in words the frames cannot give you.

## Write the spec before the code

One short file, `SPEC.md`, answering each line with what you see in a named frame:

- **Camera.** Third person behind, side view, top-down, chase? Distance to the hero (in hero heights), height,
  pitch, lens (wide and distorted or long and flat), how it follows (lag, look-ahead, roll in turns).
- **Hero.** Silhouette, proportions, colors, one or two features that make it readable at 100 px tall
  (a red scarf, a lantern, a bubble).
- **The mechanic.** The one verb the video sells: glide off a whale's tail, pour paint that makes flowers bloom,
  flip gravity, lay rails in front of a train. Write how it starts, what it changes in the world, and what
  feedback the video shows (particles, color spreading, a sound you can imagine).
- **Key moment.** The two seconds you would put in a trailer. The playtest must reach it.
- **World.** Layout (a corridor, an open field, floating islands at several heights), what repeats, what is unique,
  scale cues (houses, trees, clouds).
- **Look.** The palette from `study.json` with names (sky, haze, ground, hero accent), the light direction and
  color, the materials (painted, clay, low-poly flat, glossy), the atmosphere (fog, haze, particles), the grade
  (warm lift, teal shadows, crushed blacks).
- **Players.** One, or local co-op if the video shows two characters working together.

## Traps

- A generated video cuts and drifts. Take the camera and look from its longest steady shot, not from a cut.
- Video models invent HUD text and fake UI. Ignore it: the game shows no text.
- A video often shows a moment the game cannot sustain (a 2-second stunt). Make it a reward the player earns,
  not the whole loop.
- Measure, do not remember. Pick colors from the palette or a frame, not from the word "golden".
````

## File: skills/video-to-game/templates/game.html
size: 6982
sha: 36fcf1dc87db1f31a342e2626f4653a7366ea6d5
upstream: https://github.com/lovisdotio/video-to-game/blob/main/skills/video-to-game/templates/game.html

```text
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Game</title>
<link rel="icon" href="data:,">
<style>
  html, body { margin: 0; height: 100%; overflow: hidden; background: #000; }
  canvas { display: block; width: 100vw; height: 100vh; }
</style>
<script type="importmap">
{ "imports": {
  "three": "https://cdn.jsdelivr.net/npm/three@0.160.0/build/three.module.js",
  "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.160.0/examples/jsm/"
} }
</script>
</head>
<body>
<script type="module">
// Skeleton for a game rebuilt from a gameplay video. Keep its contract:
//  - ?autostart skips the title, ?seed=N fixes the level, so a scenario can play it the same way twice;
//  - window.__game exposes read-only state (player position, speed, phase) for the playtest scenarios;
//  - the look lives in one place: PALETTE, fog, lights and the post stack, all matched to the video's frames.
import * as THREE from 'three';
import { GLTFLoader } from 'three/addons/loaders/GLTFLoader.js';
import { EffectComposer } from 'three/addons/postprocessing/EffectComposer.js';
import { RenderPass } from 'three/addons/postprocessing/RenderPass.js';
import { UnrealBloomPass } from 'three/addons/postprocessing/UnrealBloomPass.js';
import { ShaderPass } from 'three/addons/postprocessing/ShaderPass.js';
import { OutputPass } from 'three/addons/postprocessing/OutputPass.js';

const Q = new URLSearchParams(location.search);
let seed = +(Q.get('seed') || 1);
const rand = () => ((seed = (seed * 16807) % 2147483647) - 1) / 2147483646;

// sampled from the video with study_video.py: sky, horizon haze, ground, hero accent
const PALETTE = { sky: 0xd8c4b5, haze: 0xc9b4a9, ground: 0x4f6b3a, accent: 0xc8402a };

const renderer = new THREE.WebGLRenderer({ antialias: true, powerPreference: 'high-performance' });
renderer.setPixelRatio(Math.min(devicePixelRatio, 2));
renderer.setSize(innerWidth, innerHeight);
renderer.toneMapping = THREE.ACESFilmicToneMapping;
renderer.toneMappingExposure = 1.0;
renderer.shadowMap.enabled = true;
document.body.appendChild(renderer.domElement);

const scene = new THREE.Scene();
scene.background = new THREE.Color(PALETTE.sky);
scene.fog = new THREE.Fog(PALETTE.haze, 40, 260);   // aerial perspective: far things melt into the haze
const camera = new THREE.PerspectiveCamera(50, innerWidth / innerHeight, 0.1, 2000);

const sun = new THREE.DirectionalLight(0xfff1dd, 2.4);
sun.position.set(30, 60, 20); sun.castShadow = true;
scene.add(sun, new THREE.HemisphereLight(PALETTE.sky, PALETTE.ground, 0.9));

// post stack, part of the spec from day one: bloom, then one grade + vignette + grain pass
const composer = new EffectComposer(renderer);
composer.addPass(new RenderPass(scene, camera));
composer.addPass(new UnrealBloomPass(new THREE.Vector2(innerWidth, innerHeight), 0.35, 0.6, 0.85));
const grade = new ShaderPass({
  uniforms: { tDiffuse: { value: null }, uTime: { value: 0 }, uLift: { value: new THREE.Vector3(0.02, 0.01, 0.0) },
    uGain: { value: new THREE.Vector3(1.04, 1.0, 0.95) }, uVignette: { value: 0.35 }, uGrain: { value: 0.035 } },
  vertexShader: 'varying vec2 vUv; void main(){ vUv = uv; gl_Position = projectionMatrix * modelViewMatrix * vec4(position,1.); }',
  fragmentShader: `uniform sampler2D tDiffuse; uniform float uTime, uVignette, uGrain; uniform vec3 uLift, uGain; varying vec2 vUv;
    float h(vec2 p){ return fract(sin(dot(p, vec2(12.9898, 78.233))) * 43758.5453); }
    void main(){ vec3 c = texture2D(tDiffuse, vUv).rgb; c = c * uGain + uLift;
      c *= 1. - uVignette * smoothstep(.35, .95, length(vUv - .5) * 1.25);
      c += (h(vUv * 999. + uTime) - .5) * uGrain; gl_FragColor = vec4(c, 1.); }`,
});
composer.addPass(grade);
composer.addPass(new OutputPass());

// world: replace with the fal assets (GLB from image-to-3D, textures from the image model)
const ground = new THREE.Mesh(new THREE.PlaneGeometry(600, 600), new THREE.MeshStandardMaterial({ color: PALETTE.ground, roughness: 1 }));
ground.rotation.x = -Math.PI / 2; ground.receiveShadow = true; scene.add(ground);
const hero = new THREE.Group();
const body = new THREE.Mesh(new THREE.CapsuleGeometry(0.4, 1, 6, 12), new THREE.MeshStandardMaterial({ color: PALETTE.accent }));
body.position.y = 0.9; body.castShadow = true; hero.add(body); scene.add(hero);
for (let i = 0; i < 60; i++) {
  const r = new THREE.Mesh(new THREE.DodecahedronGeometry(1 + rand() * 2), new THREE.MeshStandardMaterial({ color: 0x8a8070, flatShading: true }));
  r.position.set((rand() - .5) * 300, 0.5, -rand() * 300); r.castShadow = true; scene.add(r);
}
const loader = new GLTFLoader();
// loader.load('assets/hero.glb', g => { hero.remove(body); hero.add(g.scene); });

// input: keyboard + gamepad, read once per frame
const keys = new Set();
addEventListener('keydown', e => { keys.add(e.code); if (e.code === 'Space') e.preventDefault(); });
addEventListener('keyup', e => keys.delete(e.code));
function input() {
  const pad = navigator.getGamepads?.()[0];
  const ax = pad ? pad.axes[0] : 0, ay = pad ? pad.axes[1] : 0;
  return {
    x: (keys.has('ArrowRight') || keys.has('KeyD') ? 1 : 0) - (keys.has('ArrowLeft') || keys.has('KeyA') ? 1 : 0) + (Math.abs(ax) > .2 ? ax : 0),
    z: (keys.has('ArrowDown') || keys.has('KeyS') ? 1 : 0) - (keys.has('ArrowUp') || keys.has('KeyW') ? 1 : 0) + (Math.abs(ay) > .2 ? ay : 0),
    jump: keys.has('Space') || !!pad?.buttons[0]?.pressed,
    start: keys.has('Enter') || !!pad?.buttons[9]?.pressed,
  };
}

const S = { phase: Q.has('autostart') ? 'play' : 'title', pos: hero.position, vel: new THREE.Vector3(), grounded: true, t: 0 };
window.__game = { get state() { return { phase: S.phase, x: S.pos.x, y: S.pos.y, z: S.pos.z, speed: S.vel.length(), grounded: S.grounded, t: S.t }; } };

const clock = new THREE.Clock();
function frame() {
  const dt = Math.min(clock.getDelta(), 1 / 20);
  const i = input();
  if (S.phase === 'title' && (i.start || i.jump)) S.phase = 'play';
  if (S.phase === 'play') {
    S.t += dt;
    S.vel.x = THREE.MathUtils.damp(S.vel.x, i.x * 8, 10, dt);
    S.vel.z = THREE.MathUtils.damp(S.vel.z, i.z * 8, 10, dt);
    if (i.jump && S.grounded) { S.vel.y = 9; S.grounded = false; }
    S.vel.y -= 24 * dt;
    S.pos.addScaledVector(S.vel, dt);
    if (S.pos.y <= 0) { S.pos.y = 0; S.vel.y = 0; S.grounded = true; }
  }
  // follow camera, same framing as the video: behind, slightly above, looking ahead
  const want = new THREE.Vector3(S.pos.x, S.pos.y + 4, S.pos.z + 9);
  camera.position.lerp(want, 1 - Math.exp(-4 * dt));
  camera.lookAt(S.pos.x, S.pos.y + 1.4, S.pos.z - 6);
  grade.uniforms.uTime.value += dt;
  composer.render();
  requestAnimationFrame(frame);
}
addEventListener('resize', () => {
  camera.aspect = innerWidth / innerHeight; camera.updateProjectionMatrix();
  renderer.setSize(innerWidth, innerHeight); composer.setSize(innerWidth, innerHeight);
});
frame();
</script>
</body>
</html>
```

## File: skills/video-to-game/templates/play.mjs
size: 1642
sha: 84381e7f029196832d4015a2ea7a012816de8fcc
upstream: https://github.com/lovisdotio/video-to-game/blob/main/skills/video-to-game/templates/play.mjs

```text
// Playtest scenario for record_play.mjs: play the game like a good player, with the keyboard only.
//   node scripts/record_play.mjs "http://127.0.0.1:8000/?autostart&seed=7" templates/play.mjs -o take.mp4
//
// Read what the player would see (window.__game.state), decide, press keys. Never write into the game.
// Rules: always moving forward, never fall, never back into a wall, trigger every mechanic of the game
// where it looks best, and bring the key moment of the video to the center of the frame.
export async function play(page, { seconds, t0 }) {
  const k = page.keyboard;
  const state = () => page.evaluate(() => window.__game?.state);
  const end = t0 + seconds * 1000 - 150;

  if ((await state())?.phase === 'title') await k.press('Enter');
  await k.down('ArrowUp');                       // run forward the whole take
  let lastJump = 0;
  while (Date.now() < end) {
    const s = await state();
    if (!s) break;
    // jump on a rhythm while grounded, so the take shows the mechanic, not a straight walk
    if (s.grounded && Date.now() - lastJump > 900) {
      await k.down('Space'); await page.waitForTimeout(140); await k.up('Space');
      lastJump = Date.now();
    }
    // weave a little to show the world around the hero, never more than a step to a side
    const side = Math.sin((Date.now() - t0) / 700) > 0.6 ? 'ArrowRight' : Math.sin((Date.now() - t0) / 700) < -0.6 ? 'ArrowLeft' : null;
    for (const key of ['ArrowLeft', 'ArrowRight']) await (key === side ? k.down(key) : k.up(key));
    await page.waitForTimeout(50);
  }
  for (const key of ['ArrowUp', 'ArrowLeft', 'ArrowRight']) await k.up(key);
}
```

## File: README.md
upstream: https://github.com/lovisdotio/video-to-game/blob/main/README.md

```text
# Video to Game

A skill that teaches coding agents to turn a gameplay video into a real game you can play in the browser.
A video model designs the game (its look, camera and one memorable mechanic); the agent studies the clip frame by
frame, generates every image, 3D model and sound with fal to match it, writes the game in Three.js, then plays it
with the keyboard and films its best seconds as proof. It works with Claude Code, Codex, Cursor, Gemini and others.

**Play the 31 games:** [fal-video-to-game.vercel.app](https://fal-video-to-game.vercel.app)

![Video to Game: the generated video shrinks, the code and the fal assets come in, the game appears](gallery/hero.gif)

31 original games, every one rebuilt from a 10 to 15 s video generated with MiniMax H3 Max on fal: 1,013 fal calls
(594 images, 194 3D models, 290 sounds and music tracks), 202 USD in total, about 6 USD per game. Every game is a
single `index.html` with Three.js and nothing else: no engine, no template, no other library.

## Why

Agents cannot design a game's look. Asked for "a game where a boy glides between flying whales", they ship grey
boxes, a default sky and a camera that fights the player. A video model is good at exactly that part, so let it go
first, and make the agent match it:

- **The video is the design document.** `study_video.py` turns it into frames, a contact sheet, a palette and
  cut times; the agent writes a spec (camera, hero, the one mechanic, the key moment, the look) from named frames.
- **Every asset comes from fal, from the video's own frames.** Concept sheets with Nano Banana Pro, clean cutouts,
  image to 3D (Meshy, Tripo, Hunyuan 3D), rigging and animations, sound effects and music, all logged with
  `fal_asset.py`. Nothing drawn by hand, nothing downloaded.
- **The look is a spec, not polish.** Tone mapping, one grade, fog, bloom, vignette and grain from the first scene,
  and a side-by-side picture of the video and the game at the same three spots after every change.
- **Calm the world so the game reads.** One harmonized palette, dense but grouped, the hero and everything you can
  interact with standing out on a quiet background. No text on screen.
- **The agent plays its own game.** A keyboard scenario reads `window.__game` like a player reads the screen,
  `record_play.mjs` films it at the real frame rate, `best_window.py` finds the liveliest seconds. Falls, walls and
  dead seconds get fixed before anyone else plays.

## The gallery

Each GIF is the generated video (left) and the game (right). Click for the prompt, the fal models and the cost.

| | |
|---|---|
| [![Skyfarer](gallery/skyfarer/poster.gif)](gallery/skyfarer/) | [![Tidebound](gallery/tidebound/poster.gif)](gallery/tidebound/) |
| **Skyfarer** · 8.93 USD · [play](https://fal-video-to-game.vercel.app/O2_3D_SKYFARER/) · [code](gallery/skyfarer/game/)<br>Courier boy running and gliding between giant flying whales above the clouds. | **Tidebound** · 11.01 USD · [play](https://fal-video-to-game.vercel.app/H01_TIDEBOUND_coop2/) · [code](gallery/tidebound/game/)<br>Two-player co-op sailing game: two sailors crew a living ship grown from pink coral. One player steers the… |
| [![Storm Surfer](gallery/storm-surfer/poster.gif)](gallery/storm-surfer/) | [![Skyhook Rescue](gallery/skyhook-rescue/poster.gif)](gallery/skyhook-rescue/) |
| **Storm Surfer** · 4.38 USD · [play](https://fal-video-to-game.vercel.app/S30_STORM-SURFER/) · [code](gallery/storm-surfer/game/)<br>Photorealistic 3D game, cutting-edge real-time rendering quality, real-world materials: a surfer on a glowing… | **Skyhook Rescue** · 10.43 USD · [play](https://fal-video-to-game.vercel.app/D01_SKYHOOK-RESCUE_coop2/) · [code](gallery/skyhook-rescue/game/)<br>Two-player co-op rescue game during a thunderstorm: two small airships linked by a long cable fly together… |
| [![Paper Dragon](gallery/paper-dragon/poster.gif)](gallery/paper-dragon/) | [![Bubble Balloon](gallery/bubble-balloon/poster.gif)](gallery/bubble-balloon/) |
| **Paper Dragon** · 0.79 USD · [play](https://fal-video-to-game.vercel.app/B06_PAPER-DRAGON/) · [code](gallery/paper-dragon/game/)<br>A girl rides a long red paper dragon kite through an autumn canyon, the dragon's paper body rippling as it… | **Bubble Balloon** · 2.18 USD · [play](https://fal-video-to-game.vercel.app/B10_BUBBLE-BALLOON/) · [code](gallery/bubble-balloon/game/)<br>A child floats inside a giant soap bubble over a sunny seaside village, steering it by leaning, bouncing… |
| [![Kite Rider](gallery/kite-rider/poster.gif)](gallery/kite-rider/) | [![Aurora Skiff](gallery/aurora-skiff/poster.gif)](gallery/aurora-skiff/) |
| **Kite Rider** · 4.09 USD · [play](https://fal-video-to-game.vercel.app/S16_KITE-RIDER/) · [code](gallery/kite-rider/game/)<br>Watercolor-shaded 3D game: a girl stands on a giant kite, surfing the wind high above rice terraces, leaning… | **Aurora Skiff** · 2.79 USD · [play](https://fal-video-to-game.vercel.app/B08_AURORA-SKIFF/) · [code](gallery/aurora-skiff/game/)<br>A nomad steers a small sand skiff with a cloth sail across glowing blue dunes at night, catching the wind… |
| [![Glass Greenhouse](gallery/glass-greenhouse/poster.gif)](gallery/glass-greenhouse/) | [![Bell Tower](gallery/bell-tower/poster.gif)](gallery/bell-tower/) |
| **Glass Greenhouse** · 2.34 USD · [play](https://fal-video-to-game.vercel.app/B03_GLASS-GREENHOUSE/) · [code](gallery/glass-greenhouse/game/)<br>A tiny robot gardener tends a greenhouse inside a floating glass sphere above the clouds; it opens glass… | **Bell Tower** · 7.27 USD · [play](https://fal-video-to-game.vercel.app/A14_BELL-TOWER/) · [code](gallery/bell-tower/game/)<br>A bell-ringer swings on long ropes inside a giant pastel bell tower; each bell she rings sends a wave of… |
| [![Fog Shepherd](gallery/fog-shepherd/poster.gif)](gallery/fog-shepherd/) | [![Paper Moon Stairs](gallery/paper-moon-stairs/poster.gif)](gallery/paper-moon-stairs/) |
| **Fog Shepherd** · 5.04 USD · [play](https://fal-video-to-game.vercel.app/A07_FOG-SHEPHERD/) · [code](gallery/fog-shepherd/game/)<br>A cloaked figure with a lantern guides a herd of glowing white deer through a monochrome foggy forest; only… | **Paper Moon Stairs** · 6.13 USD · [play](https://fal-video-to-game.vercel.app/A02_PAPER-MOON-STAIRS/) · [code](gallery/paper-moon-stairs/game/)<br>A girl in a long white dress climbs a staircase of drifting paper lanterns into a pastel night sky; each… |
| [![Whale Song](gallery/whale-song/poster.gif)](gallery/whale-song/) | [![Sky Library](gallery/sky-library/poster.gif)](gallery/sky-library/) |
| **Whale Song** · 5.45 USD · [play](https://fal-video-to-game.vercel.app/A09_WHALE-SONG/) · [code](gallery/whale-song/game/)<br>A diver in a red scarf swims beside a giant translucent ghost whale through a deep blue gradient ocean; the… | **Sky Library** · 5.75 USD · [play](https://fal-video-to-game.vercel.app/A06_SKY-LIBRARY/) · [code](gallery/sky-library/game/)<br>A girl swims through warm air between towering floating bookshelves; loose pages flutter like birds and she… |
| [![Kintsugi Mountain](gallery/kintsugi-mountain/poster.gif)](gallery/kintsugi-mountain/) | [![Space Mower](gallery/space-mower/poster.gif)](gallery/space-mower/) |
| **Kintsugi Mountain** · 8.60 USD · [play](https://fal-video-to-game.vercel.app/A01_KINTSUGI-MOUNTAIN/) · [code](gallery/kintsugi-mountain/game/)<br>A small monk walks up a black-and-white ink-wash mountain; every footstep blooms gold leaf, and golden… | **Space Mower** · 8.25 USD · [play](https://fal-video-to-game.vercel.app/S28_SPACE-MOWER/) · [code](gallery/space-mower/game/)<br>Bright stylized 3D game on a tiny round green asteroid: an astronaut drives a little lawn mower around the… |
| [![Ember Path](gallery/ember-path/poster.gif)](gallery/ember-path/) | [![Clockwork Heist](gallery/clockwork-heist/poster.gif)](gallery/clockwork-heist/) |
| **Ember Path** · 10.03 USD · [play](https://fal-video-to-game.vercel.app/A13_EMBER-PATH/) · [code
```