⏳ This skill is pending AI review.

Scores will appear once the review pipeline completes.

version unknown

video-to-game

@lovisdotio⭐ 482 stars

>-

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

—/10

// RATINGS

⭐GitHub Stars
⭐⭐⭐ 482 on GitHubGitHub ↗

Popular

🟢ProSkills Score
—
📍

Not yet listed on ClawHub or SkillsMP

// README

Video to Game

A skill that teaches coding agents to turn a gameplay video into a real game you can play in the browser. A video model designs the game (its look, camera and one memorable mechanic); the agent studies the clip frame by frame, generates every image, 3D model and sound with fal to match it, writes the game in Three.js, then plays it with the keyboard and films its best seconds as proof. It works with Claude Code, Codex, Cursor, Gemini and others.

Play the 31 games: fal-video-to-game.vercel.app

Video to Game: the generated video shrinks, the code and the fal assets come in, the game appears

31 original games, every one rebuilt from a 10 to 15 s video generated with MiniMax H3 Max on fal: 1,013 fal calls (594 images, 194 3D models, 290 sounds and music tracks), 202 USD in total, about 6 USD per game. Every game is a single index.html with Three.js and nothing else: no engine, no template, no other library.

Why

Agents cannot design a game's look. Asked for "a game where a boy glides between flying whales", they ship grey boxes, a default sky and a camera that fights the player. A video model is good at exactly that part, so let it go first, and make the agent match it:

  • The video is the design document. study_video.py turns it into frames, a contact sheet, a palette and cut times; the agent writes a spec (camera, hero, the one mechanic, the key moment, the look) from named frames.
  • Every asset comes from fal, from the video's own frames. Concept sheets with Nano Banana Pro, clean cutouts, image to 3D (Meshy, Tripo, Hunyuan 3D), rigging and animations, sound effects and music, all logged with fal_asset.py. Nothing drawn by hand, nothing downloaded.
  • The look is a spec, not polish. Tone mapping, one grade, fog, bloom, vignette and grain from the first scene, and a side-by-side picture of the video and the game at the same three spots after every change.
  • Calm the world so the game reads. One harmonized palette, dense but grouped, the hero and everything you can interact with standing out on a quiet background. No text on screen.
  • The agent plays its own game. A keyboard scenario reads window.__game like a player reads the screen, record_play.mjs films it at the real frame rate, best_window.py finds the liveliest seconds. Falls, walls and dead seconds get fixed before anyone else plays.

The gallery

Each GIF is the generated video (left) and the game (right). Click for the prompt, the fal models and the cost.

SkyfarerTidebound
Skyfarer · 8.93 USD · play · codeCourier boy running and gliding between giant flying whales above the clouds.Tidebound · 11.01 USD · play · codeTwo-player co-op sailing game: two sailors crew a living ship grown from pink coral. One player steers the…
Storm SurferSkyhook Rescue
Storm Surfer · 4.38 USD · play · codePhotorealistic 3D game, cutting-edge real-time rendering quality, real-world materials: a surfer on a glowing…Skyhook Rescue · 10.43 USD · play · codeTwo-player co-op rescue game during a thunderstorm: two small airships linked by a long cable fly together…
Paper DragonBubble Balloon
Paper Dragon · 0.79 USD · play · codeA girl rides a long red paper dragon kite through an autumn canyon, the dragon's paper body rippling as it…Bubble Balloon · 2.18 USD · play · codeA child floats inside a giant soap bubble over a sunny seaside village, steering it by leaning, bouncing…
Kite RiderAurora Skiff
Kite Rider · 4.09 USD · play · codeWatercolor-shaded 3D game: a girl stands on a giant kite, surfing the wind high above rice terraces, leaning…Aurora Skiff · 2.79 USD · play · codeA nomad steers a small sand skiff with a cloth sail across glowing blue dunes at night, catching the wind…
Glass GreenhouseBell Tower
Glass Greenhouse · 2.34 USD · play · codeA tiny robot gardener tends a greenhouse inside a floating glass sphere above the clouds; it opens glass…Bell Tower · 7.27 USD · play · codeA bell-ringer swings on long ropes inside a giant pastel bell tower; each bell she rings sends a wave of…
Fog ShepherdPaper Moon Stairs
Fog Shepherd · 5.04 USD · play · codeA cloaked figure with a lantern guides a herd of glowing white deer through a monochrome foggy forest; only…Paper Moon Stairs · 6.13 USD · play · codeA girl in a long white dress climbs a staircase of drifting paper lanterns into a pastel night sky; each…
Whale SongSky Library
Whale Song · 5.45 USD · play · codeA diver in a red scarf swims beside a giant translucent ghost whale through a deep blue gradient ocean; the…Sky Library · 5.75 USD · play · codeA girl swims through warm air between towering floating bookshelves; loose pages flutter like birds and she…
Kintsugi MountainSpace Mower
Kintsugi Mountain · 8.60 USD · play · codeA small monk walks up a black-and-white ink-wash mountain; every footstep blooms gold leaf, and golden…Space Mower · 8.25 USD · play · codeBright stylized 3D game on a tiny round green asteroid: an astronaut drives a little lawn mower around the…
Ember PathClockwork Heist
Ember Path · 10.03 USD · play · [code

// HOW IT'S BUILT

// FILES

14 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • skills/video-to-game/SKILL.md6.1 KB275d1ce40730upstream ↗
    ---
    name: video-to-game
    description: >-
      Turns a gameplay video (generated by a video model such as MiniMax H3 Max or Seedance, or any clip) into a real,
      playable WebGL game in Three.js whose every image, 3D model, sound and music track is generated with fal, then
      proves it: side-by-side frames of the video and the game, and a scenario that plays the game with the keyboard
      and films its best seconds. Ships scripts to study the video (frames, palette, cuts), run and log fal models,
      record a playtest at the real frame rate, compare game and video, and find the liveliest window of a take.
      Use when the user wants a video, an AI gameplay clip or a game concept "turned into a game", "made playable",
      "recreated in the browser" or "rebuilt in Three.js/WebGL". Not for game engines (Unity, Unreal, Godot) or for
      copying existing commercial games.
    license: MIT
    metadata:
      hermes:
        tags:
          - threejs
          - webgl
          - game-development
          - video-to-game
          - fal
          - generative-assets
        category: development
    ---
    
    # Video to Game
    
    The video is the game design document, and you cannot watch it. A video model is good at what you are bad at:
    the look of a game, its camera, its one memorable mechanic. Your job is to reverse-engineer that clip into a
    game someone can play, with assets generated to match it, and to prove it with pictures, not adjectives.
    
    ## Workflow
    
    1. **Generate or take the video.** One original concept, one mechanic you can see, a third-person or side camera,
       no HUD text. `prompts/gameplay-video-prompts.md` (repo root) has the recipe and 31 prompts that worked.
    2. **Study it.** `python3 scripts/study_video.py gameplay.mp4 -o study/` saves a frame every 0.25 s, a contact sheet,
       the palette and the cuts. Open the sheet, then the frames. Write the spec before any code:
       camera (distance, height, lens), hero, the one core mechanic, world layout, palette, light, materials, mood,
       and the key moment the video sells. See `references/study-the-video.md`.
    3. **Generate every asset with fal**, one job at a time, from the video's own frames as references:
       concept sheet, background removal, image to 3D, rigging, textures, parallax layers, sound effects, music.
       `python3 scripts/fal_asset.py <endpoint> --arg k=v --file k=path -o assets/` runs a model and logs it.
       Nothing drawn by hand, nothing downloaded elsewhere. See `references/fal-assets.md`.
    4. **Build the game** from `templates/game.html`: Three.js only, one import map, a real loop (controls, goal,
       progression, feedback, end, restart), keyboard and gamepad, `?autostart` and `?seed=N`, `window.__game` state.
       See `references/game-loop.md`.
    5. **Match the look.** Palette, fog, light, grade, bloom, vignette and grain are part of the spec from day one,
       not polish. Compare a game frame with a video frame at fixed spots after every change:
       `python3 scripts/side_by_side.py gameplay.mp4 shot.png -o pairs/ --times 2,5,8`.
       See `references/look-and-post.md`.
    6. **Play it yourself.** Write `play.mjs` from `templates/play.mjs`: keyboard only, reading `window.__game`.
       `node scripts/record_play.mjs "http://127.0.0.1:8000/?autostart" play.mjs -o take.mp4` films it at its real
       frame rate; `python3 scripts/best_window.py take.mp4 --length 10` finds the liveliest stretch; look at those
       frames. Fix what a player would hit: falls, walls, dead seconds, unreadable moments. See `references/playtest.md`.
    7. **Deliver:** `index.html` + `assets/` + `README.md` (concept, controls, every fal asset with its endpoint),
       the side-by-side pairs, the best take, and an honest list of what differs from the video.
    
    Scripts need Python 3.9+, ffmpeg, Node 18+ with `playwright-core` for the recorder, and `pip install fal-client`
    with `FAL_KEY` set for `fal_asset.py`. Run them from this skill's directory or with full paths.
    
    ## Hard rules
    
    - **The game looks like its video.** Same camera, palette, light and materials. Every look decision is checked
      on a side-by-side pair, at the same three spots each time. "Looks right" without a pair image is not a check.
    - **The video's mechanic is the game.** If the clip shows a girl turning the world upside down, the game is about
      turning the world upside down, not a generic runner with her model in it.
    - **Every asset comes from fal**, generated from the video's frames as references. Log each call (file, endpoint).
      One job at a time: look at each result before asking for the next.
    - **No text on screen** beyond a minimal title and one controls line. The world explains itself.
    - **Original worlds only.** No characters, logos, names or level designs from existing games or films.
    - **Post-processing is in the spec**: tone mapping, one grade, bloom, fog for depth, vignette, grain. Keep 60 fps
      on a laptop; 30 fps is acceptable only for a much better image.
    - **Calm the world, let the game read.** One harmonized palette, no acid colors; dense but grouped; hero and
      interactive things stand out on a quiet background.
    - **Play before you ship.** A recorded take with zero console errors, no fall, no dead second, the key moment
      of the video visible.
    
    ## Reference routing
    
    | File | Read when |
    |---|---|
    | `references/study-the-video.md` | starting: frames, palette, camera, mechanic, the spec to write |
    | `references/fal-assets.md` | generating assets: which model for what, the character pipeline, prompts that work, costs |
    | `references/game-loop.md` | building: structure, input, camera, state contract, multiplayer, performance |
    | `references/look-and-post.md` | matching the video's look: grade, fog, bloom, density, hierarchy, the side-by-side routine |
    | `references/playtest.md` | playing your own game: scenarios, recording, best window, what to fix |
    
    Templates: `templates/game.html` (Three.js skeleton with the post stack, input, state contract),
    `templates/play.mjs` (a keyboard playtest scenario).
    
    ## Output expectations
    
    - Say what was checked and how: which pairs, which take, fps, console errors.
    - List the fal calls: count per endpoint and total cost.
    - List honestly what differs from the video (a simplified mechanic, a missing effect), never "identical".
    
  • skills/video-to-game/scripts/best_window.py2.6 KB3cbad03d2266upstream ↗
    #!/usr/bin/env python3
    """Find the liveliest stretch of a recorded take, to show the game at its best.
    
    Scores every frame by how much the picture changes (ffmpeg scene score), then ranks windows of --length
    seconds. A high score means action on screen; a flat one means the player stood still or a menu was up.
    Always look at the frames of the winning window before using it: motion is not the same as beauty.
    
    Usage:
      python3 best_window.py take.mp4 --length 3
      python3 best_window.py take.mp4 --length 10 --top 3 --cut best.mp4
    
    Needs ffmpeg on PATH. Standard library only.
    """
    import argparse
    import re
    import subprocess
    
    
    def scores(path):
        err = subprocess.run(['ffmpeg', '-v', 'info', '-i', path, '-vf', "select='gte(scene,0)',metadata=print",
                              '-an', '-f', 'null', '-'], capture_output=True, text=True).stderr
        t = [float(x) for x in re.findall(r'pts_time:([\d.]+)', err)]
        s = [float(x) for x in re.findall(r'lavfi\.scene_score=([\d.]+)', err)]
        return list(zip(t, s))
    
    
    def main():
        ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
        ap.add_argument('take')
        ap.add_argument('--length', type=float, default=3.0)
        ap.add_argument('--top', type=int, default=3)
        ap.add_argument('--skip', type=float, default=1.0, help='ignore the first seconds (loading, title)')
        ap.add_argument('--cut', help='also cut the best window to this file')
        a = ap.parse_args()
    
        pts = [(t, s) for t, s in scores(a.take) if t >= a.skip]
        if not pts:
            raise SystemExit('no frames scored')
        end = pts[-1][0]
        ranked = []
        step = 0.25
        start = pts[0][0]
        while start + a.length <= end:
            win = [s for t, s in pts if start <= t < start + a.length]
            # motion everywhere beats one big spike: reward the median as much as the mean
            if win:
                med = sorted(win)[len(win) // 2]
                ranked.append((sum(win) / len(win) + med, round(start, 2)))
            start += step
        ranked.sort(reverse=True)
        picked = []
        for score, st in ranked:
            if all(abs(st - p) >= a.length for _, p in picked):
                picked.append((score, st))
            if len(picked) == a.top:
                break
        for score, st in picked:
            print(f'{st:6.2f} s -> {st + a.length:6.2f} s   score {score:.4f}')
        if a.cut and picked:
            st = picked[0][1]
            subprocess.run(['ffmpeg', '-v', 'error', '-y', '-ss', str(st), '-i', a.take, '-t', str(a.length),
                            '-c:v', 'libx264', '-crf', '18', '-an', a.cut], check=True)
            print(a.cut)
    
    
    if __name__ == '__main__':
        main()
    
  • skills/video-to-game/scripts/fal_asset.py4.5 KB5c79a3e4bf15upstream ↗
    #!/usr/bin/env python3
    """Run one fal model, download what it returns, and log the call.
    
    Every asset of the game goes through this script, so the log (fal_log.json next to the outputs) lists each
    file with the endpoint that made it. Local files passed with --file are uploaded to the fal CDN first.
    
    Usage:
      python3 fal_asset.py minimax/h3-max/text-to-video -o video/ --arg prompt="..." --arg duration=10 \\
          --arg prompt_expansion_mode=disabled --arg resolution=1080P
      python3 fal_asset.py fal-ai/nano-banana-pro/edit -o assets/ --name hero_concept \\
          --arg prompt="the hero alone, A-pose, plain light grey background" --file image_urls=study/f_012.jpg
      python3 fal_asset.py fal-ai/birefnet/v2 -o assets/ --name hero_cutout --file image_url=assets/hero_concept.png
      python3 fal_asset.py meshy/v7.1/image-to-3d -o assets/ --name hero --file image_url=assets/hero_cutout.png
      python3 fal_asset.py fal-ai/elevenlabs/sound-effects/v2 -o assets/ --name jump --arg text="soft cloth whoosh"
    
    --arg key=value   JSON value if it parses (numbers, true, lists), else a string. Repeat for each field.
    --file key=path   uploads the file; repeat the same key to build a list (image_urls=a.png --file image_urls=b.png).
    
    Needs FAL_KEY in the environment and `pip install fal-client`. One job at a time: wait for it, look at it,
    then decide the next one.
    """
    import argparse
    import json
    import mimetypes
    import os
    import sys
    import time
    import urllib.request
    
    LIST_KEYS = ('image_urls', 'animation_action_ids', 'reference_image_urls')
    
    
    def parse_value(v):
        try:
            return json.loads(v)
        except ValueError:
            return v
    
    
    def urls_in(obj):
        """Every downloadable file URL in a fal result, with the key path it came from."""
        out = []
        if isinstance(obj, dict):
            if isinstance(obj.get('url'), str) and obj['url'].startswith('http'):
                out.append(('', obj['url'], obj.get('file_name') or obj.get('content_type')))
            for k, v in obj.items():
                if k != 'url':
                    out += [(f'{k}.{p}'.strip('.'), u, n) for p, u, n in urls_in(v)]
        elif isinstance(obj, list):
            for i, v in enumerate(obj):
                out += [(f'{i}.{p}'.strip('.'), u, n) for p, u, n in urls_in(v)]
        return out
    
    
    def ext_for(url, hint):
        path = url.split('?')[0]
        ext = os.path.splitext(path)[1]
        if ext and len(ext) <= 5:
            return ext
        return mimetypes.guess_extension(hint or '') or '.bin'
    
    
    def main():
        ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
        ap.add_argument('endpoint')
        ap.add_argument('-o', '--out', default='assets')
        ap.add_argument('--name', help='base name of the saved files (default: endpoint + time)')
        ap.add_argument('--arg', action='append', default=[], metavar='KEY=VALUE')
        ap.add_argument('--file', action='append', default=[], metavar='KEY=PATH')
        a = ap.parse_args()
    
        if not os.environ.get('FAL_KEY'):
            sys.exit('FAL_KEY is not set')
        import fal_client  # pip install fal-client
    
        args = {}
        for kv in a.arg:
            k, v = kv.split('=', 1)
            args[k] = parse_value(v)
        for kv in a.file:
            k, p = kv.split('=', 1)
            url = fal_client.upload_file(p)
            if k in args or k in LIST_KEYS:
                args.setdefault(k, [])
                args[k] = (args[k] if isinstance(args[k], list) else [args[k]]) + [url]
            else:
                args[k] = url
    
        os.makedirs(a.out, exist_ok=True)
        name = a.name or a.endpoint.replace('/', '_') + time.strftime('_%H%M%S')
        t0 = time.time()
        result = fal_client.subscribe(a.endpoint, arguments=args, with_logs=False)
        files = []
        found = urls_in(result)
        for i, (key, url, hint) in enumerate(found):
            dest = os.path.join(a.out, name + ('' if len(found) == 1 else f'_{i}') + ext_for(url, hint))
            urllib.request.urlretrieve(url, dest)
            files.append({'file': dest, 'key': key, 'url': url})
    
        log_path = os.path.join(a.out, 'fal_log.json')
        log = json.load(open(log_path)) if os.path.exists(log_path) else []
        log.append({'endpoint': a.endpoint, 'name': name, 'seconds': round(time.time() - t0, 1),
                    'arguments': {k: v for k, v in args.items() if k != 'prompt'}, 'prompt': args.get('prompt'),
                    'files': [f['file'] for f in files]})
        with open(log_path, 'w') as f:
            json.dump(log, f, indent=1)
        for f in files:
            print(f['file'])
        if not files:
            print(json.dumps(result, indent=1)[:2000])
    
    
    if __name__ == '__main__':
        main()
    
  • skills/video-to-game/scripts/record_play.mjs3.8 KBc5a573850e67upstream ↗
    #!/usr/bin/env node
    // Film the game while a scenario plays it with the keyboard, at its real frame rate.
    //
    //   node record_play.mjs http://127.0.0.1:8000/?autostart play.mjs -o take.mp4 [--seconds 18] [--size 1920x1080]
    //
    // play.mjs exports `async function play(page, { seconds, t0 })` and only presses keys (page.keyboard) and
    // reads state the game exposes (window.__game). The take is encoded with each frame's real duration, so a
    // slow moment stays slow instead of being stretched. Needs Node 18+, ffmpeg, and `npm i playwright-core`
    // (or playwright). Uses the system Chrome when found, for real GPU WebGL.
    import fs from 'node:fs';
    import os from 'node:os';
    import path from 'node:path';
    import { pathToFileURL } from 'node:url';
    import { execFileSync } from 'node:child_process';
    
    const argv = process.argv.slice(2);
    const opt = (k, d) => { const i = argv.indexOf(k); return i >= 0 ? argv.splice(i, 2)[1] : d; };
    const out = opt('-o', 'take.mp4'), seconds = +opt('--seconds', 18), [W, H] = opt('--size', '1920x1080').split('x').map(Number);
    const [url, playFile] = argv;
    if (!url || !playFile) { console.error('usage: node record_play.mjs URL play.mjs -o take.mp4 [--seconds 18]'); process.exit(1); }
    
    let chromium;
    try { ({ chromium } = await import('playwright-core')); } catch { ({ chromium } = await import('playwright')); }
    const chrome = ['/Applications/Google Chrome.app/Contents/MacOS/Google Chrome', '/usr/bin/google-chrome', '/usr/bin/chromium']
      .find(p => fs.existsSync(p));
    const browser = await chromium.launch({
      executablePath: chrome, headless: true,
      args: ['--enable-gpu', '--ignore-gpu-blocklist', '--use-angle=' + (process.platform === 'darwin' ? 'metal' : 'default'),
        '--autoplay-policy=no-user-gesture-required', '--mute-audio', '--disable-background-timer-throttling'],
    });
    const ctx = await browser.newContext({ viewport: { width: W, height: H } });
    const page = await ctx.newPage();
    const errors = [];
    page.on('pageerror', e => errors.push(String(e)));
    page.on('console', m => m.type() === 'error' && errors.push(m.text()));
    await page.goto(url, { waitUntil: 'load', timeout: 90000 });
    await page.waitForTimeout(2500);
    await page.mouse.click(W / 2, H / 2).catch(() => {});
    
    const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'take-'));
    const cdp = await ctx.newCDPSession(page);
    const times = []; let n = 0;
    cdp.on('Page.screencastFrame', f => {
      fs.writeFileSync(path.join(dir, String(n++).padStart(5, '0') + '.jpg'), Buffer.from(f.data, 'base64'));
      times.push(f.metadata.timestamp);
      cdp.send('Page.screencastFrameAck', { sessionId: f.sessionId }).catch(() => {});
    });
    await cdp.send('Page.startScreencast', { format: 'jpeg', quality: 90, maxWidth: W, maxHeight: H, everyNthFrame: 1 });
    const { play } = await import(pathToFileURL(path.resolve(playFile)).href);
    const t0 = Date.now();
    await Promise.race([play(page, { seconds, t0 }).catch(e => errors.push('play: ' + e)), page.waitForTimeout((seconds + 15) * 1000)]);
    await cdp.send('Page.stopScreencast');
    await browser.close();
    
    if (n < 2) { console.error('no frames captured', errors.slice(0, 3)); process.exit(1); }
    const list = times.map((t, i) => `file '${String(i).padStart(5, '0')}.jpg'\nduration ${((times[i + 1] ?? t + 1 / 60) - t).toFixed(5)}`);
    fs.writeFileSync(path.join(dir, 'list.txt'), list.join('\n') + `\nfile '${String(n - 1).padStart(5, '0')}.jpg'\n`);
    execFileSync('ffmpeg', ['-v', 'error', '-y', '-f', 'concat', '-safe', '0', '-i', 'list.txt',
      '-vf', `scale=${W}:${H}:flags=lanczos,fps=60,format=yuv420p`, '-c:v', 'libx264', '-crf', '20', path.resolve(out)], { cwd: dir });
    fs.rmSync(dir, { recursive: true, force: true });
    const dur = times.at(-1) - times[0];
    console.log(`${out}: ${n} frames, ${dur.toFixed(1)} s, ${(n / dur).toFixed(1)} fps, ${errors.length} console errors`);
    for (const e of errors.slice(0, 5)) console.log('  ' + e.slice(0, 200));
    
  • skills/video-to-game/scripts/side_by_side.py2.3 KB386b99b26cfcupstream ↗
    #!/usr/bin/env python3
    """Put the source video and the game next to each other, so "it looks like the video" becomes a picture.
    
    Writes one image per time: source on the left, game on the right, same height. Use it on a recorded take
    (record_play.mjs) or on screenshots of the game at fixed spots.
    
    Usage:
      python3 side_by_side.py gameplay.mp4 take.mp4 -o pairs/ --times 1,3,5,8
      python3 side_by_side.py gameplay.mp4 shot.png -o pairs/ --times 4        # a still of the game
      python3 side_by_side.py gameplay.mp4 take.mp4 -o pairs/ --every 1 --game-offset 2.5
    
    Needs ffmpeg on PATH. Standard library only.
    """
    import argparse
    import os
    import subprocess
    
    
    def main():
        ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
        ap.add_argument('source')
        ap.add_argument('game', help='a recorded take (.mp4) or a still (.png/.jpg)')
        ap.add_argument('-o', '--out', default='pairs')
        ap.add_argument('--times', help='comma separated times in the source, in seconds')
        ap.add_argument('--every', type=float, help='or one pair every N seconds')
        ap.add_argument('--game-offset', type=float, default=0, help='game time = source time + offset')
        ap.add_argument('--height', type=int, default=540)
        a = ap.parse_args()
    
        dur = float(subprocess.run(['ffprobe', '-v', 'error', '-show_entries', 'format=duration', '-of', 'csv=p=0', a.source],
                                   capture_output=True, text=True, check=True).stdout)
        if a.times:
            times = [float(t) for t in a.times.split(',')]
        else:
            step = a.every or 1.0
            times = [round(i * step, 2) for i in range(int(dur / step))]
        still = a.game.lower().endswith(('.png', '.jpg', '.jpeg', '.webp'))
        os.makedirs(a.out, exist_ok=True)
        for t in times:
            dest = os.path.join(a.out, f'pair_{t:06.2f}.jpg')
            game_in = ['-i', a.game] if still else ['-ss', str(max(0, t + a.game_offset)), '-i', a.game]
            subprocess.run(['ffmpeg', '-v', 'error', '-y', '-ss', str(t), '-i', a.source, *game_in, '-filter_complex',
                            f'[0:v]scale=-2:{a.height},setsar=1[s];[1:v]scale=-2:{a.height},setsar=1[g];[s][g]hstack=2',
                            '-frames:v', '1', '-q:v', '3', dest], check=True)
            print(dest)
    
    
    if __name__ == '__main__':
        main()
    
  • skills/video-to-game/scripts/study_video.py4.0 KB1570ec70d3e2upstream ↗
    #!/usr/bin/env python3
    """Turn a gameplay video into things an agent can look at.
    
    Saves a frame every --every seconds, a contact sheet, the dominant palette (hex), the cut times, and a
    study.json summary. Open the sheet first, then the frames, then write the game spec.
    
    Usage:
      python3 study_video.py gameplay.mp4 -o study/
      python3 study_video.py gameplay.mp4 -o study/ --every 0.5 --colors 12
    
    Needs ffmpeg and ffprobe on PATH. Standard library only.
    """
    import argparse
    import json
    import os
    import re
    import subprocess
    from collections import Counter
    
    
    def probe(path):
        out = subprocess.run(['ffprobe', '-v', 'error', '-select_streams', 'v:0', '-show_entries',
                              'stream=width,height,r_frame_rate:format=duration', '-of', 'json', path],
                             capture_output=True, text=True, check=True).stdout
        d = json.loads(out)
        s = d['streams'][0]
        num, den = s['r_frame_rate'].split('/')
        return {'width': s['width'], 'height': s['height'], 'fps': round(int(num) / int(den), 3),
                'duration': round(float(d['format']['duration']), 3)}
    
    
    def frames(path, out, every):
        os.makedirs(out, exist_ok=True)
        subprocess.run(['ffmpeg', '-v', 'error', '-y', '-i', path, '-vf', f'fps=1/{every},scale=960:-2',
                        '-q:v', '3', os.path.join(out, 'f_%03d.jpg')], check=True)
        names = sorted(f for f in os.listdir(out) if f.startswith('f_'))
        return [(round(i * every, 2), n) for i, n in enumerate(names)]
    
    
    def sheet(path, out, n):
        cols = 6
        rows = max(1, -(-n // cols))
        every = probe(path)['duration'] / max(1, cols * rows)
        subprocess.run(['ffmpeg', '-v', 'error', '-y', '-i', path, '-vf',
                        f'fps=1/{every:.4f},scale=480:-2,tile={cols}x{rows}:padding=6:color=white',
                        '-frames:v', '1', os.path.join(out, 'sheet.jpg')], check=True)
    
    
    def palette(path, k):
        """Dominant colors from 16 small frames, quantized to 4 bits per channel."""
        raw = subprocess.run(['ffmpeg', '-v', 'error', '-i', path, '-vf', 'fps=16/10,scale=64:36', '-frames:v', '16',
                              '-f', 'rawvideo', '-pix_fmt', 'rgb24', '-'], capture_output=True, check=True).stdout
        bins = Counter()
        sums = {}
        for i in range(0, len(raw) - 2, 3):
            r, g, b = raw[i], raw[i + 1], raw[i + 2]
            key = (r >> 4, g >> 4, b >> 4)
            bins[key] += 1
            s = sums.setdefault(key, [0, 0, 0])
            s[0] += r; s[1] += g; s[2] += b
        total = sum(bins.values())
        out = []
        for key, n in bins.most_common(k):
            s = sums[key]
            out.append({'hex': '#%02x%02x%02x' % (s[0] // n, s[1] // n, s[2] // n), 'share': round(n / total, 3)})
        return out
    
    
    def cuts(path, threshold):
        err = subprocess.run(['ffmpeg', '-v', 'info', '-i', path, '-vf', f"select='gt(scene,{threshold})',showinfo",
                              '-f', 'null', '-'], capture_output=True, text=True).stderr
        return [round(float(t), 2) for t in re.findall(r'pts_time:([\d.]+)', err)]
    
    
    def main():
        ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
        ap.add_argument('video')
        ap.add_argument('-o', '--out', default='study')
        ap.add_argument('--every', type=float, default=0.25, help='seconds between saved frames (default 0.25)')
        ap.add_argument('--colors', type=int, default=10)
        ap.add_argument('--cut-threshold', type=float, default=0.35)
        a = ap.parse_args()
    
        info = probe(a.video)
        fr = frames(a.video, a.out, a.every)
        sheet(a.video, a.out, 24)
        info.update(palette=palette(a.video, a.colors), cuts=cuts(a.video, a.cut_threshold),
                    frames=[{'t': t, 'file': n} for t, n in fr], sheet='sheet.jpg')
        with open(os.path.join(a.out, 'study.json'), 'w') as f:
            json.dump(info, f, indent=1)
        print(f"{info['width']}x{info['height']} {info['fps']} fps {info['duration']} s, {len(fr)} frames, "
              f"{len(info['cuts'])} cuts, palette {' '.join(c['hex'] for c in info['palette'])}")
        print(f'open {os.path.join(a.out, "sheet.jpg")} first')
    
    
    if __name__ == '__main__':
        main()
    
  • skills/video-to-game/references/fal-assets.md4.3 KB1219b92e74f6upstream ↗
    # Generate every asset with fal
    
    Rule of the project: every image, 3D model, sound and music track in the game is generated on fal, from the
    video's own frames as references, so the assets share the video's style. Nothing is drawn by hand or downloaded.
    
    Run each call with `scripts/fal_asset.py`, which uploads local files, downloads the outputs and appends to
    `fal_log.json`. One job at a time: open the result, judge it against the video, then ask for the next.
    Check endpoint ids and parameters in the live catalog (fal.ai/models) before a first call; do not guess them.
    
    ## What the 31 games of the gallery used
    
    1,013 calls in total, about 6 USD per game (median 6.13, from 0.79 to 15.48).
    
    | Job | Endpoint (calls) |
    |---|---|
    | Concepts, sprites, textures, sky, parallax layers | `fal-ai/nano-banana-pro/edit` (388), `fal-ai/nano-banana-pro` (73) |
    | Background removal | `fal-ai/birefnet/v2` (74), `fal-ai/bria/background/remove` (12) |
    | Image to 3D (GLB) | `meshy/v7.1/image-to-3d` (81), `tripo3d/h3.1/image-to-3d` (40), Trellis 2 (23), `fal-ai/hunyuan-3d/v3.1/pro/image-to-3d` (17), Rodin (15) |
    | Rigging and animations | `fal-ai/meshy/rigging/multi-animation` (24) |
    | Sound effects | `fal-ai/elevenlabs/sound-effects/v2` (165), Stable Audio 3 SFX (41), Sonilo SFX (26) |
    | Music | `elevenlabs/music/v2.5` (22), `google/lyria-3.5` (8) |
    | The gameplay video itself | `minimax/h3-max/text-to-video` (30), `bytedance/seedance-2.5/text-to-video` (1) |
    
    ## The character pipeline
    
    1. **Concept from the video.** `fal-ai/nano-banana-pro/edit` with 1 to 3 frames where the hero is clearest as
       `image_urls`, and a prompt like: "the same character alone, full body, A-pose, arms slightly away from the body,
       neutral expression, plain light grey background, even studio light, same art style and colors as the reference".
       A-pose (not T-pose) rigs better and keeps sleeves and capes readable.
    2. **Optional turnaround.** Same model, "front, side and back views of the same character on one sheet", for
       multi-view 3D models and to check the design from behind (the camera mostly sees the back).
    3. **Cut it out.** `fal-ai/birefnet/v2` on the concept. A clean alpha gives a clean mesh.
    4. **To 3D.** `meshy/v7.1/image-to-3d` (`image_url`, options for A/T pose and PBR) or `tripo3d/h3.1/image-to-3d`
       for characters; Hunyuan 3D Pro (`input_image_url`, plus optional back and side views) for detailed props.
       Normalize every GLB on load: scale to a known height, put the feet at y = 0, face -Z.
    5. **Rig and animate.** `fal-ai/meshy/rigging/multi-animation` with `model_url` and `animation_action_ids`
       (run, jump, idle, glide) and `height_meters`. Keep the clips in one GLB and blend them with an AnimationMixer.
    
    ## Environments and textures
    
    - **Wide layers** (far mountains, cloud banks, city skyline): one wide image per depth, then background removal,
      on planes at different distances for parallax. Ask for "isolated on a flat white background" so the cutout is clean.
    - **Tileable materials** (grass, stone, roof tiles, sand): "seamless tileable texture, top-down, flat even light,
      no shadows, no perspective". Check the tiling by repeating it 3x3 before using it.
    - **Sprites and effects** (petals, sparks, leaves, puffs): an atlas of 4x4 on a flat color, then background removal.
    - **Props** (houses, trees, lanterns): concept on grey, cutout, image to 3D. Reuse one model many times with
      scale, rotation and tint variations instead of generating hundreds.
    - Keep the video frame in `image_urls` for everything, so a texture made in isolation still belongs to the world.
    
    ## Sound
    
    - One sound per player action and per world reaction: jump, land, glide loop, collect, the key mechanic, ambient bed.
      `fal-ai/elevenlabs/sound-effects/v2` takes `text`, `duration_seconds`, `loop` (true for wind, engines, rain).
    - Music: `elevenlabs/music/v2.5` with `force_instrumental: true`, or a `composition_plan` of sections with
      durations when the music must build at a precise moment. Normalize loudness so effects sit above the music.
    
    ## Cost habits
    
    - Log every call (the script does). Report calls per endpoint and the total at the end.
    - Generate 2 or 3 variations only for the hero and the key mechanic's assets; one is enough for background props.
    - Shrink textures before shipping (WebP, 1K for props, 2K for terrain): a game with 8 characters and 80 props
      still loads in seconds.
    
  • skills/video-to-game/references/game-loop.md2.4 KB95d726b60411upstream ↗
    # Build the game
    
    Start from `templates/game.html`. One `index.html`, an `assets/` folder, Three.js from a pinned import map
    (`[email protected]` on jsDelivr) and its examples (GLTFLoader, EffectComposer). No other library, no engine,
    no build step: the file opens from any static server.
    
    ## A real loop, not a scene
    
    - **Controls** in one line: arrows or WASD to move, Space for the main action (hold for the long version:
      glide, pour, charge), one more key at most. Gamepad too: left stick, A for the action, Start to begin.
    - **A goal** the player understands without text: reach the lighthouse, wake every tree, rescue the villagers.
    - **Progression**: the world changes as the player succeeds (color spreads, a new island, faster wind).
    - **Feedback** on every action: particles, a camera nudge, a sound, the world reacting.
    - **An end** and a restart (R or Start). A short calm ending shot beats a score screen.
    - **Multiplayer** when the video shows it: local co-op on one keyboard (WASD + arrows) or AI teammates that
      play well enough to be shown in a trailer.
    
    ## The contract with the playtest
    
    - `?autostart` skips the title, `?seed=N` fixes the level, and optional debug parameters can place the player
      (`?x=120`) or the camera (`?camd=24`) so a scenario reaches the key moment fast. The rendered game must stay the
      real one.
    - `window.__game.state` returns read-only numbers a player would see on screen: phase, position, speed, grounded,
      the next obstacle or target. Scenarios read it to time their keys; they never write into the game.
    
    ## Camera
    
    Reproduce the video's framing first, then make it playable: smooth follow with damping (`1 - exp(-k dt)`),
    a little look-ahead in the direction of motion, a slight lag on height so jumps read, and no clipping into the
    world. The hero stays big enough to read, near the center, as in the video.
    
    ## Performance
    
    - 60 fps on a laptop is the target. Instancing for anything repeated (trees, rocks, grass tufts, birds),
      merged static geometry, shadows from one light with a tight frustum that follows the player.
    - Texture sizes from the asset step (1K props, 2K terrain), mipmaps on, anisotropy 4 to 8.
    - Measure: log `renderer.info.render.calls` and the frame time; drop the most expensive effect before cutting beauty.
    
    ## Text
    
    The only text: a minimal title, one controls line, maybe an ending line. No tutorials, no HUD chrome, no fake
    UI copied from the video. Counters, if any, are small icons in a corner.
    
  • skills/video-to-game/references/look-and-post.md2.5 KB4d39851da743upstream ↗
    # Match the look
    
    The first versions of these games were playable and wrong: the right mechanic in a flat, colorful, generic
    world that looked nothing like its video. What fixed them was treating the look as a spec with measurements,
    and comparing pictures after every change.
    
    ## The side-by-side routine
    
    1. Pick three spots that cover the game: the start, the key moment, and somewhere far from the start
       (expose a `?x=` or a `flyTo` for it).
    2. Screenshot the game at those spots, and pick the video frames that show the same kind of moment.
    3. `python3 scripts/side_by_side.py gameplay.mp4 shot.png -o pairs/ --times 4` and look at each pair.
    4. Write what differs (too saturated, sky too clean, fog too thick, hero too small), fix one thing, repeat at the
       same three spots. Always the same spots, or you are comparing luck.
    
    ## Post-processing is part of the build
    
    Set it up with the first scene, not at the end:
    
    - **Tone mapping** ACES Filmic, exposure tuned on a pair.
    - **One grade** for the whole game, like a LUT: lift, gain, a hue shift in the shadows. Change the grade, not
      forty materials.
    - **Fog / aerial perspective**: far things melt into the haze color from the palette. It gives depth and hides
      the edge of the world.
    - **Bloom** only on what glows in the video (sun, lanterns, magic). Threshold high.
    - **Vignette and grain** at low strength; they glue generated textures together.
    - **Depth of field / tilt-shift** only if the video has it, and keep the horizon readable.
    
    ## Calm the world
    
    The usual failure after "make it richer" is noise: random props everywhere, saturated greens, detail at every
    pixel. Then nothing reads.
    
    - **One harmonized palette.** Olive and sage instead of acid green, slate blues, earthy ochres: whatever the video
      uses, pulled together by the grade.
    - **Group, do not scatter.** Trees in groves, rocks in clusters, open meadows and plazas between them, gaps of sky.
    - **Hierarchy.** Landmark buildings > houses > props > ground detail. Interactive things (towers, rings, targets,
      pickups) get the accent color and stand out on a calm background.
    - **Painted foliage stays calm**: leaf cards with soft shading, not lumpy green balls.
    - **Density is welcome** (more than the video), as long as the frame still reads at a glance.
    
    ## Checks before calling it done
    
    - A pair at each of the three spots where the game could pass for a frame of the video at thumbnail size.
    - The hero is readable at the size the video shows it.
    - 60 fps (or 30 fps with a clearly better image), zero console errors.
    
  • skills/video-to-game/references/playtest.md2.3 KB7de85e95499fupstream ↗
    # Play your own game
    
    A game nobody played is a scene. The agent plays it, films it, watches the film and fixes what a player would hit.
    The same takes become the trailer.
    
    ## Write a scenario
    
    `templates/play.mjs` exports `async function play(page, { seconds, t0 })`:
    
    - **Keyboard and mouse only** (`page.keyboard.down/up/press`), like a player.
    - **Read what a player sees** through `window.__game.state` (position, speed, grounded, next gap or target)
      to time jumps and turns. Never write into the game, never teleport the hero.
    - **Play well**: keep moving forward, sprint and boost when the game has them, jump at the right moment, never
      fall, never run into a wall, never walk backwards, no menu in the middle.
    - **Show every mechanic**, and bring the video's key moment to the center of the frame early in the take.
    - Use the game's own URL parameters (`?autostart`, `?seed=7`, `?x=120`) to start close to the action. A built-in
      autopilot is fine only if it plays naturally; otherwise it fights your keys.
    
    ## Record and pick
    
    ```bash
    python3 -m http.server 8000 &                                   # from the game folder
    node scripts/record_play.mjs "http://127.0.0.1:8000/?autostart&seed=7" play.mjs -o take.mp4 --seconds 18
    python3 scripts/best_window.py take.mp4 --length 10 --top 3     # liveliest windows
    python3 scripts/best_window.py take.mp4 --length 3 --cut best.mp4
    ```
    
    The recorder captures every frame with its real timestamp and encodes it at that pace, so the take shows the true
    frame rate (it prints it, with the console errors). `best_window.py` ranks windows by how much the picture changes;
    then extract a frame every 0.25 s of the winner and look at every one. Reject a window if a single frame shows a
    title or tip text, a cut-off or tiny hero, a shaking camera, a fall, a step back or a glitch.
    
    ## What to fix after watching
    
    - A dead second: the level is too empty there, or the scenario waits too long.
    - The key moment happens off-center or too late: move the trigger, adjust the camera, start closer.
    - A fall or a wall: the level design punishes the obvious path; fix the level before the scenario.
    - A frame drop: find the effect or the draw calls behind it.
    
    Up to five takes per game is normal. Keep the best one, and write in the README which take and window it is.
    
  • skills/video-to-game/references/study-the-video.md2.3 KB9660c9d1e113upstream ↗
    # Study the video
    
    You will build from frames, so make them first and look at all of them.
    
    ```bash
    python3 scripts/study_video.py gameplay.mp4 -o study/            # frames every 0.25 s, sheet.jpg, palette, cuts
    python3 scripts/study_video.py gameplay.mp4 -o study/ --every 1  # a lighter pass for a 15 s clip
    ```
    
    Open `study/sheet.jpg`, then every frame around the moments that matter. Read the prompt that generated the
    video if there is one: it names the mechanic and the world in words the frames cannot give you.
    
    ## Write the spec before the code
    
    One short file, `SPEC.md`, answering each line with what you see in a named frame:
    
    - **Camera.** Third person behind, side view, top-down, chase? Distance to the hero (in hero heights), height,
      pitch, lens (wide and distorted or long and flat), how it follows (lag, look-ahead, roll in turns).
    - **Hero.** Silhouette, proportions, colors, one or two features that make it readable at 100 px tall
      (a red scarf, a lantern, a bubble).
    - **The mechanic.** The one verb the video sells: glide off a whale's tail, pour paint that makes flowers bloom,
      flip gravity, lay rails in front of a train. Write how it starts, what it changes in the world, and what
      feedback the video shows (particles, color spreading, a sound you can imagine).
    - **Key moment.** The two seconds you would put in a trailer. The playtest must reach it.
    - **World.** Layout (a corridor, an open field, floating islands at several heights), what repeats, what is unique,
      scale cues (houses, trees, clouds).
    - **Look.** The palette from `study.json` with names (sky, haze, ground, hero accent), the light direction and
      color, the materials (painted, clay, low-poly flat, glossy), the atmosphere (fog, haze, particles), the grade
      (warm lift, teal shadows, crushed blacks).
    - **Players.** One, or local co-op if the video shows two characters working together.
    
    ## Traps
    
    - A generated video cuts and drifts. Take the camera and look from its longest steady shot, not from a cut.
    - Video models invent HUD text and fake UI. Ignore it: the game shows no text.
    - A video often shows a moment the game cannot sustain (a 2-second stunt). Make it a reward the player earns,
      not the whole loop.
    - Measure, do not remember. Pick colors from the palette or a frame, not from the word "golden".
    
  • skills/video-to-game/templates/game.html6.8 KB36fcf1dc87dbupstream ↗
    <!doctype html>
    <html lang="en">
    <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Game</title>
    <link rel="icon" href="data:,">
    <style>
      html, body { margin: 0; height: 100%; overflow: hidden; background: #000; }
      canvas { display: block; width: 100vw; height: 100vh; }
    </style>
    <script type="importmap">
    { "imports": {
      "three": "https://cdn.jsdelivr.net/npm/[email protected]/build/three.module.js",
      "three/addons/": "https://cdn.jsdelivr.net/npm/[email protected]/examples/jsm/"
    } }
    </script>
    </head>
    <body>
    <script type="module">
    // Skeleton for a game rebuilt from a gameplay video. Keep its contract:
    //  - ?autostart skips the title, ?seed=N fixes the level, so a scenario can play it the same way twice;
    //  - window.__game exposes read-only state (player position, speed, phase) for the playtest scenarios;
    //  - the look lives in one place: PALETTE, fog, lights and the post stack, all matched to the video's frames.
    import * as THREE from 'three';
    import { GLTFLoader } from 'three/addons/loaders/GLTFLoader.js';
    import { EffectComposer } from 'three/addons/postprocessing/EffectComposer.js';
    import { RenderPass } from 'three/addons/postprocessing/RenderPass.js';
    import { UnrealBloomPass } from 'three/addons/postprocessing/UnrealBloomPass.js';
    import { ShaderPass } from 'three/addons/postprocessing/ShaderPass.js';
    import { OutputPass } from 'three/addons/postprocessing/OutputPass.js';
    
    const Q = new URLSearchParams(location.search);
    let seed = +(Q.get('seed') || 1);
    const rand = () => ((seed = (seed * 16807) % 2147483647) - 1) / 2147483646;
    
    // sampled from the video with study_video.py: sky, horizon haze, ground, hero accent
    const PALETTE = { sky: 0xd8c4b5, haze: 0xc9b4a9, ground: 0x4f6b3a, accent: 0xc8402a };
    
    const renderer = new THREE.WebGLRenderer({ antialias: true, powerPreference: 'high-performance' });
    renderer.setPixelRatio(Math.min(devicePixelRatio, 2));
    renderer.setSize(innerWidth, innerHeight);
    renderer.toneMapping = THREE.ACESFilmicToneMapping;
    renderer.toneMappingExposure = 1.0;
    renderer.shadowMap.enabled = true;
    document.body.appendChild(renderer.domElement);
    
    const scene = new THREE.Scene();
    scene.background = new THREE.Color(PALETTE.sky);
    scene.fog = new THREE.Fog(PALETTE.haze, 40, 260);   // aerial perspective: far things melt into the haze
    const camera = new THREE.PerspectiveCamera(50, innerWidth / innerHeight, 0.1, 2000);
    
    const sun = new THREE.DirectionalLight(0xfff1dd, 2.4);
    sun.position.set(30, 60, 20); sun.castShadow = true;
    scene.add(sun, new THREE.HemisphereLight(PALETTE.sky, PALETTE.ground, 0.9));
    
    // post stack, part of the spec from day one: bloom, then one grade + vignette + grain pass
    const composer = new EffectComposer(renderer);
    composer.addPass(new RenderPass(scene, camera));
    composer.addPass(new UnrealBloomPass(new THREE.Vector2(innerWidth, innerHeight), 0.35, 0.6, 0.85));
    const grade = new ShaderPass({
      uniforms: { tDiffuse: { value: null }, uTime: { value: 0 }, uLift: { value: new THREE.Vector3(0.02, 0.01, 0.0) },
        uGain: { value: new THREE.Vector3(1.04, 1.0, 0.95) }, uVignette: { value: 0.35 }, uGrain: { value: 0.035 } },
      vertexShader: 'varying vec2 vUv; void main(){ vUv = uv; gl_Position = projectionMatrix * modelViewMatrix * vec4(position,1.); }',
      fragmentShader: `uniform sampler2D tDiffuse; uniform float uTime, uVignette, uGrain; uniform vec3 uLift, uGain; varying vec2 vUv;
        float h(vec2 p){ return fract(sin(dot(p, vec2(12.9898, 78.233))) * 43758.5453); }
        void main(){ vec3 c = texture2D(tDiffuse, vUv).rgb; c = c * uGain + uLift;
          c *= 1. - uVignette * smoothstep(.35, .95, length(vUv - .5) * 1.25);
          c += (h(vUv * 999. + uTime) - .5) * uGrain; gl_FragColor = vec4(c, 1.); }`,
    });
    composer.addPass(grade);
    composer.addPass(new OutputPass());
    
    // world: replace with the fal assets (GLB from image-to-3D, textures from the image model)
    const ground = new THREE.Mesh(new THREE.PlaneGeometry(600, 600), new THREE.MeshStandardMaterial({ color: PALETTE.ground, roughness: 1 }));
    ground.rotation.x = -Math.PI / 2; ground.receiveShadow = true; scene.add(ground);
    const hero = new THREE.Group();
    const body = new THREE.Mesh(new THREE.CapsuleGeometry(0.4, 1, 6, 12), new THREE.MeshStandardMaterial({ color: PALETTE.accent }));
    body.position.y = 0.9; body.castShadow = true; hero.add(body); scene.add(hero);
    for (let i = 0; i < 60; i++) {
      const r = new THREE.Mesh(new THREE.DodecahedronGeometry(1 + rand() * 2), new THREE.MeshStandardMaterial({ color: 0x8a8070, flatShading: true }));
      r.position.set((rand() - .5) * 300, 0.5, -rand() * 300); r.castShadow = true; scene.add(r);
    }
    const loader = new GLTFLoader();
    // loader.load('assets/hero.glb', g => { hero.remove(body); hero.add(g.scene); });
    
    // input: keyboard + gamepad, read once per frame
    const keys = new Set();
    addEventListener('keydown', e => { keys.add(e.code); if (e.code === 'Space') e.preventDefault(); });
    addEventListener('keyup', e => keys.delete(e.code));
    function input() {
      const pad = navigator.getGamepads?.()[0];
      const ax = pad ? pad.axes[0] : 0, ay = pad ? pad.axes[1] : 0;
      return {
        x: (keys.has('ArrowRight') || keys.has('KeyD') ? 1 : 0) - (keys.has('ArrowLeft') || keys.has('KeyA') ? 1 : 0) + (Math.abs(ax) > .2 ? ax : 0),
        z: (keys.has('ArrowDown') || keys.has('KeyS') ? 1 : 0) - (keys.has('ArrowUp') || keys.has('KeyW') ? 1 : 0) + (Math.abs(ay) > .2 ? ay : 0),
        jump: keys.has('Space') || !!pad?.buttons[0]?.pressed,
        start: keys.has('Enter') || !!pad?.buttons[9]?.pressed,
      };
    }
    
    const S = { phase: Q.has('autostart') ? 'play' : 'title', pos: hero.position, vel: new THREE.Vector3(), grounded: true, t: 0 };
    window.__game = { get state() { return { phase: S.phase, x: S.pos.x, y: S.pos.y, z: S.pos.z, speed: S.vel.length(), grounded: S.grounded, t: S.t }; } };
    
    const clock = new THREE.Clock();
    function frame() {
      const dt = Math.min(clock.getDelta(), 1 / 20);
      const i = input();
      if (S.phase === 'title' && (i.start || i.jump)) S.phase = 'play';
      if (S.phase === 'play') {
        S.t += dt;
        S.vel.x = THREE.MathUtils.damp(S.vel.x, i.x * 8, 10, dt);
        S.vel.z = THREE.MathUtils.damp(S.vel.z, i.z * 8, 10, dt);
        if (i.jump && S.grounded) { S.vel.y = 9; S.grounded = false; }
        S.vel.y -= 24 * dt;
        S.pos.addScaledVector(S.vel, dt);
        if (S.pos.y <= 0) { S.pos.y = 0; S.vel.y = 0; S.grounded = true; }
      }
      // follow camera, same framing as the video: behind, slightly above, looking ahead
      const want = new THREE.Vector3(S.pos.x, S.pos.y + 4, S.pos.z + 9);
      camera.position.lerp(want, 1 - Math.exp(-4 * dt));
      camera.lookAt(S.pos.x, S.pos.y + 1.4, S.pos.z - 6);
      grade.uniforms.uTime.value += dt;
      composer.render();
      requestAnimationFrame(frame);
    }
    addEventListener('resize', () => {
      camera.aspect = innerWidth / innerHeight; camera.updateProjectionMatrix();
      renderer.setSize(innerWidth, innerHeight); composer.setSize(innerWidth, innerHeight);
    });
    frame();
    </script>
    </body>
    </html>
    
  • skills/video-to-game/templates/play.mjs1.6 KB84381e7f0291upstream ↗
    // Playtest scenario for record_play.mjs: play the game like a good player, with the keyboard only.
    //   node scripts/record_play.mjs "http://127.0.0.1:8000/?autostart&seed=7" templates/play.mjs -o take.mp4
    //
    // Read what the player would see (window.__game.state), decide, press keys. Never write into the game.
    // Rules: always moving forward, never fall, never back into a wall, trigger every mechanic of the game
    // where it looks best, and bring the key moment of the video to the center of the frame.
    export async function play(page, { seconds, t0 }) {
      const k = page.keyboard;
      const state = () => page.evaluate(() => window.__game?.state);
      const end = t0 + seconds * 1000 - 150;
    
      if ((await state())?.phase === 'title') await k.press('Enter');
      await k.down('ArrowUp');                       // run forward the whole take
      let lastJump = 0;
      while (Date.now() < end) {
        const s = await state();
        if (!s) break;
        // jump on a rhythm while grounded, so the take shows the mechanic, not a straight walk
        if (s.grounded && Date.now() - lastJump > 900) {
          await k.down('Space'); await page.waitForTimeout(140); await k.up('Space');
          lastJump = Date.now();
        }
        // weave a little to show the world around the hero, never more than a step to a side
        const side = Math.sin((Date.now() - t0) / 700) > 0.6 ? 'ArrowRight' : Math.sin((Date.now() - t0) / 700) < -0.6 ? 'ArrowLeft' : null;
        for (const key of ['ArrowLeft', 'ArrowRight']) await (key === side ? k.down(key) : k.up(key));
        await page.waitForTimeout(50);
      }
      for (const key of ['ArrowUp', 'ArrowLeft', 'ArrowRight']) await k.up(key);
    }
    
  • README.mdupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/lovisdotio-video-to-game-skills-video-to-game.md

// REPO STATS

482 stars