Camera Preview & Coordinates

A BodyLink game runs in an opaque-origin sandboxed iframe and never touches the camera. The host owns the sensor: it draws its own webcam + skeleton pixels behind your transparent game frame, and streams you pose landmarks over tracking.frame. What your game controls — declaratively and at runtime — is:

1. Which coordinate space your landmarks arrive in (coordinate.mode).
2. Which body region tracking requires (coordinate.framing).
3. Where the host draws its camera preview (the camera.preview.set slot).

The manifest coordinate block is a plain declarative field; the preview slot channel and the runtime setCoordinateMode bind are gated by the camera.preview capability. Declare it in your manifest capabilities (and set the camera.preview manifest field to true if your game sends preview slots), or the channel is refused and the host default governs. No camera pixels ever cross into your iframe — the slot API carries geometry and intent only. (The single, narrow exception is the separate single-shot photo capability.)

1. Coordinate modes: full vs roi

The host stamps every game-facing tracking.frame with a coordinateSpace. Your game binds which one it gets:

The default — no declaration, no call — is full. A game that reads landmark.x as a screen fraction gets exactly that. ROI is an explicit opt-in; when you request roi before any ROI transform exists (no calibration yet), the host degrades gracefully to camera-fov-v1 — it never fabricates a transform.

Declaring the mode

Two paths, and the runtime call supersedes the manifest:

// bodylink.game.json — declarative default
        {
          "capabilities": ["tracking.frame", "camera.preview"],
          "coordinate": {
            "mode": "roi",
            "framing": "upper-body"
          }
        }
        

// Runtime — supersedes the manifest declaration.
        game.tracking.setCoordinateMode('roi', { framing: 'upper-body' });
        game.tracking.setCoordinateMode('full');   // back to camera-FOV [0,1]
        

setCoordinateMode throws on anything other than 'full' / 'roi' (and 'full-body' / 'upper-body' for framing), so a typo fails loud in development rather than silently resolving to a default.

2. The framing axis: which landmarks are required

coordinate.framing is orthogonal to the mode. Since the 0.6 contract update, framing means which landmarks are required — it no longer changes the shape of the coordinate space:

Both framings share the uniform 4:4 square framing box ({ halfWidthUnits: 2, halfHeightUnits: 2 }), so coordinates are isotropic: the same physical motion produces the same normalized delta on both axes, and a hand raised overhead stays inside the nominal box.

upper-body makes seated play first-class. A seated player with no visible hips or legs is fully supported — readiness never waits on lower-body visibility, and an upperBody calibration installs the same transform whether the player calibrates seated or standing. The two framings differ by anchor (shoulder center for upper-body, hip center for full-body) and by required landmarks — not by box shape.

Per mode: for "roi", framing sets the ROI box (1:1 for both); for "full", it is the calibration body-region hint (the stamped space stays camera-fov-v1).

All four {full, roi} × {full-body, upper-body} combinations are valid.

3. Consuming frames: onNormalizedTrackingFrame

Rather than branching on coordinateSpace yourself, let the SDK normalize. game.onNormalizedTrackingFrame(listener, { replaceFrame: true }) converts landmarks into one unified screen-normalized [0, 1] space regardless of the underlying mode (roi-centered-v2 values are mapped (v + 1) / 2 and clamped):

const game = createBodyLinkGame({ gameId: 'my-game' });

        game.onNormalizedTrackingFrame((event) => {
          const { frame, pointer } = event;

          // frame.landmarks are screen-normalized [0,1] (replaceFrame: true).
          const rightWrist = frame.landmarks?.[LANDMARKS.RIGHT_WRIST];
          if (rightWrist) {
            drawCursor(rightWrist.x * canvas.width, rightWrist.y * canvas.height);
          }

          // pointer is a ready-made [0,1] cursor (fallback chain: right wrist ->
          // left wrist -> nose, then hand points).
          if (pointer) aimAt(pointer.x, pointer.y);
        }, { replaceFrame: true });
        

One sharp edge: frame.hands is NOT converted. With replaceFrame: true the frame's landmarks, body, tracks, and pointer are all screen-normalized, but the raw frame.hands block keeps the source coordinate space. Do not mix frame.hands.right.index with normalized landmarks — derive hand positions from the converted landmarks instead (LANDMARKS.LEFT_WRIST 15, RIGHT_WRIST 16, LEFT_INDEX 19, RIGHT_INDEX 20), or read the converted copy at frame.normalizedHands.

The full normalization result — raw frame, transform metadata, per-track normalization — rides on event.normalized if you need it.

4. Directing the host preview: the slot API

game.cameraPreview declares a preview slot — a normalized rectangle the host composites its own camera pixels into. All coordinates are normalized [0, 1] in display space, top-left origin.

game.cameraPreview.show(slot);    // declare (or replace) the slot
        game.cameraPreview.update(patch); // patch the live slot (animate rect, swap roi)
        game.cameraPreview.hide();        // remove it — host returns to its opaque default
        

On the wire this is a single camera.preview.set message; hide() sends visible: false. Two established patterns cover almost every game:

Pattern A — full-bleed camera background

The slot covers the whole surface, so the host camera shows through behind your (transparent-background) game:

game.cameraPreview.show({
          rect: { x: 0, y: 0, w: 1, h: 1 },
          mirror: true,
          fit: 'cover',
        });
        

Pattern B — positioned picture-in-picture

A corner badge, optionally cropped to a source region:

game.cameraPreview.show({
          rect: { x: 0.66, y: 0.66, w: 0.32, h: 0.32 },
          roi:  { x: 0.30, y: 0.10, width: 0.40, height: 0.70 }, // optional source crop
          shape: 'circle',   // 'rect' | 'rounded' | 'circle'
          mirror: true,
        });
        

Slot fields: rect (destination), roi (optional source crop; omitted ⇒ full-frame cover-crop), mirror (default true), fit ('cover' default | 'contain'), shape (default 'rect'), and showSkeleton. The host-drawn skeleton is off by default for the play-time preview (a clean camera); pass showSkeleton: true to opt in — your game already receives landmarks and can draw its own overlay.

The slot is intent, not command: the host clamps rect to the visible surface and roi to [0, 1], and drops malformed geometry. The preview slot and the coordinate mode are independent — a full-bleed slot with roi coordinates (or vice versa) is perfectly legal.

5. Preview precedence

When multiple preview intents exist in one session, the host resolves them in this order — first match wins:

camera.preview.set slot (API)
          > legacy diagnostic camera-preview marker
            > manifest launcher.cameraPreview declaration
              > roi-mode default (a game whose resolved coordinate mode is 'roi'
                gets a player-following preview even with no declaration)
                > none (opaque frame, host default)
        

A live slot always wins; hiding it falls back down the chain. The diagnostic marker and launcher.cameraPreview paths remain supported for already-shipped games — new games should use the slot API.

Related pages

• No-Calibration Opt-In — stream pose frames with no get-ready ceremony (coordinate.calibration: "none").
• Debug Overlay — the host tracking view (F8), and the coordinate.debugOverlay manifest field.
• Camera Photo — the single-shot still-photo capability.