What This Guide Is For

This page is a practical primer for artists and operators. It explains the sensing language, the live portal controls, and how the depth-only tracking layer fits into the larger dome stack. It is based on the current Colordome implementation, not a generic computer-vision tutorial.

To activate the full Color Dome system, use this agent prompt: /Users/jonbulette/repos/colordome-code/docs/AGENT_PROMPT_BRING_UP_FULL_COLORCUBE_SYSTEM.md

[ Orbbec Depth ] --> [ floor reference subtraction ] --> [ morphology / merge ]
                                                              |
                                                              v
                                             [ connected components / occupancy center ]
                                                              |
                                                              v
                                            [ relative height + metric unprojection ]
                                                              |
                                                              v
                                                   [ PersonObservation ]
                                                              |
                                                              v
                                                     [ PersonTrackerShell ]
                                                              |
                                                              v
                                                        [ PersonState ]
                                                       /             \
                                                      v               v
                                          [ blob adaptation ]   [ Color Cube transport ]
                                                      |               |
                                                      v               v
                                             [ occupancy blobs ] [ colorcube_signal ]
                                                      |               |
                                                      v               v
                                           [ generic CDRS behavior ] [ Color Cube bridge / mapping / OSC ]

Blobs remain the compatibility path for all existing CDRS occupancy and pattern behavior. The colorcube_signal is a separate expressive control path for Color Cube only. Under the current depth-only production path, Color Cube control is torso/centroid-driven and uses planar position plus relative height rather than hand selection.

1. Floor Reference Subtraction

The live depth frame is compared against a saved floor reference. The goal is not to find skeleton joints first; it is to find meaningful vertical structure above the floor.

2. Segmentation, Morphology, and Merge

Thresholding, cleanup, edge masking, and merge dilation turn noisy depth differences into stable person-like components. One real body should ideally become one connected component.

3. Occupancy Center and Relative Height

For each retained component, the system derives an occupancy center on the floor plane and a relative height above the floor. Those values are then unprojected into metric camera-space.

4. Tracking and PersonState

The tracker decides whether a new observation belongs to an existing person ID or whether it should spawn a new track. Each stable tracked person becomes a PersonState whose centroid semantics are WorldCoord(cam_x_m, height_m, cam_y_m).

5. Lifecycle, Presence, and Confidence

A person does not become artistically meaningful the instant a component appears. Tracks move through lifecycle states such as tentative, confirmed, and lost. Confidence describes measurement quality; presence is the smoother participation signal that keeps downstream behavior from popping abruptly.

6. Occupancy Blobs

For generic dome behavior, confirmed PersonState tracks are adapted into simplified blobs — ID, 3D position, confidence, and presence. This is the occupancy path.

7. Color Cube Transport

For Color Cube expressive control, the current production path derives control from the depth-only tracked person centroid and relative height. The result is emitted as colorcube_signal carrying control-space position and confidence. Under depth-only, torso/centroid behavior is normal.

Floor Reference and Segmentation

• Floor Reference: The saved empty-scene baseline. If this is stale or mismatched to the current mode/resolution, everything downstream degrades.
• floor_band_tolerance_m: How much variation around the floor is tolerated before the system decides something is actually above the floor.
• min_height_m: The minimum rise above the floor required for a component to count as a person-like vertical body.
• Edge Masks: Keep persistent wall, stand, or sensor-edge artifacts from entering the person segmentation path.

Morphology and Merge Controls

• merge_dilate_kernel: Controls how aggressively nearby component fragments are merged into one body.
• Morphology: Cleanup operations that remove speckle and close holes. Tune these before touching tracking persistence.
• Occupancy Center: The floor-plane center of the retained component. If this is unstable, downstream color/control behavior will drift.
• Relative Height: The main expressive signal under depth-only. Watch it for noisy spikes and under-segmentation.

Tracking Persistence and Lifecycle

• Confirm Frames: Consecutive observations required before a track becomes confirmed.
• Lost After Missed Frames: Frames a track can miss before being marked lost.
• Delete After Missed Frames: Frames a track can remain unmatched before full deletion.
• Lost Blob Hold (ms): Portal/blob hold after loss. This influences how long a downstream person can feel present after a short dropout.
• Prediction Max Missed Frames: How long the tracker keeps coasting on prediction during brief occlusion.

Association and Recovery Controls

• Association Distance (m): Approximate 3D centroid separation allowed when matching observations to existing tracks.
• Recovery Distance (m): Slightly larger match radius for recovering recently lost tracks.
• Assignment Ambiguity Margin: How conservative the tracker is when two assignment choices are nearly tied.
• Association goal: Keep one real person on one stable ID through crossings and short loss.

Fallback RGB-D Controls

The older rgbd_yolo path still exists as fallback, but it is no longer the main operator path. If that adapter is active, plausibility, keypoint, and duplicate-suppression controls matter again. They should not be the first thing artists reach for in the current production setup.

Display / Debug Toggles

• Show Components / Segmentation
• Show Track IDs
• Show Lifecycle State
• Show Confidence / Presence Diagnostics
• Show Depth-Only Masks or Debug Layers when available

These change the browser overlay only. They do not change CV behavior.

Overlay Status

• [TENTATIVE] recently appeared, not yet confirmed as a stable tracked person
• [CONFIRMED] stable tracked person that can drive occupancy and expressive behavior
• [LOST] recently missed but still held by lifecycle logic for short continuity

These are lifecycle states. They are not the same thing as raw confidence, and they are not the same thing as artistic output state.

Per-Track Debug Fields

• fsN = frames_since_seen
• hsN = hit_streak
• aN = age_frames

Example: [CONFIRMED] p_1778 fs0 hs344 a489 means a confirmed tracked person, seen this frame, with a long stable history.

Presence and Confidence

Confidence tells you how trustworthy the current measurement is. Presence is the smoother downstream participation signal used to avoid abrupt pop-in/pop-out. A person can stay lifecycle-confirmed while confidence varies, and short dropouts may still retain participation briefly.

Segmentation and Debug Layers

In the depth-only path, the most useful overlays are often the component masks, floor-subtraction result, and retained track IDs. If those look wrong, downstream control will look wrong even if the tracker itself seems healthy.

Suggested Order

1. Verify the floor reference is valid and matches the live mode/resolution.
2. Clean up segmentation with floor-band, height, edge-mask, and morphology controls.
3. Tune merge behavior so one real person becomes one stable component.
4. Stabilize IDs with tracking persistence and association controls.
5. Check lifecycle and presence behavior: people should feel stable, not brittle or sticky.

If You See X, Try Y

Current Production Semantics

• Planar movement: moving around the floor changes the control-space position used for hue and saturation.
• Relative height: standing taller, stretching upward, or otherwise increasing measured body height changes the intensity axis.
• Torso/centroid control: the system is not waiting for wrist visibility in this production path.
• Lifecycle matters: control should fade in and out with stable tracked-person participation rather than pop instantly.

What Changed From The Older Skeleton-Based Feel

Before: Color Cube work often focused on skeleton-derived anchors such as highest-hand selection.

Now: The production path on Kurt is depth-only. Planar body position drives control-space movement, and relative height drives brightness/intensity. This is simpler, more robust, and aligned with the current sensing hardware.

What To Observe

• Planar sweep: Moving through the tracked area should change control-space position smoothly, without jumps from segmentation noise.
• Height responsiveness: Stretching upward should increase brightness/intensity; crouching should reduce it.
• Lifecycle continuity: Color should not pop on/off instantly when someone briefly occludes or leaves.
• Calibration feel: If the control range feels compressed or biased to one area of the space, revisit the overhead calibration values.
• Torso anchor is normal: Do not treat lack of hand selection as a fault in the current production path.

Control-Space Semantics

The colorcube_signal carries a position already converted to Color Cube control-space. This is not the same as raw body or world coordinates. The sender converts the tracked person's depth-only position and height into control-space before transport.

• position.x / position.y — planar control coordinates (drive hue and saturation)
• position.z — height / intensity control (drives brightness)

Downstream receivers use these coordinates directly. They do not need to reinterpret camera-space axes or know about CV internals.

If You See X, Try Y

Why The Boundary Matters

Control should not depend on fragile CV internals. CV owns floor subtraction, segmentation, tracking, and PersonState. Downstream consumers receive either simplified blobs (occupancy) or control-space positions (colorcube_signal). The current production centroid semantics are WorldCoord(cam_x_m, height_m, cam_y_m); downstream code should consume the normalized transport contracts, not reinterpret raw sensing axes.

Learn The Vocabulary

The fastest way to get good at tuning is to use the system’s current terms: floor reference, occupancy center, relative height, tentative, confirmed, lost, lifecycle, presence, confidence, blob, PersonState, and colorcube_signal.

Use Side-By-Side Reference

Keep three things visible when possible: the live portal overlay, the tuning panel, and the downstream dome or visualization output. For depth-only tuning, segmentation quality and lifecycle behavior are often more important than any single downstream color symptom.

Helpful Visual Aids

• Annotated screenshots of clean components, fragmented components, confirmed tracks, and lost tracks
• One short crossing/occlusion video with notes about which lifecycle knobs mattered
• A small symptom-to-control cheat sheet printed near the operator station