Interactive visuals with a webcam, Kinect and OSC

Phosphene can watch the room as well as listen to it. Any webcam can track up to four moving people and push the visuals around them, with no extra hardware. Add Phosphene Link, a free helper app, and a Kinect, a phone running TouchOSC, TouchDesigner or Ableton can drive the visuals over OSC.

Open PhospheneFree, no sign-up. It runs in your browser.

Body tracking with any webcam

Turn on body tracking and Phosphene uses your webcam to follow up to four moving people. It works from motion, not a depth sensor, so a built-in laptop camera or a cheap USB webcam is enough. The video is processed on your computer and never leaves it.

How movement shapes the picture

Movement pushes. A tracked hand or the mouse shoves the image along the direction it travels and smears it, like a hand drawn through liquid, and the picture relaxes back over about a second once the movement stops. In the Live tab, Hand & mouse push has three sliders: Strength, Size and Recovery set how hard the push is, how large an area it moves and how quickly the picture settles back.

The push works on every engine, bending the picture around each person it tracks. Two engines respond especially clearly: Fluid, where movement pushes the dye, and Flow Particles, where hundreds of thousands of particles are drawn toward hands. In Shader Lab, the tracked points arrive in iInput.

Getting good tracking

• Point the camera at the people, not at the projection. Tracking follows motion, so a camera that sees the moving visuals will chase them.
• Light the people. A little light on the audience goes a long way. In a dark room where a webcam struggles, a Kinect's infrared depth camera does better.
• Expect motion, not poses. It responds to people who are moving. Someone standing perfectly still gives it little to follow.

The webcam can also be a picture source rather than a tracker: Phosphene can use the live camera image as a texture in the visuals.

Phosphene Link, the OSC bridge

A web page can't receive OSC directly, because OSC usually travels over UDP and browsers can't open UDP sockets. Phosphene Link is a small free helper that runs on your computer, listens for OSC and passes it to the Phosphene tab.

TouchOSC · NI mate · TouchDesigner · Ableton
        │  OSC over UDP, port 7400
        ▼
Phosphene Link   node phosphene-link.js
        │  WebSocket, ws://127.0.0.1:7377
        ▼
Phosphene        in Chrome or Edge

• A Node.js script that runs on macOS, Windows and Linux.
• Listens for OSC on UDP port 7400, reachable from phones and tablets on the same network.
• Talks to the browser on ws://127.0.0.1:7377, with a status page at http://127.0.0.1:7377/.
• Can also send OSC out, with beat, BPM and current engine, so lights or other software can follow Phosphene's beat clock. The destination is configurable.

Set up Phosphene Link

1. Install Node.js 18 or newer from nodejs.org.
2. Download Phosphene Link (phosphene-link.zip) and unzip it.
3. Start it. Open a terminal in the unzipped folder and run node phosphene-link.js. Leave the window open while you perform.
4. Check that it's running by opening http://127.0.0.1:7377/ in your browser.
5. Connect Phosphene. In Phosphene, open the Live tab, find Phosphene Link and press Connect.
6. Send OSC to this computer's local IP address on port 7400. Software on the same machine can use 127.0.0.1.

Use Chrome or Edge for Phosphene Link. Safari may block the browser's connection to a local helper. If your operating system asks whether Node.js may accept incoming network connections, allow it, or phones and tablets won't reach port 7400.

Kinect with NI mate

For skeleton tracking, use Delicode NI mate, which supports Kinect v1 and v2 on macOS and Windows. Set NI mate to send skeleton joints as OSC to port 7400 on the computer running Phosphene Link (127.0.0.1 if it's the same one). Phosphene picks up the hands and treats them as interaction points, so they push the picture the same way the mouse and webcam tracking do.

A Kinect is worth the extra setup in dark rooms and when you want hands rather than general motion: performers on a stage, a single dancer, or an installation where visitors reach toward the screen.

TouchOSC, TouchDesigner and Ableton

TouchOSC on a phone or tablet

Join the same Wi-Fi network as the Phosphene computer, point TouchOSC at that computer's IP address on port 7400, and send to the addresses in the reference below. A fader sending /phosphene/param/warp with a 0–1 value sweeps warp across its full range; a toggle on /phosphene/fx/vhs switches the VHS effect. If your layout already uses its own addresses, map them in the config file instead of rebuilding the layout.

TouchDesigner

Send OSC from TouchDesigner's OSC Out operators to 127.0.0.1, port 7400. That lets a TouchDesigner patch built on sensors, a timeline or your own tracking steer Phosphene's controls and interaction points.

Ableton Live

Live doesn't send OSC on its own, but Max for Live devices can, such as the OSC Send device in Ableton's free Connection Kit. Fire /phosphene/cue/next at the start of each song section, or draw automation for /phosphene/xfade, and the visuals follow the arrangement.

Lights and other software following Phosphene

Phosphene Link can send beat, BPM and the current engine as OSC to a destination you configure, so lighting software or another visual app can lock to Phosphene's beat clock instead of running its own.

OSC address reference

All addresses start with /phosphene/. Argument types are OSC's: f float, i integer, s string. Positions are 0–1, with 0,0 at the top left.

Control ids for param and paramabs

• symmetry
• complexity
• zoom
• detail
• warp
• fractal
• kaleidoscope
• speed
• rotation
• pulse
• morph
• saturation
• brightness
• contrast
• hueShift
• colorSpeed
• glow
• edge
• bloom
• vignette

Effect names for fx

• glow
• edgeGlow
• chromatic
• rgbShift
• mirror
• barrel
• droste
• liquid
• led
• duotone
• posterize
• pixelate
• hueRotate
• vhs
• scanlines
• grain

Look names for look

• clean
• vhs
• ledwall
• acid
• spiral
• poster
• ghost
• cyber

Example messages

/phosphene/param/warp   0.75        warp at 75% of its range
/phosphene/engine       9           engine 10, Fluid (0-based)
/phosphene/fx/vhs       1           VHS effect on
/phosphene/look         "ghost"     Ghost look
/phosphene/input/1      0.5 0.5 1   point 1 at the center, full strength
/phosphene/cue/next                 advance the cue list

Custom address mapping

If a controller layout already sends its own addresses, map them to Phosphene in the map section of phosphene-link.config.json rather than editing the layout:

"map": {
  "/1/fader1": "param:warp",
  "/1/fader2": "param:zoom"
}

Restart Phosphene Link after editing the file so it picks up the change.

Getting Phosphene into Resolume or MadMapper

A browser can't publish Syphon, Spout or NDI directly, so Phosphene won't appear as a source inside Resolume or MadMapper on its own. One free route is to open Phosphene's output window (O), capture that window in OBS Studio, and send it on with an NDI output plugin such as DistroAV. Resolume Arena, Resolume Avenue and MadMapper all take NDI inputs, so Phosphene can then sit in a layer or a mapping like any other source.

The simpler option is often the better one: run Phosphene full screen on the projector output and let it be the show. Its own projection mapper handles up to eight surfaces.

Open PhospheneIt's free, and there's nothing to install.