ChordWeb

Gesture × sound

I designed a hand-tracked instrument that connects movement, a visible web, and layers of sound.

Interaction design, computer vision, and creative prototyping in TouchDesigner.

Performance recording · ChordWeb in use, with the instrument’s original audio.

ChordWeb explores the space between an instrument and an interface. Moving the hands becomes something to see and hear, with a web of connecting lines making the physical relationship visible. My work brought hand tracking, visual feedback, and audio control into the same TouchDesigner project. Each part depends on the same starting point: turning a camera image into useful coordinates.

Making hand data usable

I connected browser-based MediaPipe tracking to TouchDesigner through a local WebSocket connection. The tracking results arrive as structured data, which the hand network separates into landmarks for each hand. Wrists, fingertips, and finger joints become named channels with horizontal, vertical, and depth values. That gives the rest of the instrument specific positions to work with.

The network adjusts the vertical axis to match TouchDesigner’s coordinate system and supplies default values when a hand is absent. These details matter at the boundary between tracking and interaction: a missing hand should not become a new position just because its coordinates were transformed. A normalized output keeps this preparation separate from the visual and musical mappings that follow.

Actual TouchDesigner hand-tracking network, with landmark parsing on the left, normalized hand channels in the center, and helper outputs on the right — Hand tracking · Landmark data becomes named coordinate channels. The normalized_data output feeds the instrument; additional branches provide hand geometry and gesture helpers.

See the MediaPipe connection

Actual MediaPipe component in TouchDesigner, showing its local WebSocket connection, browser component, video output, and tracking result outputs — Tracking input · A local browser supplies the camera image and MediaPipe results. Hand data and video leave this component through separate outputs.

Drawing the web

I used the hand coordinates to build the visible instrument. Separate paths select each hand’s axes, reorder the values into points, center them, and adjust their proportions. Color channels accompany those positions. A proximity operator then connects nearby points, giving the web its structure as the hands move.

I composited the rendered geometry over the camera image. Keeping the performer and the web in the same frame makes their relationship visible: the hands remain the reference for the structure they are controlling.

Actual TouchDesigner web_sim network: two hand-coordinate paths at left feed proximity geometry and a camera composite above, with audio layer routing below — The instrument network · Hand coordinates feed the proximity web above and audio controls below. The rendered lines are composited over the camera image.

Shaping the sound

For the sound, I mapped hand-derived values to the level of existing audio layers. The project receives multichannel audio through BlackHole, separates channel pairs, and routes them into parts named for a major triad and harmonic extensions. Coordinate values are analyzed and remapped into volume controls, while hand presence controls whether the audio input is active.

This makes the musical interaction a way of shaping a layered sound. The visual and audio paths share the hand data, while each gives that data a different role: position becomes connected geometry on screen and a control value in the mix. ChordWeb brings those relationships together as a performance prototype, with the camera, the web, and the sound forming one instrument.

Process evidence: the supplied TouchDesigner project, saved as Musical Playground.toe, and original network captures. The film shows the instrument in performance.