Sound and Video (a.k.a. Jitter)

We started off by watching two short videos (Amazilia by Brian Evans and Layered Histories by Bob Gluck and Cynthia Rubin) from the recent International Computer Music Conference in Miami, where the theme was "Expanded Horizons" and part of that meant works including both video and sound components.

Then we took a look at Jitter, an extension to Max/MSP for the manipulation of matrices which uses Quicktime and OpenGL to do all kinds of neat things with video and 3D graphics. Using an Apple iSight webcam, we tracked the position of a light stick and used the x and y coordinates to make sound.

At the end of class, we briefly looked at how to go in the other direction to use audio analysis to affect a visual display in the spirit of the iTunes Visualizer. We used fiddle~ to analyze mic input and drew ovals to filter a live video feed based on the first two sine wave components of the sound.

Where to Go From Here

If you want to learn Jitter, the best place to start is the excellent Jitter manual. There are a series of tutorials similar to the ones for Max and for MSP. At the end of the manual are references to books with even more information about digital video.

Depending on what you want to do with Jitter, you'll probably want to use a video camera too. The CMC has an Apple iSight which you can use during business hours; just ask for it in the office. Eventually, you might want to buy your own firewire webcam (the Mac-only iSight is less than $150 and the cross-platform Unibrain Fire-I is even cheaper), or if you have a miniDV camcorder with a firewire / ilink port then you can use that with Jitter. Just stay away from USB webcams, they are nothing but trouble!

Applications and Examples

There are two Max/MSP/Jitter patches here. To run them, you need a demo or licensed copy of Jitter installed and you need a video camera:

week10a-examples.sit -- Zip archive with the class max/msp/Jitter patches

individual patches as text files (for Windows users)
- track-lightstick.txt
- visualize-audio.txt

Two big things to always remember with Jitter:

Jitter doesn't know anything about the frame rate of the movie / video camera / etc. it is reading from. The frame rate in Jitter depends on how often to send a bang to the top of the video processing chain. Usually, we do something like "qmetro 5" at the top of the video processing chain. It doesn't actually trigger a new frame every 5 milliseconds, it just goes as fast as the computer possibly can. The "q" in "qmetro" says that these bangs are a low priority (e.g. compared to audio) and that if it can't keep up, it should just forget about the ones that it misses.
Jitter represents video data as a 4-plane matrix. The 4 planes corresponds to A (alpha), R (red), B (blue), and G (green) components of the image. Each plane is a 2-dimensional matrix, e.g. 320 x 240 pixels.