Sonic and Visual Representation of Data



Columbia University
Spring, 2022-- GU6602
Brad Garton and Nicola Hein


[note: We will be meeting on zoom for at least the first two weeks of class because of the pandemic. I will be sending out the zoom link the day before each class. If you have not received it, send me e-mail. I will check my e-mail just before the class and will send/resend the link if necessary.]


course syllabus


This class will be different from past versions. Normally this course is taught by Ben Holtzman, a renowned researcher at the Lamont-Doherty Earth Observatory. Ben knows Real Science, and does Real Stuff with it. He's on leave this term, though, so the course is being done by Brad Garton (that's me!) and Nicola Hein. We're not Real Scientists. We're musicians. But I think we'll still cover a lot of the ground that Ben would.

There will be a few differences, though. For one thing, we won't be using the Python programming language to merge the data we'll be visualizing and sonifying. Instead we'll be using the Max/MSP programming enivironment as our 'base', and RTcmix for audio with the Processing graphical programming environment for visualization. As the course progresses, we will also be employing more features of Max/MSP as well as several substantial Machine Learning/AI packages designed to work in that context.

The course is conceptually (and loosely) divided into three main 'sections'. The first is the visualization and sonification of existing data 'models', more specifically things like chaotic equations and generative systems. These produce data on-the-fly, and the present a relatively simple methodology for doing graphics and audio with the output. The second section will be what people generally imagine when they hear 'sonification' or 'visualization' of data -- we'll be tapping large datasets (Big Data!) for the graphing and auralization processes. We will use both pre-existing datasets as well as real-time internet data sources.

The last section will be concerned with higher-level feature extraction and employment. This encompasses machine learning and AI techniques, and it should be interesting to see what is possible.

In all three sections we will be showing more and more capabilities of the RTcmix and Processing languages, as well as expanding our knowledge of how best to draw upon the Max/MSP environment for our work.

If you feel totally lost or bewildered, please be sure to get in touch with us! We're here to help! Honest! Much of this isn't all that difficult, and we believe everyone can come up with an interesting project for the class. Obviously We will also be doing a fair amount of coding, BUT don't be worried about it! Everything we do will be with a musical goal in mind, and I think you'll pick up what we're doing without too much trouble. We will link all of the code, and patches for each class in the syllabus for you to download and use. I am planning to record our class sessions for you to review later if we can do it without too much trouble.


Structure

As mentioned above, class will meet on Thursdays. from 6:10 PM to about 8:00 PM. We may make assignments throughout the term intended to get you working with the development tools and also to spur class discussion. Hopefully some of the assignments may lead to your final projects for the class.

Collected here are links to the initial software and resources that we will be using in class:

As noted above, we'll also be introducing some additional packages as the term progresses.

We will also be having several guest speakers coming into the class later in the semester; watch for announcements on the syllabus. We may need to rearrange the classes as the term progresses to accommodate these guests. Plus if we're getting totally behind, then we'll modify the schedule accordingly.


Facilities

Initially the CMC studios will only be available for semi-public use because of the pandemic. This will hopefully change for the better as the situation evolves through the spring. We'll be announcing changes in the policies in class. To be honest, you can do most of the work for this class on your own machines.

Do let us know if you are having problems getting the software to work for you. In general, if you are having difficulty understanding the programming paradigms we are using or the applications and information we are covering, be sure to talk to us. We'll be happy to sit down and work through any issues with you.


Syllabus

The course syllabus is located here.

Like we said, each week we do will become a link to information relating to that class. We will place recordings of the class meetings on these pages, too, if we can work them out. We'll try to keep up with linking in class patches, examples and information, but we may fall behind. Yell at us when we do.


Contact Information

The best way to reach us is through e-mail: We won't be having official "office hours"; use e-mail to make an appointment.


Security and Access Policy

If you do use the CMC for work, you will need to abide by the Columbia regulations in place at that time for campus access. The information for this (and other Columbia/COVID-19 issues) is here:

Grades

Grades will be determined mainly by the final projects you do, and of course they will be completely subjective and based on our own personal whims -- so ya better treat us right! We will discuss projects as the semester progresses. As noted above, we may be making some assignments throughout the term and discussing these in class. It'a always fun to see what you all can do.

I say this every year, and generally people believe it (I think): by this point in your career the last thing you should be worrying about is a grade. The main thing is to find something that you'd really like to do and then do it. Please don't try to impress us with your consummate knowledge and skill, we are more impressed by people who do things. Honest!

Hope you enjoy the class!