Sound: Advanced Topics II (GR6611, Fall 2018) week 6

audio and buffers

This week we explored the exciting world of pre-digitized audio (i.e. samples, soundfiles) and even touched on a bit of real-time microphone input towards the end of the class.

Links

No real good links for this class. This is pretty basic stuff for doing digital audio in the world these days.

Class Downloads

week6-html-files.zip -- all the html/javascript files we created in class (covered below). This archive also contains the digital audio files we used in the class (like "cat1.wav" and "bradaaah1.wav", etc.) as well as the startlocalserver command enabling us to load soundfiles into the 'localhost' server for testing. The startlocalserver command is set up for unix/OSX machines, but with a little tweaking it -- or something like it -- should work under Windows, too. If not you can just do your development on a real web server page.

Here is a tour through the class work (if not listed, use your "view source" option to see the full code code, or look at the corresponding file in the archive above):

We started out encountering several problems. Using this Tone.js code:
```
<script>
	 // load the buffer with sound
	 var thebuff = new Tone.Buffer("cat1.wav");

	...
```
if we try to load a buffer with the running/debugging scheme we've been using prior to this class, i.e. simply loading the local .html file into our browser and running as a file not loaded into the browser via a web server, we get this error in our javascript console:
```
Failed to load file:///Users/brad/papes/CLASS2018/SOFTspring/week6/cat1.wav: Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https.
```
(note the "file:///" in the address -- this means we are not using a webserver).

It turns out that this is a security hole of some sort. We need to accomplish our buffer/soundfile-loading from a web-served document. We can do this by loading our code onto a handy webserver somewhere, but that gets a little cumbersome. If you'd rather just work from your local machine, it turns out that you can run a very simple (but effective) web-server by issuing the following command in a terminal window:
```
python -m SimpleHTTPServer 8000
```
You can then access this server by using the address:
To make things even simpler, I built and included a command, startlocalserver, that will do this for you simply by doule-clicking it from the OSX Finder. I am sure there is a way to do this on Windows machines, but I don't have the command in-hand (yet). (Simply close the terminal window that opens with the command to shut the localhost server down. Also, the startlocalserver needs to be in the same directory as your html files to work properly.)
(oh, one more thing: the "Tone.js" file needs to be in the same directory with all these other files for this simple server to work properly. You can still reference it elswehere, but it looks for it here. I'm not sure why this is...)
So with this knowledge, we tried running the following code, thinking it would load in a cat-meowing soundfile and allow us to play it by clicking the html button:
```
<!DOCTYPE HTML>
<html>
<head>
	 <title>buffer1</title>

	 <script src="../Tone.js"></script>
</head>
<body>

<button onclick="startbuttonfunction()">meow!</button>

<script>
	 // load the buffer with sound
	 var thebuff = new Tone.Buffer("cat1.wav");

	 // this will not work!	It takes time to load the cat!
	 var player = new Tone.Player(thebuff);
	 player.toMaster();

	 function startbuttonfunction() {
			player.start();
	 }
</script>
</body>
</html>
```
Note the comment in the code, "this will not work!" It will load properly, but when you click the button the following error will appear in the javascript console:
```
Tone.js:21000 Uncaught Error: Tone.BufferSource: buffer is either not set or not loaded.
```
Why is this? The problem is that loading a soundfile takes a certain amount of time, even if it is fairly quick. When we immediately tried to put our buffer into a Tone.Player() object and then tried to connect it, the buffer didn't fully 'exist' yet, and the Tone.Player() threw an error when we tried to trigger it.
The solution is to use a function to wait until the buffer is loaded, and then in the context of that function do the instantiating and connecting at that point. The Tone.Buffer.on() function does exactly this when given the 'load' argument:
```
<!DOCTYPE HTML>
<html>
<head>
	 <title>buffer1a</title>

	 <script src="../Tone.js"></script>
</head>
<body>

<button onclick="startbuttonfunction()">meow!</button>

<script>
	 // load the buffer with sound
	 var thebuff = new Tone.Buffer("cat1.wav");
	var player;	// declare this here so we can access it in different functions

	 // this will wait to initialize the player when all buffers have been loaded
	 Tone.Buffer.on('load', function(){
			player = new Tone.Player(thebuff);
			player.toMaster();
	 })

	 function startbuttonfunction() {
			player.start();
	 }
</script>
</body>
</html>
```
click here to run the above code

It turns out the the Tone.Player() object already has this wait built-in. We don't need to load a buffer prior to instantiating the Tone.Player() object -- we can give it a parameter with the name of the soundfile:
```
<!DOCTYPE HTML>
<html>
<head>
   <title>buffer1b</title>

   <script src="../Tone.js"></script>
</head>
<body>

<button onclick="startbuttonfunction()">meow!</button>

<script>
	var player = new Tone.Player("cat1.wav");
	player.toMaster();

   function startbuttonfunction() {
      player.start();
   }
</script>
</body>
</html>
```
click here to run the above code

(Note that the "cat1.wav" file is in the same directory as the html files we are using. I also discovered that Tone.js does not work with aiff files, so use .wav. I think .mp3 and .ogg also work well.)

We had some fun with the cat-meowing buffer, using various looping capabilities of the Tone.Player() object:
- buffer2.html -- basic looping
- buffer2a.html -- set loop start/stop points within the buffer
- buffer2b.html -- reverse loop
- buffer3.html -- this does not loop, but instead repeatedly schedules itself; each time it sets a different playback speed (pitch shift)
Notice that the repeated scheduling of playback, often at a time shorter than the buffer, 'interrupted' itself and caused a kind of 'skipping' in the sound playback. We will address this issue later.

"cat1.wav" is a fairly short soundfile, and we haven't protected against hitting the "play" button before the soundfile has loaded. To demonstrate a longer load-time, I put a longer soundfile on my personal web page and used the following code to load it:
```
var thebuff = new Tone.Buffer("http://sites.music.columbia.edu/brad/loocher/loocher441.wav");
```
As you might have guessed, this is a variant of that security problem, and it generated the following error in the javascript console:
```
Failed to load http://sites.music.columbia.edu/brad/loocher/loocher441.wav: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8000' is therefore not allowed access.
```
If I had run the html file from my web server, though, it would have worked.

With that in mind, we loaded a much longer soundfile ("Clongsrc.wav") from our localhost server so that the security problem would not manifest, and we were able to see the load time. We added a couple of "innerHTML" messages to show when the file was loading and signal when it was loaded:
- lbuf1a.html -- show when the file is loaded
That does not prevent a user from clicking the start ("drooooone") button, though. We can do that by disabling and enabling the button when the soundfile completes loading:
- lbuf1b.html -- watch the "droooooone" button
What would be even better is to show the progress of soundfile loading. This is especially important for very long (large/many) soundfile loads, as the user may think the browser has hung. It turns out that the Tone.Buffer.on() function can also report on the 'progress' of all the buffers being loaded:
```
Tone.Buffer.on('progress', function(){
	document.getElementById('loadprogress').innerHTML = calcprogress();
})
```
The calcprogress() function is included in the code. It uses some internal Tone.js constructs to find out the total progress of the soundfile downloading.
- lbuf1c.html -- this will show the download progress
Next we made a little piece using a soundfile we designed to fade-in/fade-out for each note ("Cnotesrc.wav"). We then used an array of Tone.Player()s so that we could overlap the notes without interruption -- the problem we noticed in the "buffer3.html" file above. We created an array of playback-speed values that would yield some nice, just-intonation notes and set it up to repeat indefinitely:
- multiplay.html -- play overlapping notes, just intonation
Next we took a look at the Tone.Sampler() instrument. We used the goofy "bradaaah1.wav" and "bradaaah2.wav" soundfiles for the samples. Although they were an octave apart, the Tone.Sampler() 'fills in' the missing notes:
- sampler1.html -- play a scale (shows implicit note scheduling)
- sampler2.html -- random 3-note chords (Tone.Polysynth())
Just for fun, we decided to try out the Tone.js live-input capabilities. Not all browsers will support this (Chrome does), and not all web sites will allow this (there is something in th documentation about how the web site has to be an "https" type, or one with a certificate). Ok, ok, I get why being able to randomly turn on a mic on someone's computer may be a security risk...
The good news is that this does run with the simple startlocalserver setup we've been using, so it's possible to see how a web site wih live input capabilities might work.
To check if you can use the mic on a given web page, you will need to have code like this in your javascript:
```
<script>
   // check if we can use the mic
   if (Tone.UserMedia.supported) {
      document.getElementById('micmessage').innerHTML = "the mic will work!"
   } else {
      document.getElementById('micmessage').innerHTML = "oh sadness; the mic will not work..."
   }

	...
```
and you can report it back to the web page using a document.getElementById('micmessage').innerHTML approach.
Here's a simple mic system that will report it's numeric input value every 100 milliseconds:
```
<!DOCTYPE HTML>
<html>
<head>
   <title>mic1</title>

   <script src="../Tone.js"></script>
</head>
<body>

<button onclick="openmicbuttonfunction()">open mic</button>
<br>
<button onclick="closemicbuttonfunction()">close mic</button>
<br>

<output id="meterval"></output>

<br>
<br>
<br>

<output id="micmessage"></output>

<script>
   // check if we can use the mic
   if (Tone.UserMedia.supported) {
      document.getElementById('micmessage').innerHTML = "the mic will work!"
   } else {
      document.getElementById('micmessage').innerHTML = "oh sadness; the mic will not work..."
   }

   var mic = new Tone.UserMedia();
   var meter = new Tone.Meter();

   mic.connect(meter);

   // setInterval is a javascript function that will repeat every 100 msecs
   setInterval(function() {
      document.getElementById('meterval').innerHTML = meter.getValue();
   }, 100); // this is why it is every 100 msecs

   function openmicbuttonfunction() {
      mic.open();
   }

   function closemicbuttonfunction() {
      mic.close();
   }
</script>
</body>
</html>
```
click here to run the above code

The setInterval() javascript command will repeatedly be called (every 100 milliseconds because of the "100" as the second parameter after the function definition) and will check the value of the mic.

Connecting the mic to a Tone.PingPongDelay() allowed us to hear some real-time Tone/mic interaction (note: this probably won't work from the CMC web site because it is not set up as https):
- mic2.html -- delay a mic input signal