Potential Projects



The primary aim of our proposal is to create an environment within which new and innovative approaches to using music technology can flourish. By melding the native music educational and research talent in the Music Department with a solid support structure for true technological development, we know we can achieve a position of leadership in the evolving world of digital audio.

One of the attractive features of the Music Department for technology funding is that we have already demonstrated our capacity for taking maximum advantage of this support. As a result of previous funding initiatives and grant income, we have a number of innovative projects currently in various stages of completion. In the process of developing this proposal, we have also identified key areas for growth, especially given the resources that will be made available for us to tap (most of these projects will greatly benefit from the Senior Faculty position in audio technology we are proposing).

We have organized our example projects into pedagogical, research, and interdisciplinary categories according to what we see as the primary attribute of a particular project. However, most of the projects we describe blur these distinctions to the point of near non-existence. This taxonomy should serve as a guide primarily for presentation purposes. In particular, nearly all of the projects described have a strong pedagogical component -- particularly the research projects. Our belief in the interpenetration of "research" and "education" should be apparent; the work done as part of our research agenda both informs and is informed by our educational mission.

We have also marked several projects as FAST TRACK. These are projects that are already developed to a significant degree, and are at a stage where an immediate and focused development effort is warranted. After each project, we have also identified one or more "benefits". These are potential outputs from the project, either intellectual results form the work done or possible products or systems of use that may be developed further by the University.

Finally, this listing should be seen as a sample of current and near-future projects, but should not be considered a comprehensive listing of nascent work within the Department, nor should it be seen as a specific template for work to be done under this proposal. We offer it as an indication of existing potential here at Columbia.



Pedagogy


Music Humanities:
   Online Reserves/Virtual Tapes/Sonic Glossary   [FAST TRACK]

Professor Ian Bent in cooperation with Maurice Matiz has been developing an on-line system of definitions and "virtual tapes" for students enrolled on the Music Humanities core curriculum course. The unique aspect is the on-demand use of music over the internet to student dormitories to define terminology and extend an instructor's classroom activities.

We feel that this project represents exactly how an influx of technology hardware and expertise can work to fashion a new educational tool. A pilot project in the Department in delivering CD-quality audio into Departmental classrooms (several courses taught by Professor Bent) led to a collaboration with AcIS (Maurice Matiz) using existing RealAudio technology. The project thus far has involved nearly half the Department faculty and a number of graduate students, with technical work done by AcIS and Computer Music Center students/staff.

A number of students are already using the Sonic Glossary and Music Humanities On-Line resources in their Music Humanities classes. We anticipate that this approach has the potential for profound impact on how we teach "Music Hum".

We have designated this a FAST TRACK project because it has already reached a rather mature stage, and the time to extend this into a finished "product" of some sort is at hand.

benefits:


(click here to see the Sonic Glossary, and here for the Music Humanities On-Line Resources)


Music Theory Instruction

In teaching music theory, Professor Jonathan Kramer has developed a superior methodology that could be easily translated into an intelligent, interactive learning system. Professor Kramer's approach would not only play back a student's exercises in a variety of timbres, but would also analyze the work and suggest alternatives and improvements, point out what works musically, explain why some things sound good and others do not, etc. His system is partially rule-based, but also draws upon a large dose of musical intuition; which in this case could be be modeled convincingly. The feature that sets this project apart from others, in fact, is that it goes considerably beyond a fixed set of rules and actually models musical intuition and taste.

Professor Joseph Dubiel echoes a similar sentiment in describing a pedagogical methodology for teaching tonal music theory/composition that goes beyond rote symbol-manipulation, and uses a sophisticated internal model to help teach possibilities to students -- the students learn by exploring the capabilities and implications of the model.

benefits:



Asian Music Humanities

Many of the supporting course materials for The popular Asian Music Humanities course taught at Columbia have been converted for interactive web-based access. Professors Dieter Christensen and Aaron Fox have been working to find new ways to incorporate sound, image and text in the the Asian Music Humanities pedagogy. At present, they are at a critical juncture where improvements in technology can offer them the chance to employ new methodologies for presenting this musical literature.

benefits:
(click here [userid = "AMH", password = "Shankar"] to see the current Asian Music Humanities web resource page)


Teaching Math with Music

Six years ago Professor Brad Garton co-authored a grant with Professor Gautam Dasgupta in the Columbia University Civil Engineering Department. The premise of the grant was that the teaching of undergraduate mathematics could be accomplished by using sound and music to demonstrate mathematical concepts. Although the NSF ultimately rejected the grant, Garton and Dasgupta have since been invited to present their ideas at several international conferences (including a keynote event at the 1996 International Mathematica Conference in Finland, and a series of workshops for high-school-aged students in Europe, sponsored by the European Children's Television Center). Given the educational initiatives being announced by the Federal government, we believe it is an ideal time to resurrect this project.

benefits:


Research


Renaissance Music Modeling

For the past several years, Professors Leeman Perkins and Garton have been working to construct a database and constructive grammar for the simulation of the creative output of selected Renaissance composers. The ultimate goal is to use the finished grammar as an analytic tool to assist in identifying elements of a particular composer's style, but the project has had a significant pedagogical aspect. The development of the database and grammar has taken place in separate seminars taught by Professors Perkins and Garton, and the process of identifying stylistic characteristics of a given composer, and then coding the music in a computer-grammar accessible form has been instructive for both historical musicology and computer music students in the Department.

benefits:
(click here to see the data system currently being used for this project)


Musical Performance Style Modeling

Working with graduate student Matthew Suttor, Professor Garton has developed a suite of programs that recreate the performance characteristics of different musical cultures (i.e. Irish flute playing, or Greek gaida performance, or heavy-metal guitar soloing). Recently these programs have been used to demonstrate "new educational possibilities" for network use throughout the European Union -- the stylistic performance rules can be used to teach aspects of performance as well as cultural understanding (click here for an example of this). Professor Garton's system is also being used to model a recreation of a performer intended to "play" computer simulations of recently unearthed ancient instruments in Greece.

benefits:

RTcmix

RTcmix is a real-time sound synthesis and signal-processing language developed at Columbia University by Professor Garton. RTcmix is the audio software underlying many of the projects we describe. It is designed to take advantage of contemporary network architectures and standard internet protocols, allowing for inter-machine (and inter-institution) cooperation at a fundamental level.

benefits:
(see the RTcmix home page for extended information)


Music Cognition

Professor Fred Lerdahl has been working with graduate students in various seminars to develop models of musical understanding, drawing on his music-theoretic work on rhythmic and event hierarchies and on his theory of pitch space ("A Generative Theory of Tonal Music," Cambridge, MA: MIT Press, 1983; "Tonal Pitch Space," New York: Oxford University Press, in press). The ability to create sophisticated computer models implementing aspects of the cognitive theories is central to the continuation of this ground-breaking work. This project represents an optimal interweaving of research and pedagogy, as the exploration of how we hear music informs students in both the structure of music and in the methodology used in the investigation.

There is also a significant interdisciplinary characteristic of this work, involving research in psychology as well as computer science. We have been discussing the possibility of another high-profile appointment bridging this area with our colleagues in the Psychology Department; see both the Interdisciplinary and Future sections of this proposal. The resources we begin to establish through this proposal will make Columbia an extremely attractive institution for the finest researchers in this area.

benefits:


(click here for a brief overview of the current cognitive computer models we have done)


Analysis of Music Performance

The complex relationship of the performer to the repertoire performed is being investigated by Professor Walter Frisch. He could be greatly assisted by the development of software that would facilitate the acoustic analysis of recorded works, using advanced feature detection techniques to find correlations that would help understand what makes a "good" performance (for example). An interesting aspect of Professor Frisch's work is that it could tie directly into work also being done at the University of Southampton in the UK (the "CHARM" project). New technology can allow Columbia faculty to collaborate directly with colleagues working on related projects throughout the world; this is a perfect application of how these collaborative ventures might happen.

Professor Jonathan Kramer is doing related work in the analysis of live performance. In fact, several faculty members at Columbia are interested in bringing together the skills of computer scientists, music psychologists, music theorists, musicologists, and performers to study various aspects of music performance. The appointment of a new Director of our Music Performace Program, plus the impending formation of a performing ensemble at Columbia that will include some of the country's very best players, will provide homegrown performances worthy of careful study. We hope to develop programs at the Computer Music Center that will enable researchers to extract and analyze data about tempo, duration, expression, dynamics shaping, etc., in sophisticated musical performances. Preliminary research in this area done by Columbia faculty members Garton and Kramer was featured in a television production of The Discovery Channel.

benefits:




Interdisciplinary


Sonic Rehabilitation   [FAST TRACK]

Working with Columbia University Medical Center faculty member Claude Ghez, Professor Thanassis Rikakis and graduate student Luke DuBois have been developing a system using auditory feedback as a method to rehabilitate brain-injury patients. Over the past several years Professor Ghez has studied how the brain uses different forms of sensory input to control and coordinate our movements. Sensors in our muscles and joints (termed muscle proprioceptors) play a unique role in coordination and especially in motor learning by providing the brain with time-varying rhythmic signals. The brain uses this information to learn the complex mechanical properties of our limbs and this learning is necessary for movements to be smooth and accurate. Neurological patients with either peripheral neuropathy or damage to specific parts of the brain following stroke can no longer detect motions of their joints and develop dramatic motor impairments. A crucial finding, however, is that when the missing information is provided visually performance improves for a brief period. This indicates that brain representations are ephemeral and need to be updated at frequent intervals. Since hearing is the only sensory channel with sufficient bandwidth to represent proprioception our goal is to teach patients to utilize sound and music as a substitute for proprioception, when this is defective.

Our approach has been to use the real-time synthesis technology developed at the CMC along with musical knowledge about the salience of sonic parameters to create a system that will first provide meaningful spatio-temporal infomration about hand movement in space. Pilot studies have shown this approach to be feasible and we are ready to begin experiments in patients. In a second phase we intend to generate musical representations of joint and body movement that can be utilized for rehabilitation. We see this as a FAST TRACK project because of the major potential benefits of this work in medicine, but we do need the resources to make it happen.

benefits:


(click here for a description of the software/hardware used)


Acoustic Spatialization   [FAST TRACK]

Professor Tristan Murail has used a number of sophisticated systems for the "diffusion" of sound in various performance venues around the world. No state-of-the-art system for the complex spatial dispersion of audio exists in theaters in New York (and indeed in most of the United States). Using Prof. Murail's expertise coupled with the technological capabilities of the faculty and staff we hope to hire, we can build a truly unique "high-tech" performance space here at Columbia. Our goal is to attract significant funding for an acoustic renovation of Miller Theater, as well as an experimental space (possibly in Prentis Hall -- see the Future section of this proposal) that will build on our exceptional competence in this area. Columbia is already known as a place where this knowledge resides -- we were given a prototype 8-channel audio card for SGI computers to "beta-test" for SGI two years ago (the card is now included as standard equipment in SGI Octane and audio-configured O2 workstations).

Professor Steven Feiner in the Computer Science Department has expressed much interest in this work (see the Interdisciplinary section), as the use of an auditory spatialization system can be integrated into his own work in data presentation. We also believe that this project can become a marketable feature of Miller Theater. Many people pay additional money to attend an IMAX or THX-certified movie theater -- imagine extending the rudimentary spatialization capabilities of those venues in the context of a live performance.

This is also an obvious FAST TRACK project. We already have investment interest from several Japanese audio manufacturers, possibly in doing a collaborative project with the Tokyo College of Engineering and Keio University (Professors Akira Takaoka and Toru Iwatake are our principal contacts at present). With the growing interest in "surround sound" technology, a high-end research and performance facility is sorely needed.

benefits:



Data Sonification

A number of graduate students in the Music Department have been involved in developing new ways to visualize and manipulate digital audio data (see the sample applications developed by Doug Geers, Emily Laugesen and Johnathan Lee for examples). Because the data is intended to be ultimately realized as sound, it is easy to transform these applications into software that can "auralize" generic data from other sources. The auditory modality has been tremendously underutilized in contemporary research, but we suspect that this is about to change dramatically as new techniques for "hearing" data are developed. The fact is that certain features of large datasets can be heard more easily than they can be seen -- anything involving time-series can be potentially realized as audio with specific sonic attributes. We can establish Columbia as a ground-breaking leader in this emerging field.

benefits:


The Computer Music Center



It is important to emphasize once again that one of the more outstanding features of the Music Department with respect to technology is the Computer Music Center -- many of the projects currently underway at the CMC are not listed here as the purpose of this list is to show examples of the spectrum of work that this proposal will support. The context for most of these projects was provided by advanced work done at the CMC, and our proposal seeks to build upon and extend this tradition of music technology excellence.

Students working at the CMC also learn unique skills which give them a significant boost in competitive job markets. These students, then become a contact-resource in the industry, one that will be increasingly important as we build bridges to the corporate media production marketplace. Maintaining a high proportion of state-of-the-art software and hardware will guarantee our continued competitiveness.


There is additional development activity that will necessarily take place as a result of this proposal being funded. Much of this work won't necessarily reach the status of a real "project"; activities like creating courseware for all Departmental classes, making extended use of the renovated classrooms, or basic software development work for the Department.

Music resources will need to be digitized and placed on-line, and extended "marking up" will need to be done for optimum use of on-line materials. Professors Walter Frisch and Elaine Sisman both expressed keen interest in a system allowing playback of digitized music keyed to a digital score, and Professors Tom Payne and Timothy Taylor have both highlighted the use of digital images to support and expand the on-line music that they could access.

Other faculty members are intrigued by research possibilities brought into being by a technologized Music Department, both from the aspect of consumer and producer of music knowledge. Tim Taylor maintains a web site as an integral part of his recently-published Global Pop: World Music, World Markets, while Professors Fred Lerdahl and Jonathan Kramer both have books that could benefit from auxiliary web-based material. Tim Taylor and Aaron Fox -- among others -- see the internet as an integral part of their teaching/research, with Professor Fox anticipating a course that would focus on the development of cutting-edge web sites as a central activity for ethnomusicology (we have the talent; see The Freight Elevator Quartet and The Jazz Corner home pages for samples of Music Department student work).

The Music Department can also act to increase its web-presence with this proposal, and establish a reputation as a known "portal" for information with value. Most Columbia faculty members are already involved as editors of respected journals, and many of these journals are moving towards a networked existence (see Ethnomusicology Online, Current Musicology, and ARRAY -- the Journal of the International Computer Music Association for examples). Columbia can move to collect, collate and (through the use of technologies we develop) improve this growing world of on-line journals.

These examples are almost literally the "tip of the iceberg" for the potential of new technology use in the Music Department. To be sure, a fair amount of effort will be required for just the basic foundational work (i.e. digitizing/marking up sound, or creating an automated system to allow individual faculty members to do this transparently). We expect this work to be handled by the Technology Liaison and graduate student fellows to a certain extent, with the rest of the workload passed to existing University resources such as the Center for New Media Teaching and Learning. If the workload proves too intense, we may be forced to seek minor additional funding to pay for labor costs. There may come a time when a specific project reaches a stage of development where a significant investment in time by a faculty member may be necessary to produce a viable and innovative "product". Should this occur, we will want to investigate mechanisms for providing this time-relief, and for providing focused resources to complete the project.

Specialized digital audio work, however, will require a well-outfitted CMC. A strong CMC will also continue the support of on-going activities central to the Music Department. Many of the particular areas of concentration depend on the vitality of the Center. For example, recently-hired faculty member Tristan Murail composes by combining digital with live instrumental sounds in path-breaking ways, but his work would be impossible to do if he was unable to access the technology for this work.

We also expect the CMC to become the primary digital audio resource for the University. At present there is no place to do fine audio recording, mixing and mastering at Columbia. Our goal is to provide services that can benefit a broad range of interests within the University community.