Virtual Performance Modelling

Brad Garton
Music Department
Columbia University
New York, NY  10027


This paper presents a "layered" approach to the creation of artificial
performers intended to mimic human performance characteristics.  Building
upon synthesis algorithms conceived as physical models of actual dynamical
systems, this performance model employs rules ranging from constraints on
possible actions (i.e. the amount of time it takes for a human player to
shift hand positions, etc.) to rules governing harmonic and melodic
unfolding.  The layered model differs from hierarchical models in that
control within the program is not governed from a fixed level (neither
top-down nor bottom-up), but instead flows between layers as particular
ad-hoc local decisions are made.  Programs based upon this model were used
to create the pieces Rough Raga Riffs and Almost Real -- the realization of
these pieces is discussed along with possible directions for future work.


This paper discusses two related computer programs (Piece-o-Matic and
Riff-o-Matic) I have developed recently.  I should probably begin with a few
disclaimers:  although the programs are intended to model human performance
characteristics within certain musical styles, they are not intended to be a
generalized set of performance rules like those developed for "traditional
tonal music" by Friberg, et. al. (Friberg, et. al. 1991).  Nor are these
programs meant to function as algorithmic models of human composition or
improvisation in the same sense as the connectionist approach taken by Peter
Todd (Todd, 1989) or any of the cognitive models described by Otto Laske
(Laske, 1988).  Piece-o-Matic and Riff-o-Matic have no "deep knowledge" of
how music should be constructed.

Instead, these programs grew as I was working on specific pieces of music.
Any "knowledge" imbedded into the programs in the form of  rules or
procedures was coded to meet particular musical demands.  Thus these
programs were written on a very ad-hoc basis, with no guiding model existing
prior to the programming.  The performance model discussed in the next
section came from an analysis of the working programs -- it did not dictate
their development.

The Model

Piece-o-Matic and Riff-o-Matic are intended to simulate the improvisational
behavior of musicians working within the "string folk band" idiom and the
"solo rock guitar" idiom, respectively.  Riff-o-Matic (the first one
written) was originally intended to test various combinations of parameters
for Charles Sullivan's "strum" sound synthesis algorithm (Sullivan, 1990).
Soon I noticed that the simple test procedures were producing music which
was unusual and quite interesting.  I began to add more and more procedures
for making note-by-note decisions, mostly based on idiomatic playing
techniques I heard in solo rock guitar playing.  Piece-o-Matic was an
elaboration of Riff-o-Matic, with the focus being a simulation of an
ensemble of folk musicians performing on various stringed instruments.  It
is worth noting that all of my work is predicated upon the existence of a
sophisticated synthesis technique such as the "strum" algorithm.  In order
for my virtual performers to work, there must be some good virtual
instruments for them to play.

The types of rules included in the programs can be divided conceptually into
four separate "layers":

-- the physical layer:  Rules at this layer consist of decision procedures
related to the physical actions involved in producing sound on an actual
instrument.  A slight delay before each note in a strummed chord reflecting
the travel time of the pick from one string to another is an example of this
type of rule.  Timbral differences due to different note articulations (such
as an up-pick or a down-pick) are part of this level.  The sounding of
intermediate notes during a single-string glissando across a fretted guitar
neck is another example of a rule at this level.  There are also a number of
rules for choosing parameter values probabilistically within certain ranges.
This reflects the "imperfection" of human performance, plus these rules can
be used to simulate statistical tendencies, such as "pushing" certain beat
values, or flattening particular scale degrees.

-- the inflection layer:  This layer built upon the previous layer to
encompass particular stylistic articulations such as pitch-bends, vibrato,
hammer-ons, etc.  It is at this level that I feel much of the
idiomatic-specific information was coded.  Ways of articulating small groups
of notes, such as the "Van Halen" hammer-on technique in rock guitar
playing, or a double-picking effect in Irish folk music seem to be a large
part of the stylistic cues we hear.

-- the riff layer:  Stringing sets of inflections together into longer sets
of notes happens at this layer.  Inflection rules are used to guide the
intersection of pitch and rhythm templates to produce short, motif-like
musical units.  The idea behind this level came from observations of how
rock and folk guitarists learn musical gestures.  Rhythm and pitch patterns
(colloquially known as "riffs") are practiced repeatedly, and then used to
build longer musical passages.  the pitch and rhythm templates encoded at
this layer are meant to function as these motivic building blocks.

-- the shape layer:  Rules at this level are meant to establish a context
for the sequencing of riffs.  Most harmonic and melodic knowledge comes from
this layer.  In Riff-o-Matic, rules at this level established  melodic
trajectories and determined the level of rhythmic activity.  Piece-o-Matic
also included rules governing the behavior of a group of virtual performers.


Although I have presented the model as a nicely-structured hierarchy, the
actual parameter decisions are made in a very tangled manner.  It is easy to
think that the procedures are invoked logically through the layers I have
described -- the harmonic and melodic context is set by the shape layer,
which then constructs sequences of riffs, which call upon particular sets of
inflections, etc.  In actuality, however, the note-by-note decisions are
made at an extremely local level.

Whether or not a particular rule or procedure from any layer is being used
is usually determined probabilistically.  The programs work by writing cmix
scorefiles.  To write a single note, Piece-o-Matic (or Riff-o-Matic) uses a
list of instrument parameters to invoke procedures necessary for the
assignment of numerical values.  Which set of procedures is being used for
each parameter might be determined by the riff currently being played, or by
a particular inflection being chosen, or because of some physical-level
constraints.  All of these are controlled by probabilistic choice.  In other
words, it is extremely difficult to predict which rules will be used to
determine the pitch, duration or timbre of a given note because coins are
being tossed at some fundamental level in the program.  I suspect that this
is what makes the output interesting.

Some Observations

I was surprised at how few rules it took (and how "dumb" the rules were) to
produce some stylistically-passable music.  Having a powerful synthesis
algorithm with the appropriate "handles" for hooking in a set of physical
performance constraints was probably the reason for this.  Even a simple
"windowed" pitch trajectory up and down a scale (i.e. pitches are chosen
randomly within a window which slides along a list of pitches) sounded
musical when performed by a program with some rudimentary knowledge of what
was physically possible.  After the programs had reached a certain level of
complexity, it seemed that I could almost make up any rules I wanted
governing the unfolding of the riffs, and some sort of bizarre music would
result -- but the music would be stylistically coherent.

I had great fun creating pieces by interacting with these programs.  I doubt
that they contain many ideas which could be applied to the synthesis of
"music in general".  A criticism from those wishing to discover fundamental
principles underlying human musical behavior might be that these programs
are so highly idiosyncratic as to be almost piece-specific.  For me,
however, the real "kick" of doing computer music is the ability to design
and implement particular working methodologies for the creation of specific
pieces.  Designing Piece-o-Matic and Riff-o-Matic was a part of the
compositional process; I don't consider writing them to be separate from the
actual composing of the music I created with them.

I was also intrigued by the notion that with these programs I was in a sense
simulating different cultures.  As our world becomes more homogenized and
more disconnected from diverse cultural traditions, virtual recreations of
what we lose may come to represent and replace the original.  Through music,
people can vicariously participate in particular cultures and societies.  It
may be that simulations of folk idioms will meet human needs for communal
continuity by providing a nostalgic representation of communitarian ideals.
I'm not sure that this is a real good thing.


Friberg, A., Fryden, L., Bodin, L., and Sundberg, J. 1991.  "Performance
Rules for Computer-Controlled Contemporary Keyboard Music."  Computer Music
Journal 15(2): 49-55.

Laske, O. 1988. "Introduction to Cognitive Musicology."  Computer Music
Journal 12(1): 43-57.

Sullivan, C. 1990. "Extending the Karplus-Strong Algorithm to Synthesize
Electric Guitar Timbres with Distortion and Feedback."  Computer Music
Journal 14(3): 26-37.

Todd, P. 1989.  "A Connectionist Approach to Algorithmic Composition."
Computer Music Journal 13(4): 27-43.