lue>
Analysis of Affective Musical Expression With the Conductors Jacket M.I.T. Media Laboratory Perceptual Computing Section Technical Report No. 475
Appears in Proceedings of the XII Colloquium on Musical Informatics, Gorizia, Italy, September 1998
Analysis of Affective Musical Expression
With the Conductors Jacket
Teresa Marrin and Rosalind Picard
MIT Media Lab
20 Ames Street, E15-491
Cambridge, MA 02139
marrin, picard@media.mit.edu
Abstract
The Conductors Jacket is a wearable physiological
monitoring system that has been built into the clothing
of an orchestral conductor; it was designed to provide a
testbed for the study of emotional expression as it relates
to musical performance. We used the Conductors Jacket
to gather and analyze data from a professional conductor
in Boston during rehearsals of Prokofievs Romeo and
Juliet Suite No.2. Our findings indicate that several
forms of expressive communication can be measured and
detected in physiological signals. These include the use
of handedness to emphasize musical changes, the
signaling of upcoming events with sudden changes in
effort, the difference between information-bearing and
non-information-bearing gestures, the indication of
intensity and loudness with changes in muscular force,
and the use of breathing to express phrasing in the
music.
Introduction
Recent work in the domain of affect recognition and
physiological monitoring has yielded important results
on the nature and expression of human emotions[5]. For
example, several early studies have pointed to the
presence of a contagion effect whereby emotions can be
transmitted from one person to another[2]. The presence
of this effect explains why stress can be communicated
between people under various conditions; it has also
been hypothesized that other states can be transmitted
contagiously. The precise mechanism through which
this transmission occurs remains unknown, although we
suspect that gestures and body language play a big role.
One promising new direction for the study of contagious
emotional expression is in the performing arts,
particularly in music. Music has often been described as
a direct conduit for the communication of emotion; it
might be said to be an ideal carrier channel for the
transmission of affective information. Correspondingly,
musical performers might be said to modulate the
structure of musical scores in order to convey affecting
and dramatic performances. The act of performing for an
audience often requires the performer to project amplified
or enhanced emotional states, and to this end performers
often train for years to be able to effectively
and intentionally express these states. Several early and
influential studies on emotional expression discuss this
phenomenon with respect to performed classical
music[1].
We chose to look at a very specialized form of musical
performance, which is optimized for the transmission of
emotional and dramatic expression: the role of the
orchestral conductor. Conductors use a unique gestural
language that combines both technical and affective
information about a piece of music in real-time in order
to aid those who are performing it. We hypothesize that
conductors form expressive intentions for certain pieces
that they then convey by means of gestures, and that the
affective information is essentially encoded in the carrier
signal of the beat-pattern. We hypothesized that the
affective content of these signals might be decoded (as by
musicians in an orchestra) by noting the difference
between the conducted signals and the minimum
amount of information that would have been required to
execute an unexpressive (or minimally expressive)
version of the same piece.
In order to test our hypotheses, we designed and built a
system to robustly and unobtrusively sense expressive
information from conductors under professional rehearsal
conditions. This system had to be noiseless, light, not
distracting or uncomfortable to wear for long periods of
time, and able to withstand punishing conditions of
extensive muscular activity, heat, and sweat. The
resulting system, called the Conductors Jacket, is a
wearable network of physiological sensors that has been
custom designed and embedded in clothing that is fitted
to the wearer[4]. The jacket contains sensors for heart
rate, respiration, skin conductance, temperature, and
muscle tension. For muscle tension, we used four
electromyogram (EMG) sensors, one on each bicep and
tricep. These measure the small voltage created when
the muscle generates force; the voltage is proportional to
the instantaneous force output of the muscle. All of the 2
sensors are held in place by elastic bands that have been
sewn into the cloth of the jacket.
The data we collected supports three major features in
the standard conducting technique: the use of the left
hand to add emphasis and expressive information, the
turning of pages so as to not attract attention or convey
musical information, and the use of force in performing a
beat gesture to indicate the volume and articulation with
which that beat should be played. We also found some
surprising results, including several instances where the
muscles went limp right before a major event, which
suggests that the sudden absence of information has been
encoded to signal a heads-up to the players in
anticipation of an important future event.
Conductor Study
The first study using the Conductors Jacket system
took place during several weeks in February 1998, with
a professional conductor during rehearsals of a youth
orchestra in Boston. During the few minutes before each
rehearsal, the subject fitted the jacket on himself, the
sensors were adjusted, and the entire system was tested.
Then for the duration of the three-hour rehearsal, the
system was used to record numerous files of
physiological data timed with the external video camera.
Notes were taken during the data acquisition trials,
which were used to correlate and analyze the data and
video files afterwards.
Initial Data
Initial results indicate several promising features in the
data, including clear separation between the expressive
use of both hands, context-dependent variations in the
respiration signal, and enticing indicators of emotional
arousal in skin conductance. Out of more than twenty-
two files that were recorded, four have been analyzed in
detail and found to contain useful correlations between
expressive parameters and the musical score.
In general, the quality of the signals was surprisingly
very good. The four EMG signals demonstrated a
particularly high signal-to-noise ratio; that is, if there
was no observable motion, then the signal was generally
almost completely flat. This signal clarity suggests that
signal-processing algorithms could be developed to
yield good results for the automatic recognition of the
above features. Such work remains to be done; however,
we present below some preliminary findings extracted
from the data by inspection.
Among many observations of the data that have been
documented, several features were found to be
particularly noteworthy. The following section
demonstrates these features with graphical data taken
from several rehearsal segments; they have been analyzed
for their correlations with known features of traditional
conducting technique. These features indicate that our
subject:

used his left hand to provide supplementary
information and expression

suddenly withdrew gestural information when he
intended to signal the onset of a major event

showed fundamental differences in the way he made
information-carrying gestures vs. non-information
carrying gestures

modulated the force output of his muscles when
generating a beat gesture in order to indicate the
overall loudness or intensity of the music at that
beat

modulated his respiration to express the phrasing in
the music
In our first two examples, EMG signals from the right
and left biceps demonstrate how the left hand was used
to provide extra information to supplement the
information given by the right hand. In the first
example, our subject chose to modulate the meter from 4
to 2. At the moment just before he intended for the
meter to change, he reached out his left hand (which was
until that moment at his side) and reinforced the new
meter with both hands. Figure 1, shown below,
demonstrates how the previous faster meter (where only
the right hand was used) transitioned to a slower meter
as the left hand entered:
Figure 1. EMG signals from both biceps during a
metrical shift.
The top graph shows the use of the right arm; in the first
200 samples of this