Designing around Unspoken Rules

How does one approach interactive system design in a context where human subjects are unable to (or even decline) to give clear guidance about what they expect or prefer?

Summary

Designed a humanlike social agent capable of real-time face-to-face interaction
Built the system to emulate the social behavior of members of a specific subculture based on extensive ethnographic participant-observation (task analysis, field studies)
Evaluations of the system (usability testing, contextual interviewing) by nearly 100 human subjects indicate that the system effectively “passes for” a member of the target subculture

My Roles

Interaction designer, programmer, UX researcher

Objective

The aim was to design a virtual performer of “free improvisation,” a form of music-making wherein musicians are more or less at complete liberty to play as they wish.

The end goal is that the system “listens” (i.e., using microphones, audio signal processing, etc.) and respond (using synthesized sounds) in real time to human players just like it were a fellow human musician.

Building a machine that makes people feel like it’s an insightful, inspired creative collaborator is a daunting, if not impossible, task. The purpose of this project was to fully embrace this infeasibility. Even if one only achieves a fraction of end goals, taking such challenges only leads to new discoveries.

Challenges

The potential “users” of this system — improvisers — largely decline to state what behaviors they expect of another player, whether human or machine. For example, as a cellist named "Carl" in Chicago put it to me when I tried to understand what constitutes an ideal improviser, he refused the very idea:

"there’s no such thing as a ‘good’ improviser."

Improvisers go to great lengths to avoid any indication of what they want others to do or that they find particular ways of playing good or bad. That means that the only basis of designing a system to play like them is observation of their actual behavior.

To make matters more complicated, however, the way they play tends to sound almost as if it were completely structureless:

On top of that, improvisers are a resolutely obscure subcultural group. Where or when concerts take place or who is involved in the scene can be difficult information for outsiders to gather. This makes observation a rather difficult task.

Design

Given these constraints, this project demanded participation in and observation of music-making itself (i.e., task analysis, field studies). The first phase of this focused on locating members of this subculture themselves. After building contacts, I spent considerable time playing with other improvisers (as a woodwind player), observing interactions in live performance, and analyzing recordings.

While improvisers frequently refuse the idea of a “typical” profile of free improvisation, my ethnographic work on this practice made it clear that a handful of parameters define this form of music:

avoidance of
- pulse-based rhythm (i.e., groove, beat, etc.)
- tonal harmony (chords, melodies, etc.)
exploration of harsh or dissonant sounds (i.e., rubbing a styrofoam ball on a snare drum)

Based on these principles, I designed a virtual improviser, “Maxine.” Here's a short clip of the system playing a simulated version of an electric guitar with a trumpeter named “Laurie":

Evaluation

I tested the system with experienced improvisers in Chicago by asking them to play with it like it was another musician and talk about how it compared to a human player, a mix of contextual inquiry, usability testing, and open-ended ethnography. After initial tests in Chicago, I have also had nearly 100 improvisers in various locations (mostly Berlin and San Francisco) interact with the system and compare it to a human player. This portion of the project was funded by the Berkeley-Mellon Fellowship, the Fulbright Commission of Germany, and the Berlin Program for Advanced German and European Studies.

Most test subjects have found that the system is much like a human improviser. For example, “Wouter,” a Dutch woodwind player in Berlin, found that playing with it:

"didn’t feel radically different than playing with an unknown musician,

meeting them for the first time and playing, not knowing what to expect,

not knowing what the other person will do,

so, in that sense, 'comfortable.'"

Only a handful of performers have taken issue with the way the system avoids pulse or harmony or its focus harshness or dissonance. Thus the initial hypothesis behind the system’s design has proven valid.

But while few criticized its basic design principles, many have a lot to say about how their feelings about this system remind them of things they have had to deal with when working with other human players (see next case study).

Related Publications

2016. “Balancing Defiance and Cooperation: The Design and Human Critique of a Virtual Free Improviser.” Proceedings of the International Computer Music Conference. Utrecht.

​

​