Expressive Interaction Design for Music Games: Emergence and Improvisation?

Sega’s “maimai” arcade game – Photo by Lysander Yuen on Unsplash (free license)

This exploratory research challenges the lack of non-linearity in music games and sound-related art installations. I propose implementations of improvisation-focused interaction design and emergent (invention-focused) gameplay to achieve a higher level of (group) improvisation and expressivity. 🚧 Article version 0.9

Chapter 1 defines the core limitations of current music games, and a scope of potential innovations. Chapters 2, 3 and 4 give an overview of non-linear gameplay/art/audio. Chapter 5 explores the evolution of music games. Chapter 6 proposes (concrete) ideas for gameplay based musical interaction.

Key points:
– music games (incl. rhythm games)
– non-linear music (micro/meso/macro improvisation)
– non-linear game mechanics (interaction design)
– procedural (algorithmic) audio/visual generation
– adaptive (reactive, interactive) audio
– creative / artistic / expressive / improvisational interaction
– emergent gameplay (incl. physics simulation and a bit of AI)


1.0 Expressive Music Games?
1.1 Can Improvisation Make Music Games More Expressive and Creative?
1.2 Can Emergent Gameplay Make Musical Improvisation Accessible and Inclusive?

2.0 Non-Linear Game Mechanics
2.1 Basics (e.g. Path Finding)
2.3 Open Worlds
2.2 Emergent Gameplay (e.g. Physics, Chemistry)
2.3 Sandbox (e.g. Theatrical Behaviour)

3.0 Non-Linear Level Design 🚧
3.1 Fractal Art
3.2 Procedular Generation

4.0 Non-Linear Sound Design
4.1 Adaptive Music
4.2 Musical Creativity as a Byproduct
4.3 Procedurally Generated Music

5.0 Branches of Music Games
5.1 Pre-History
5.2 Karaoke
5.3 Rhythm Games: Call-and-Response
5.4 Rhythm Games: Dance Games
5.5 Rhythm Games: Other
5.6 Games Controlled by Real Instruments
5.7 Sound Toys
5.8 Playful Sequencers
5.9 Augmented Music
5.10 Interactive Music Installations

6.0 Proposed Design Challenges 🚧

Screencap from ‘Kinetic Toys’ video via article “Making Kinetic Toys” at Native Instruments

1.0 Towards Non-Linear Music Games

1.1 Is Improvisation the Key to More Expressive and Creative Music Games?

Limitations in Current Music Games
Not many music games aim to cultivate their players’ artistic expressivity: players can’t usually develop their own musical identity through performance or composition, nor can they create interactive mechanisms, such as instruments. In other words: most music games – however enjoyable they are – are designed for chronological, linear interaction – favouring competitive goals over (artistic) creativity. As a composer and designer, I’ve always been passionate about the opposite: non-linearity and improvisation. But opposites attract – the (creative) universe oscillates – therefore it is my belief that we need to learn from both non-linear “complexities” and linear “limitations” in order to design innovative music games and interactive sound experiences. The same goes

Official Promo Image for PaRappa the Rapper Remastered via

Imitation and Improvisation
In 1996, Masaya Matsuura and his company NanaOn-Sha released the first modern rhythm game: PaRappa the Rapper. The game challenges players to copy – and improve upon β€“ the rhythms of various teacher characters. This call-and-response game mechanic was visually supported by a small animated timeline giving additional cues.

In 2011, fifteen years after the release of PaRappa, Masaya Matsuura explained, in an interview for gaming website Kotaku, how music game design still faced limitations:

“Strictly speaking I do not believe that ‘Music Games’ as a genre really exists yet. We just have ‘Rhythm Games’. We can’t really grow the genre until we have some games that explore areas of music other than just rhythm. I really want us to help overcome this deficiency. I’d like to for example do something that features extremely accurate musical performance animation. As an example, if we could zoom in and see lifelike fingerwork in an animated pianist, the opportunities afforded to uncover and develop new methods of playing beyond the capabilities of humans would be a crucial step in musical evolution. The ability of ‘games’ to allow us to take part in such advances is crucial.

Unlike many later rhythm games, PaRappa embraced improvisation. To beat the teachers/opponents, players had to improvise variations of the given rhythms (more info in Chapter 5). In 2009, Masaya Matsuura stated in the Wall Street Journal (paywalled):

“For me, the improvisation is finding a new phase. Playing the instrument, creating the game ideas, or watching the characters allows me to improvise. Those kinds of things are very important”

[CC-BY] photo of DDR machines via DJ Damien’s Flickr

My favourite music game (besides Karaoke) is Dance Dance Revolution (1998, Bemani / Konami), an arcade rhythm game played by dancing (…or at least by stepping on arrows). On a macro-level, its linear approach means that players can’t create, improvise or discover alternative branches of a composition. And on a micro-level it means that the players’ actions always revolve around a rigid rhythmic grid, disconnected from the essence of expressivity, micro-improvisation: character and individuality arise from how one defies the grid – pushing, dragging and bending the notes slightly out of their “intended”, “logical” or “calculated” positions. A performance needs a little struggle, unless it aims to be robotic (which can be a struggle/art in itself, see Kraftwerk).

Neural Rhythmic Synchronisation
Research (see link below) revealed how most rhythm games (including DDR) are not (yet) suitable to improve ‘rhythmic feel’, and thus neither to gain musical skills. This is a result of the games not instructing/stimulating players to synchronise their bodies and minds to a beat. Generally, players don’t learn to feel the swing and groove – which relates back to the concept of off-grid micro-improvisation mentioned in above paragraph. A notable exception is the game Beat Sneak Bandit (Simogo, 2012).

Referenced research paper
– Music Games: Potential Application and Considerations for Rhythmic Training
BΓ©gel, Di Loreto, Seilles, Dalla Bella | Frontiers in Human Neuroscience, 2017

Beat Snake Bandit official promo image via

Scan of ‘Frequency 2’ (PS2 game by Harmonix) – via Escapist Forum review post by ‘Clerill’

Improvise to Remix
Like PaRappa, Frequency (Harmonix, 2001) rewarded skilful players with an improvisation mode β€“ but this time players also had the opportunity to improvise on a more macro level, as sequences of notes could be looped, arranged, fx-transformed, saved and shared. And on top of that, 4 people could use this remix mode simultaneously, allowing them reconstruct / deconstruct the (famous) songs included in the game. Remarkable about this way of composing is that it’s done in real-time, retaining a game-like sense of flow and challenge.

Sentris (Timbre Interactive, 2015) innovated this perspective by fusing it with a puzzle mechanic. “When you become better at Sentris, you become better at making music”, is what designer Samantha Kalman said in an interview with game magazine Giant Bomb.

One of the offical Sentris promo images – via Steam

Potentials of Emergence
But even when a game does not promote micro-improvisation (fine expressivity), meso-improvisation (e.g. note improvisation) or macro-improvisation (e.g. remixing tracks), improvisation can still emerge spontaneously

“The mission of Dance Dance Revolution is not to make people with good dancing skills enjoy dancing more but to allow people who cannot dance to learn the fun of dancing. In order to do so, we set the game rules and converted analogue dancing to digital patterns so that anyone can enjoy dancing,” says “DDR sets certain rules, and the players try to dance well according to the rules. Therefore, improvisation is a directly-opposed idea from the concept of DDR.”

This statement was made in 2009 by DDR’s producer Yoshihiko Ota stated in the Wall Street Journal (paywal). But, in the early 2000’s, a freestyle DDR movement (Youtube) *did* emerge, based on expressive, energetic improvisation, ignoring the high-difficulty game modes. I think this is a testament to DDRs versatile (level-based) design, and hints at the concept of emergent (co-creatively evolving) gameplay (more on that later). There’s great power in our bodily intuition, if we dare to let go.

GIF of “Sega Vegas – Top The Charts | 2nd Place – DDR Freestyle – New Decade” (2020) via Kanaura on Youtube

Conclusions of this section
– Both linear and non-linear elements are important to music game design.
– Micro-improvisation (realistic expression) is generally lacking, yet it has the potential to emerge through body movement, performance and intrinsic creative drive, see the ‘DDR Freestyle’ movement as a rare example.
– Remix games and loop functions (macro-improvisation methods) are relatively rare.
– Computer-assisted composing is even rarer.
– ‘SoundSelf’ is unique in offering controls through vocal micro-improvisation
– ‘Beat Snake Bandit’ is unique for relying on true rhythmic synchronisation (feeling the groove, or at least the downbeat)

Additional thoughts on interactive installation design
Similar to music games, interactive installations face expressive challenges. How could visitors/players interact artistically and leave their unique marks? How could a social artwork emerge through (group) improvisation? Can visitors be incentivised to perform in front of a public? Could player remix each others works? And so forth. I believe that innovation will arise from design exchange between entertainment arts (like video games), performance arts (music, dance, DJing, martial arts, etc.), experimental arts (like installations) and public/social space (incl. architecture), and more.

Diagram (Mindmap) of Common Music Game Mechanics

1.2 Does Game Design Hold The Key To Accessible Improvisation?

The spiritual roots of musical non-linearity and improvisation

Some music aims to tell a story, a chronological narrative that moves through a certain emotional spectrum. Other music appears to have no beginning, no ending, and no well-defined emotions, instead: trance-inducing loops and patterns that repeatedly evolve and break apart.

While linear music relates to emotional expression and/or storytelling, non-linear music reveals our primal connection with pre-emotional consciousness and organic chaos . All music possesses some of both perspectives, I believe. Take for example Bach. He was able to compose such intriguing music because he was a virtuous improviser in (probably) most of the styles he composed in. About Bach, piano professor John Mortensen writes in his book “The Pianist’s Guide to Historic Improvisation” (via The Guardian):

β€œBach could improvise fugues not because he was unique, but because almost any properly trained keyboard player in his day could. It was built into their musical thinking from the very beginning of their training.”

So, Bach – and probably many of his contemporaries – blurred the lines between improvisation and composition, and thus between linearity and perpetuity. Bach’s compositions were so powerful because they were seamless fusions of spirituality and drama – offering the listener emotional catharsis and a sense of meditation, simultaneously.

Now of course, we don’t need only Bachs, we also need music that exploits emotional bombast (let’s celebrate for example the cathartic power of ballads and arias). Equally so, we need music that dives deep into the patterns and silence of nature – like Korean sanjo music, which is rooted in Shamanic traditions (Wikipedia). Awareness of the cultural interactions between narrative (emotional) music and abstract (spiritual) music, enables us to inspire each other in unexpected and original ways, to preserve/revive/reinterpret underrepresented forms of music, to celebrate cultural diversity and interplay. What (new) types of music would work in less linear music games? But first we should ask, in what ways can we make improvisation-based music games accessible?

Screencap from graphic score animation “Bach, Contrapunctus 1, Art of Fugue” by Stephen Malinowski via Youtube. Performed by by Kimiko Ishizaka (public domain project Libre Art of the Fugue).

How much non-linearity can a music game handle?

If precise movements, skills and high scores are the appeal of music games, are players even interested in malleable compositions and intricate, free expressivity? What are the alternatives? Competitiveness is surely a driving factor behind many music games. It raises excitement. High speed, hyper precise action can be hypnotic. But, we can surely say it’s not the only way to achieve immersion and flow.

Screencap from “Cadavre for Trio” by SunFalls on Vimeo. Made with Iannix graphical sequencer.

Let’s draw a parallel. First person perspective (3D) gameplay has branched out beyond violence into mindful exploration, with games like Flower (Thatgamecompany, 2009) and Everything (David OReilly, 2017). Is there a musical counterpart to this? Yes, Tarik Barri’s Versum II (2009). This 2-player, joystick-controlled experience represents sound as objects in a 3D space, kind of like a graphic score (see this Guardian article), but in a virtual and “living” environment.

[CC-BY-SA] David OReilly – Animation-capture from zen-inspired simulation game Everything

Tarik Barri’s interactive installation “Versum II” via his website

Zen-philosopher Allan Watts said:

The physical universe is basically playful. There’s no necessity for it whatsoever. It isn’t going anywhere; that is to say, it doesn’t have a destination that it ought to arrive at. But it is best understood by analogy to music, because music as an art form is essentially playful. (…)  When Bach writes a line of melody, he doesn’t mean anything; he doesn’t try to imitate the thundering of horses hoofs, or the sound of streams, or factory whistles and the uprising of the workers. It has no social message; it’s pure playing with sound. And for that reason, among others, it’s sublime.
–Four Ways to the Center via “HD Philosophy Lectures” on Youtube

Music is inherently exploratory. Whether improvised or composed, it always moves within this YX-grid of Relaxed-Stressed VS Stoic-Chaotic:

I made this illustration inspired by a chart I saw my teacher Rens Machielse draw in class, around 2010. It struck me that the technical essence of music could be so simple. It doesn’t only apply to the state of a composition, it also applies to the state of any instrument within the composition, see these examples:

– Relaxed + Stable = a church choir singing a lengthy harmonic chord
– Relaxed + Chaotic = a hectic jazz drum solo played with very soft brushes
– Stressed + Stable = a demonic sounding organ chord droning on and on
– Stressed + Chaotic = a fast death metal guitar solo at its climax

Can we translate this spectrum into a playful interface?

A quote from the Aeon essay “How ambient music can help ease the trauma of the ICU” by psychologist Charles Fernyhough:

“Rhythmic unpredictability is only one of the things going on. By not commanding the listener’s attention, ambient music frees the mind to wander.

Improvisational freedom
I once had a discussion with someone who argued that Bach’s way of improvising fugues was very algorithmic (rule-based) and therefore “not free at all”. I found it a frustrating argument, because I didn’t understand why he couldn’t just appreciate it for what it was. And at the same time, I was unable to decide if I agreed with him or not. I think I was stuck in binary thinking, as I now see the answer (once again) requires spectral thinking.

In 1806, German writer Goethe wrote (freely translated):

Brilliance can reveal itself only within restrictions – we need laws to gain freedom.

So, don’t those “mathematical” classical rules offer freedom as they allow a near-endless amount of variation? Simpler rules, like those of for example Japanese flute (shakuhachi) music, also offer a near-endless amount of variation (with intricate expressive variations in dynamics playing a large role). Improvisation can thrive on rules (simple or complex, formulated or unspoken). The only restriction these rules produces – regardless of virtuosity and creativity – is a fixed mood or atmosphere, to some degree. Don’t these (cultural) define the intent of the music? Does a higher improvisational freedom arise from being able to change the improvisation’s mood or sense of cultural/ritual identity?

Some of Bach’s Baroque-instruments, like harpsichords, offer little means of micro-expressivity. A harpsichord can only pluck strings at a single level of intensity/velocity. This is like the polar opposite of jazz improvisation, where musicians explore dynamics from near-silence to screeching intensity. Can true improvisational freedom be found in these expressive finesses and extensive dynamic ranges? And does that perhaps relate to unrestrained body movements? Well, without dynamics, a fugue will have some level of tempo-fluctuations, some natural grooviness that will differ from person to person. Also, amazing jazz can be played on harpsichords or non-dynamic 8-bit synthesizers. The effect of micro-expressivity on improvisational freedom is again not a binary issue.

[CC BY-SA] Illustration by Njohan and Jeff Dahl via Wikipedia: Harpsichord

For reasons as such, I like to think of improvisational freedom as an XY-spectrum, with one axis showing the amount of emergence (how often rules and cultural codes are broken and something highly unexpected yet musical emerges), and the other axis showing the amount of think-ahead-time the improvisational actions are consciously or subconsciously based on. Unrestrained body movement is very subconscious – while thinking ahead about what track to DJ is a conscious decision, especially in case of a live performance process of assessing context (incl. listeners) and then making a conscious decision.

And, there’s always potential for crossovers between music of various rules, various cultures, various instruments, various art forms. Not only by playing together, but also indirectly: by carrying influences (however small or large) from one music to another. Music’s abstract, universal language is its most powerful feature. It makes music is both extremely (sub)cultural and extremely cross-cultural. This particularly applies to non-linear, spiritual music – as it’s most suitable for cross-improvisation. It’s up to us to create an open-minded (global) culture in which these direct or indirect forms of creative communication can flourish.

Zooming in on Macro & Micro Improvisation

Similar to the explorative gameplay we looked at before, group improvisation is based on mindfulness. It’s crucial for all players to be deeply aware of each other. From that state of awareness, players can respond through:

– Silence: giving space to other musicians to place the focus on their output.
– Synchronisation: complementing one or more of the currently playing elements, move the music forward in its current direction through variations or dynamic changes.
– Recontextualisation: transforming one or more of the currently playing elements by playing something around them that gives them a completely different meaning.
– Disruption: playing something that completely disturbs current atmosphere, thereby giving space for a drastic change of collective course.

These four states are in their essence not very complex to grasp, plus they are applicable to all kinds of improvisational forms, such as dance, contact improvisation, and perhaps even comedy improv (which also plays various forms of expectedness and unexpectedness).

Screencap from Nathan Dryden’s “We Rise and Fall Together – Contact Improv“, TEDxSaltLakeCity, Youtube

Music’s improvisational flexibility can be roughly explained by looking at the core structural elements of music:
– dynamics (the level of intensity/pressure, resulting in fluctuations in volume and timbre)
– harmony (e.g. stacking a tone upon an existing one)
– melody (morphing and shifting motifs and patterns)
– rhythm (changing the tempo, swing, meter, or adding a polyrhythm, etc.)

But it’s not necessary for all four of these elements to be improvised. As stated before, improvisation is compatible with fixed elements, presets and rules. Jazz improvisation, for example, is traditionally based on the harmonies and rhythms from the common jazz standards repertoire. And in case a drum circle, there is of course no melody or harmony present whatsoever – it places a strong focus on synchronisation and silence, through mimicking and call-and-response – and recontextualisation is achieved through polyrhythm, the act of stacking asymmetrical rhythms (for example person A looping KA-ta-KA-ta-KA-ta-GA-ta, while person B is looping KA-ta-ga-KA-ta-ga-KA-ta-ga-KA-ta-ga). Surely we can employ beatboxing as a game mechanic?

Thanks to music technologies – such as recording media, sequencers, samplers, loopers and algorithmic music generators – improvisation can also be done on a more macro level: by manipulating larger chunks of music, sometimes ahead of time, in anticipation of their playback.

Both improvisation and algorithmically-generated (random) elements, could bring a new level of mindfulness to music games. And social creativity could open new artistic potentials. Funnily, a game as simple as Karaoke already allows players to be expressive and social, without the need for an explicitly stated score (though many new Karaoke machines do that too).

Game-Design Based Improvisation – Instinct & Intuition

So, then we arrive at the question: is it possible to be musically expressive and/or inventive without any practical or theoretical musical training? Most people can, from a young age, mentally dissect a composition into basic elements like instruments, melody, tempo and intensity – and imitate them by amateuristically singing, humming or beatboxing. Take for example the beat of Queen’s “We will rock you”: we’re able to encode the song into a meme, a simple rhythmic gesture, which we can perform by stomping our feet and clapping our hands. The receiver of this gesture, or “sound symbol”, automatically decodes the message using their memory, thereby triggering the emotions of the song.

Music is very memorable. Arguably, visual arts do not have such an efficient ‘meme’ quality. Drawing a recognisable Mickey Mouse is generally more complex than mimicking a melody, and if it’s recognisable, does it remind us of any specific Mickey Mouse story? Language-based artworks (such as jokes and short stories) can be easily reproducible in an amateur way, but they are encapsulated in an extra level of, cultural encoding, potentially causing communicative (language) barriers.

Since music doesn’t necessarily need a language, it also needs no stories (per se). Music somehow relates to our instinctive and primal forms of communication: non-verbal vocalisation, movement and dance (chimps and bonobos use gestures too). Babies can dance to music, intuitively.

We compare a matrix of the meanings of bonobo gestures with a matrix for those of chimpanzees against 10,000 randomised iterations of matrices constrained to the original data at 4 different levels. We find that the similarity between the 2 species is much greater than would be expected by chance. Bonobos and chimpanzees share not only the physical form of the gestures but also many gesture meanings.

– ‘Bonobo and chimpanzee gestures overlap extensively in meaning‘ (2018 | PLOS Biology |  Graham , Hobaiter, Ounsley, Furuichi, Byrne)

Photo ‘black primate’ by Susanne Schwarz on Unsplash (free license)

In video games, intuition is crucial, it’s an inherent mechanism:

Bushnell’s Law: “All the best games are easy to learn and difficult to master. They should reward the first quarter and the hundredth.” –Atari founder Nolan Bushnell (Wikipedia)

Not all games have tutorials, and even tutorials are playful. So, could game design hold the key to make expressive (musician-like) improvisation accessible to anyone? And what (non-linear) game mechanics could we design for that?

From the paper ‘Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks‘ (Elfwing, S., Uchibe, E., Doya, K., 2018), using Atari 2600 games.

[CC-BY] “Computer simulation of the branching architecture of the dendrites of pyramidal neurons” by Hermann Cuntz via Wikimedia Commons

Game Design Essences β€“ Risk & Reward

One of the core philosophies of playfulness (and perhaps of life) is the so called ‘risk-reward ratio’. This dynamic isn’t always noticeable or quantifiable, it can operate on a primal psychological level – hence not all games have concretely defined risks (punishments), rewards (scores, prizes) and goals. See for example play-fighting (rough-and-tumble play, romping), which is common among humans and other animals.

Thus, we can boil ‘risk VS reward’ down to the simpler terms ‘enjoyment VS challenge’: having fun while challenging one’s mind and/or body. Then we see that the idea of play applies to a wide range of activities, like Lego-building, jamming, snorkelling, and you name it. Through challenge, play allows for a comfortable confrontation with struggle, chaos and uncertainty – it’s a window to the unknown, and thus a path to exploration, discovery, wonder and inspiration – eventually coming full circle by feeding back into the player’s motivation. Play is all about feedback loops, just like life. But what are the rules (and those loops)?

Game Design Essences β€“ What’s Unexpected? 🚧

We’re inherently curious and social creatures. Improvisation arises through intuition and spontaneity. It’s a subconscious process. Yet, as stated before, improvisation is never devoid of rules. There is an odd duality at work, or once again: a spectrum. A) Rules can be inherently intuitive or they can be culturally learned and trained. B) Rules can be either formulated or unspoken.

Improvisers – like freestyle rappers – learn an idiom (algorithms) by heart. They train these ‘unspoken rules’ methodically. When kids play-wrestle, they instinctively understand and communicate the rules of non-violence and fair power balance. Even within rule-based, competitive sports β€“ like chess and football – players improvise, as they constantly use their (trained) intuition to deal with unexpected situations.

Good games are “random” enough to never be fully predictable, like life. Therefore, it’s no surprise that the (evolutionary) essence of human playfulness can be traced back to how a mother ‘babbles’ with her baby through the simple rules of call-and-response communication. This is how she teaches the baby to communicate and deal with (unexpected) emotions [source: The Origins and Future of Playfulness by Gwen Gordon]. Play is thus a way of making sense of both the randomness and the patterns of life, the expected and the unexpected.

For gameplay to be artistically expressive – musically or otherwise β€“ its experience needs to revolve around control (flow) and exploration (wonder). Just like “easy to play, hard to master”, flow is about ease of control, exploration is about keeping the mind questioning. Control arises from a clear, intuitive understanding of one action’s, no matter whether it yields successful or unsuccessful results. Exploration arises when the game transforms the player’s creation in a somewhat understandable way β€“ or vice versa – when the player’s input/creation affects the game.

Game-Design: Emergent Gameplay is Exploration by Rule Breaking

Like improvisation, the philosophy of emergent gameplay seeks for balance between unpredictability (surprise) and predictability (clear rules / limitations / goals) – but it juxtaposes a creativity question on top of this all: how can players keep re-inventing the rules without breaking that predictable-unpredictable balance / without breaking the flow of the gameplay.

Emergent gameplay makes use of playful interpretations of complex systems close to our evolution, like physics. Within these highly dynamic virtual environments, countless variables affect each other. Yet, we have an intuitive understanding of how the the laws of nature affect the behaviour of tools, objects and characters. This also relates to (quasi-)chemistry, and simulated social interactions (which will undoubtedly improve with advancements of AI).

Gameplay ’emerges’ whenever a player manages to bend or break the rules of the game without tipping the risk-reward over to either side (which would make the game too difficult or too boring). Emergence can be the result of the player…
– designing an ingenious tactic and/or tool (eureka!)
– achieving a moment of extraordinary agility (damn!)
– discovering an accidental change of perspective (wtf?)

So to embrace emergent gameplay, game design should allow these situations to happens as often as possible. Emergence is like hacking: it aims for ingenuity and question the game designer’s authority. It explores the concept of freedom and gives players a bit of design-power. Can emergent gameplay give people the freedom of music improvisation?

The evolution of emergent (non-musical) gameplay is explored in the next chapter, in section 2.2.

Game-Design: The Potentials of Emergent Musical Gameplay 🚧

My earliest inspiration for this explorative research on ’emergent musical improvisation’ dates back 10 years. In 2010 I was part of an interdisciplinary team that designed a 10 x 10 meter, 10-player installation called Thresholder. We came up with the idea to “bind” players together in a web of floor-projected quasi-elastic lines, provoking them to exert force upon each other’s sounds (exchanging energy intricate ways). When a part of the web was stretched out too far (beyond its threshold), it would snap, with social consequences. But no musical skills were needed: every sound, including the snapping, remained coherent. Exploration and social interaction were the motivators.

The project turned out fruitful, as people remained engaged and mindful as they took minutes to explore the thresholds of the “web”. This video gives an (unfortunately slightly vague) impression of the prototype we launched at Born Digital Festival. By the way, the installation had no physical thresholds, making it really inclusive, bringing all kinds of people together.

Non-linearity, playfulness, improvisation, ingenuity and finally innovation. Two of my biggest passions revolve around this philosophy. 1) Since around 2008, I improvise electro-acoustic music with classical and traditional instrumentalists. 2) In 2008 I also started to study Composition & Sound Design for Adaptive Systems, which allowed me to explore non-linear interactivity (such as game design used for open-ended, spiritual experiences) and adaptive sound design (which we’ll explore further down). It slowly dawned on me how my fascinations for improvised music and improvised gameplay were rooted in the same spiritual origin. With this article, I hope to travel from this origin into an exciting, social, expressive future.

Thresholder photos by Joram Wolters

Around 2015, it started to dawn on me how the physics-based game mechanics of Thresholder (its web of lines inspired by elasticity, tension and fragility) showed similarities with a physics-related concept that was gaining traction in the world of popular games: emergent gameplay.

Musyc (Fingerlab, 2013) [App Store] is a well-executed physic-based improvisation instrument / playground.

Screencap from ‘Musyc’ promo video via

2.0 Non-Linearity in Games

2.1 Non-Linear Game Mechanics β€“ Path-Finding

Two important elements of emergent gameplay are: tactics (thinking, re-interpreting, designing) and agility (exploring, expressing, improvising). Both have non-linear components, or are on a linear/non-linear spectrum. We can explore tactics in a game as simple as Pac-Man (1980): players are allowed to roam through the maze in directions of their choice, thereby triggering the enemy ghosts to traverse diverse paths as well. Pac-Mans game mechanics don’t revolve only around hand-eye coordination, they also incentive players to come up with more clever tactics.

Pacman level 256, an unforeseen finale, the Kill Screen

Poetry is a way of non-lineair storytelling. Even when a poem expresses a chronological narrative, it aims to be abstract enough for subjectivity: readers have to zig-zag through their minds to come up with an interpretation of their own. Non-linearity is also about giving people freedom of interpretation. I think Pac-Man is quite poetic. And philosophical. Do the ghosts have a free will? Free will – or rather the illusion of it – arises within a context, which this case is a ‘game-AI’ called path finding. Ghosts Inky, Pinky, Blinky and Clyde, all have their own personalities that define their path-finding behaviour towards you. Clyde is direct but gets distracted, Inky is smart but gets confused, Blinky is angry and fast, and Pinky is tactical.

Graph Data Structure 6. The A* Pathfinding Algorithm” by Computer Science on Youtube

This next one made me think… what if we leave the path finding ‘paths’ visible in the game, could they become musical instruments?

Dynamic 3D pathfinding (Kynapse AI)” by Kynogon on Youtube

Here’s a flowchart I sketched, illustrating non-linearity in creative gameplay:

– Pacman Ghosts AI explained in the video:
Nuclear Fruit: How the Cold War Shaped Video Games” by Ahoy on Youtube
– “Why Pac-Man was light years ahead of its time” by GamesRadar+
– “Pacman Killscreen” by koolkid9997 on Youtube

2.2 Non-Linear Game Mechanics β€“ Emergent Gameplay (Defying the Laws of Nature)

Of course, various kinds of games combine agility with tactics, e.g. sports games and first person shooters. Yet, few games revolve specifically around emergent gameplay: the concept of giving players the ability to “design” or improvise their own unique tactics, unforeseen by the game’s designers.

The essence of emergent gameplay is psychological, it’s the player’s feeling of “doing something uniquely creative or ingenious”.

Screenshot by The Incredible Machine, a game by Jeff Tunnell Productions

Banjo Kazooie: Nuts & Bolts – Part 77” via ‘Dumb Jason’ on Youtube

Emergent gameplay can flourish when objects and environment both have highly dynamic and complex properties (variables), such as (playful) interpretations of physics and chemistry. Think of gravity, viscosity, fragility, flexibility, reactivity to water, fire, wind, and so forth. Due to advancements in AI, verbal and “physical” communication with virtual characters will also become (more) emergent.

“Plans must be devised by the player; it’s not about how clever you are [as a game designer]. (…) Games that don’t offer quote-unquote ‘real choices’ might be just fine. Some people may want a game that’s all about squeezing a virtual trigger, or moving forward like a shark, or solving a puzzle that shows more about how clever the designer is, rather than how clever they are. I’m just more interested in emergence than in scripted adventures… and I believe once players get a taste of that kind of game, it’s very hard for them to go back”

– Warren Spector interview, Gamasutra, 2013 (13 years after designing Deus Ex)

Examples of games that innovated emergent gameplay:

– The Incredible Machine [wikipedia] (puzzle game, 1993)
– Banjo Kazooie: Nuts & Bolts [wikipedia] (action-adventure, 2008)
– Scribblenauts [wikipedia] (puzzle game, 2009)
– Infinifactory [wikipedia] (puzzle game, 2015)
– Deus Ex [wikipedia] (role playing action game, 2000)
– Zelda, Breath of the Wild [wikipedia] (action-adventure, 2017)

GDC 2017 Youtube – ‘Breaking Conventions with The Legend of Zelda: Breath of the Wild’

Screencap of “E3 2016 Trailer” for “The Legend of Zelda: Breath of the Wild” via

Deux Ex (2000) screenshot via Raving Luhn

Gameplay can also be emergent in games not primarily aimed at emergent gameplay (but the emergence is then often only momentarily). See for example ‘rocket jumping’ (Wikipedia). I remember when my friends and I were exhilarated to discover this in Unreal (1998): blasting a rocket at one’s own feet while jumping, would propel the player to unforeseen (and undesigned) locations. This required agility: one had to figure out the optimal angle for the explosion, while somehow managing to retain enough health to survive.

– “Systemic Games – A Design Philosophy” (by alexbolano82 for The Artifice)
– “Scribblenauts Director Explaining Game” ( via Youtube)
– “The Rise of the Systemic Game” (Game Maker’s Toolkit on Youtube)
– “Breaking Conventions with Zelda: Breath of the Wild” (GDC on Youtube)
– “Spector: Go Emergent, Game Design Is Not All About You” (Gamasutra interview)

2.3 Non-Linear Game Mechanics – Sandbox Creativity (and Theatre)

Games like Garry’s Mod (Facepunch Studios, 2004) and Minecraft (Markus Persson, 2011) don’t even contain any specific problems that need to be solved. Instead, their sandbox design stimulates creativity and artistry by allowing players to express themselves (socially) through inventively and improvisingly playing with objects, physics, camera perspective, NPCs (non playable characters) – as a sort of on-the-fly level editing experience.

Gary’s Mod via Steam (official promo image)

Minecraft with PlasmaCraft mod via PlasmaCraft Wiki on Fandom (CC-BY-SA)

– “Gary’s Mod: Black Hole VS Destructible Town (phys_bibridgeton)” (PieNinja on Youtube)
– “101 Minecraft Build Hacks” (Grian on Youtube)

2.4 Non-Linear Game Mechanics – Open Worlds

Open world games like The Legend Of Zelda (Takashi Tezuka & Shigeru Miyamoto, 1986) and Red Dead Redemption 2 (Imran Sarwar, 2018), allow players to wander around freely and tackle quests in various (non-linear) orders.

Screencaps of Nintendo’s The Legend of Zelda, 1986

About Zelda, Gamespot said:

Never had a game so open-ended, nonlinear, and liberating been released for the mainstream market, and Nintendo of America was downright concerned that it would go right over the public’s head. Thus, it included a toll-free number that stumped players could call to have a genuine Nintendo employee talk them through any of the game’s many enigmas. Soon after the game’s release, Nintendo’s phone lines were deluged with calls, forcing it to establish gaming’s first major pay hint-service. The Nintendo Game Counselors were then born, and the rest is history.

There is however no clear consensus on the definition of an open world. I believe this is because there’s a spectrum: non-linearity and linearity can be combined in various ways and to various degrees…

Re-Cap 🚧

Non-linear storytelling: variation a game offers in terms of completing challenges in different orders, and how much effect this has on changes in the story: in how many directions can the story branch out?

Bonus play: additional variation a game offers by offering optional gameplay. For example by rewarding the players extra points for finding hidden objects or executing more difficult to perform moves.

Sandbox elements: ways in which the game allows the players to leave their artistic (incl. social, theatrical) marks.

I will address the following forms of non-linearity in the next chapters:

Adaptive audio: how the audio transforms along with the player’s behaviour and other variables in the game.

Procedural generation: the organic feeling the landscapes express through how they are generated.

NPC AI (incl. enemy AI): how do the non-playable characters interact with the players?
these techniques differ from machine learning and deep learning. Algorithms used are for example path-finding and and decision tree learning.

Here’s a diagram that roughly recaps the distinctions in non-linearity made so far:

Not a 100% accurate diagram, but food for thought.

3.0 Non-Linear Level Design

3.1 Non-Linear Level Design – Fractals

🚧 This section is under construction 🚧

Math is not me expertise (understatement!), but during this research, I was once again reminded how everything, at one point, leads back to math. So without getting too technical, let’s have a peak at the geometric art concept calle Fractal Art, as it helps us philosophise about the difference between linearity and non-linearity.

Fractal geometry reveals that some of the most austerely formal chapters of mathematics had a hidden face:  a world of beauty unsuspected until now.” – Benoit Mandelbrot (The Fractal Geometry of Nature, 1982)

The concept of fractal art is to execute a repetitive (iterative) algorithm (set of instructions) that feed some parts of its output back into its input: the instructions create a drawing and subsequently sample a part of that drawing to create the next drawing. The feedback loop within this process is recursive: it creates that typical fractal-style self-simularity (organic growth-like structuring) between elements within the structure.

Fractals grow in different want, some seem to only scale, other seem to scale and evolve. Fractals that “scale” find their strength in predictability, rhythm, linearity. Fractals that “evolve” also express a level of predictability, but they captivate us with their unpredictability, which replaces rhythm with a sense of organic shapeshifting.

3.2 Non-Linear Level Design Procedular Generativity

🚧 This section is under construction 🚧

Procedular algorithms (algorithms for controlled randomness) can be used to add a sense of variation to to creative processes (organic, realistic, or otherwise), which is aesthetically pleasing, and also reduces the need for time-costing manual labour (such as creating huge forests or underwater sea worlds). Procedural generation and can (help) shape micro, meso and macro elements of games: textures, objects, levels, behaviours, etc.

Procedural level generation was first used in the Apple II game Beneath Apple Manor (1978), which also happened to be the first commercially available role playing game (RPG). Randomisation can add an element of surprise to a game, giving it a high ‘replay value’.

Beneath Apple Manor screencaps via

Playstation 4 megahit Horizon Zero Dawn used a procedural placement system to generate rich natural environments. Level designers were able to adjust the generative system and save its output, after which they could make manual changes to the landscape, or adjust the system once again.

Screencap from video of Jaap van Muijden (Guerilla Games) explaining procedural design β€“ via Digital Dragons 2017 on Youtube

– “GPU-Based Run-Time Procedular Placement in Horizon Zero Dawn” (Lecture by Jaap van Muijden at Digital Dragons 2017 on Youtube)
– “Fractals are typically not self-similar” (3Blue1Brown on Youtube)

4.0 Non-Linear Sound Design

4.1 Non-Linear Sound Design β€“ Adaptive Music

In many contemporary high-end games, the music seamlessly adapts itself to the player’s current situation, in terms of: health, location, activities within that location (relaxed vs threatening), room acoustics (small toilet vs big church), etc. This kind of adaptive music can work in (at least) three ways:

A. Horizontally: by chopping a composition into loops that can be re-sequenced.
B. Vertically: by adding or removing certain instruments from the aforementioned loops, also known as re-orchestration.
C. Modulation.: by adding direct flexibility to musical parameters (timbre, intensity, speed, etc.) of certain instruments.

The first game to use a combined horizontal-vertical adaptive music system, with smooth transitions, was the point-and-click adventure Monkey Island 2 (1991).

Screenshot of Monkey Island 2
Monkey Island 2’s interactive sound system – via GDC lecture “What We Can Learn from “Classic” Game Music” (Youtube) by Lucasarts / iMUSE composer Peter McConnell

The sound design of VR-compatible Tetris Effect (Monstars & Resonair, 2018) pairs every single action (movement, rotation, etc.) with a corresponding sound effect, thereby blurring the lines between sound effects and music. GlitchHiker (2011) β€” a game I co-created at Global Game Jam β€” also used interactive music as SFX, as the various instrumental layers would respond to the gameplay by glitching.

Screencap from the official Tetris Effect trailer on Youtube channel Enhance Experience

Screenshot of GlitchHiker

– “Monkey Island 2 Special Edition – imuse comparison” –Youtube video by FluffyQuack
– “GlitchHiker: the game that was programmed to die” –Article by PCGamesN

4.2 Non-Linear Sound Design β€“ Musical Improvisation as a Byproduct

Otocky (SEDIC, 1987) was a very musical game for the Nintendo Famicom (NES) game console. In this side-scrolling-shoot-em-up the player was able to shoot in 8 directions, each of which represented one of the 7 notes of a (western music) minor/major scale, plus the next octave’s root note (e.g. DO-re-mi-fa-sol-la-si-DO. This meant that you would simultaneously shoot at targets and improvise melodies on top of the background music. Your melodies would automatically be rhythmically quantised: the timing of each note would be slightly nudged to align it with the rhythmic grid of the background music.

Screencap from Youtube video “Retronauts Micro Video Edition #004: Otocky”

4.3 Non-Linear Sound Design β€“ Generative, Procedular Music and A.I.

In Portal 2 (Gabe Newell, Valve, 2011), every action (such as shooting) instantaneously generated a unique bit of music.

Recently the first tech demos of an AI-driven music system for the real-time generation of adaptive music were created by Melodrive (2018).

Screencamp from Youtube video “Portal 2: Interactive Music” by ‘portal2soundvideos’

5.0 Branches of Music Games

5.1 Music Games – Pre-History

What about music games? A beautiful example is Mozart’s Dice Game from 1787, for which he wrote 272 measures to be arranged in any random order (non-linearly). In the video game world, most music games are neither non-linear nor improvisatory, we’ll get to the exceptions later. You can play scmb1’s version of Mozart’s Dice Game online at MIT Scratch.

The Dice Waltz programmed by Chris Earnshaw – via Tim’s Atari Midi World

In the next chapter, let’s check out the invention that might have kickstarted the whole music video game genre…

5.2 Karaoke

A Japanese invention from 1971. Initially it didn’t include a television screen β€” see Daisuke Inoue’s Juke 8 β€” but as karaoke spread across Asia in the 1980s, it transformed into the karaoke box, also known as KTV (karaoke television): private, sound-proof karaoke rooms with on-screen lyrics.

Although karaoke binds its players to relatively linear ruleset β€” singing the songs as accurately as possible β€” it does give players room to express themselves creatively; after all, a little improvisation or weirdness can add to the fun and social interaction. 

Photo of a Japanese Karaoke Booth by AKX_ on Flickr

Photo of a Korean Karaoke Remote Control by Catherine on Flickr

In relatively recent Japanese karaoke systems, players can score skill-points, based on various elements of singing. A spin-off of this scoring system appeared in Karaoke Joysound (Hudson Soft, Xing, 2008) for Nintendo Wii. Playstation game Singstar (London Studio, 2004) visualised the melody as it approached, and gave its players a live-indication of their accuracy.

Karaoke Joysound (Wii) – image source unknown

– “Monkey Island 2 Special Edition – imuse comparison” –Youtube video by FluffyQuack
– “GlitchHiker: the game that was programmed to die” –Article by PCGamesN

5.3 Rhythm Games – Call-and-Response

After Karaoke, Japan once again gave birth to an interactive music idea that would become hugely popular: rhythm games. Most commonly, the objective of these 1990’s games was to anticipate to rhythmic cues scrolling across the screen – thereby setting the mainstream rhythm game mechanic in stone. But in the early days, some games favoured a turn-based game mechanic, requesting the players to repeat bits of music (call-and-response).

Computer-controlled / electronic “tabletop” memory game Simon (Baer & Morrison, 1978) is probably the spiritual predecessor of call-and-response video games, as it already produced melodies. Found via ‘A History of Video Game Music‘ by Gamespot.

Screencap of ‘1992 TV Commercial for melody/memory game Simon‘ via ‘Scooper Dee’ on Youtube

The first real rhythm (video) game, PaRappa the Rapper (NanaOn-Sha, 1996) for Playstation 1, did show a stream (sequence) of notes, but mainly challenged the players to repeat the teacher’s rhythms; to copy in a call-and-response way. Another thing that made PaRappa’s game mechanic quite unique, was its emphasis on improvisation: by playing extra notes in the rhythmic grid, the player could “beat” the teacher and get a chance to perform on the “cool mode” freestyle stage.

‘PaRappa The Rapper 2’ promo image via

Call and response – Screencap from “Let’s All Play Our Drum Game” video by Kalani Das on Youtube

Jazz musician Bobby McFerrin played a melodic game (Youtube video) with his audience at the World Science Festival. This shows our intuitive (instinctive?) understanding of the pentatonic (ancient 5-note) musical scale. Could we translate this understanding to a video game?

Screencap from video “Bobby McFerrin Demonstrates the Power of the Pentatonic Scale” via World Science Festival on Youtube

5.4 Rhythm Games – Dance

In 1997, just one year after PaRappa came out, the rhythm virus spread from the Playstation to the Arcade Hall, thanks to Konami’s DJ simulator Beatmania (Yuichiro Sagawa, 1997). Players now had to bash buttons in sync with scrolling notes. 1998 marked another revolution: Dance Dance Revolution (DDR), the first rhythm game with a dance controller.

[CC-BY-SA] photo “Keyboard and turntable controls for beatmania” by Chardish on

The game didn’t award its players with any extra points for improvising, yet that didn’t stop a ‘DDR freestyle’ scene from emerging. Freestylers are not focused on the game’s expert levels or extreme scores; they instead prefer to come up with new choreographies and/or improvisations. Thus DDR reached cult status… in my book at least.

Screencap from Youtube video “Best DDR Freestyles” on channel N64Moments

Another arcade example is Pop’n Music (Bemani/Konami, 1998).

[CC-BY-SA] ‘pop’n music’ – photo by η©Ίη·΄ via

Later games, like Dance Central (Harmonix, 2010) and Dance Evolution / Dance Masters (Bemani/Konami, 2010), challenged the players to dance with their full bodies. Hand and foot gestures appear on the screen right before they have to be executed. These games use the Kinect 3D-sensor for full body tracking. DANCERUSH (Bemani/Konami, 2008) continued in that tradition.

Screencap of “Crazy Dancerush Stardom Shuffle Lv.9 – Matching Dance Step” by KAITEN on Youtube
Screencap of “Dance Evolution Arcade – Din Don Dan” by Max X on Youtube
Screencap of “Dance Central 2 – Milkshake – Hard 100% – Kelis” by Latty2cute on Youtube

🚧 Still a lot of games (Japanese arcade games) missing 🚧

5.5 Rhythm Games – Other

Many other rhythm games would follow: rail-shooter Rez (Tetsuya Mizuguchi, 2001), “traditional” Japanese drum game Taiko no Tatsujin (Bandai Namco, 2001), puzzle game Lumines (Tetsuya Mizuguchi, 2004), Guitar Hero (Harmonix, 2005), countless mobile games (see this Engadget article), motion controlled games like Wii Music (Kazumi Totaka, 2009) which included a Custom Jam mode for improv, “piano” game Chunithm (2015) and VR games like Beat Saber (2018).

Screencap from Youtube video “Wii Music – Part 5 – Custom Jam Zelda and Mario Themes” by ‘Dumb Jason’

One of the official promo of Rez Infinite on Steam

Screencap from Youtube video “PERFECT MODDED BEAT SABER: Kaskade – Never Sleep Alone” by Ruirize

Frequency (2001), for Playstation 2, featured a remix mode that allowed up to 4 players to improvise collaborative compositions online.

Screencap of Youtube video “Frequency Tutorial Remix Mode” by IrrevokableBrazzaVille

Vib-Ribbon (NanaOn-Sha, 1999) was the first rhythm game to feature an audio-based level generator; it could create new levels based on any audio CD you would insert. 

[CC-BY-SA] ‘Vib-Ribbon’ in-game screenshot by El Shorpo on

5.6 Music Games controlled by Real Instruments

In recent years we’ve seen innovative games that can be controlled by improvising on real (acoustic or electronic) instruments, like Cello Fortress (Joost van Dongen, 2012): an action game in which a cellist defends a fortress from 4 gamepad-players who try to destroy it. And SoundSelf (2012): a psychedelic, therapeutic VR-compatible experience controlled by chanting (singing). 

Cello Fortress promo photo via Joost van Dongen’s website

Youtuber ‘kung foo man’ playing SoundSelf (1/2)
Youtuber ‘kung foo man’ playing SoundSelf (2/2)

5.7 Music Education Games

Still focused on hitting the right note on the right cue, but somewhat less rhythm oriented are music-education games, as they allow you to practice on real instruments. Examples are Rocksmith  (Ubisoft, 2011 – remastered 2014) for electric guitar and Synthesia (2006) for MIDI keyboard. Robert Nichols hacked Rocksmith to make it work with violin as well.

Official promo image for ‘Rocksmith 2014 Edition – Remastered’ via STEAM

Screencap of Synthesia promo video via their website

5.7 Sound Toys

Sound toys are arguably similar to sandbox games (having no fixed goals). Let’s look at various examples.

– SimTunes (Toshio Iwai, 1996)

Screencap of Youtuber Levi Niha playing SimTunes

– Electroplankton (indieszero & Nintendo, 2005)

Screenshot of Elektroplankton via Gamespot‘s review video

– BjΓΆrk’s Biophilia app suite (Scott Snibbe, M/M Paris, and others, 2011)

Biophilia promo image via presskit at

– Physics-based Musyc (Fingerlab, 2013)

Screencap from Youtube video ‘Musyc app time machine’ by Bernard Miner Jr.

– Remix/jam app Ninja Jamm (Ninja Tune, 2013) [App Store] [Playstore]
– ….recently updated(?) to Jamm Pro (Ninja Tune, 2020) for iPad [App Store]

Screencap from Youtube video “#Jamm Pro for iOS from #NinjaTunes review” by Nu-Trix The Synth Guy

– And the more graphical remix/jam app Playground [App Store] (HLO S.A, 2017)

Screencap from Youtube video “Herrmutt Lobby TEDx Liege – PLAYGROUND iOS APP”

– Table-top inspired ReacTable (2005), which can be scaled large enough for multi-player improvisation.

[CC-BY-SA] ‘Reactable’ photo by Daniel Williams via Wikipedia

– SimTunes composition ‘Abblahh’ by Dogman15 on Youtube
– Performance of ‘Electroplankton Opus 1-3 (Nintendo DS x 5)’ by AABattery on Youtube
– Video review of Musyc by Jarrid Pennypacker on Youtube
– bjΓΆrk’s official promo video of the Biophilia app suite (Youtube)
– Playground app promo video by Herrmutt Lobby on Youtube
– ‘The Making of Ninja Jamm’ video on the Ninja Tunes Youtube channel
– ‘ReacTable Trance live performance #2’ video by ReacTj’s Youtube channel

5.8 Playful Sequencers

Quite similar to the sound toys category, but a little less based on direct control, and a little more based on patterns and sequences….

– Sentris (Samantha Kalman / Timbre Interactive, 2014) is possibly the first real / advanced hybrid of a puzzle game and a sequencer (software to write and produce music with):

Screencap of ‘IGN plays Sentris‘ via Youtube

Non-game influenced sequencers have also become more playful and non-linear

– Stochastic sequencer SECTOR (Kymatica, 2014):

Promo image of the ‘SECTOR’ iPad App – taken from its website

Multi-playhead sequencer Fugue Machine (Alexander Naut, 2015).

Screenshot of ‘Fugue Machine’ iPad app β€“ taken from its website

Cellular automata inspired sequencer JR Hexatone (Ilya Plavunov, 2009):

Screencap from Jordan Rudess demoing the ‘JR Hexatone app – via Youtube

John Conway’s Game of Life interpreted by Marin Vlastelica PogančiΔ‡

Playful yet feature-packed Max for Live device New Path (Studio The Zooo / Jerke van den Braak, 2017):

Screencap from The Zooo Studio’s trailer of ‘New Path’ – via and Youtube

iPad app Gestrument (2018), a customisable interactive music engine/instrument that supports eye tracking, gestural movements, audio tracking, and more:

Screencap from ‘Gestrument’ demo video by ‘The Sound Test Room’ on Youtube

Neural Beatbox (Qosmo, Ltd., web-version 2020) uses AI to generate beats from sounds the user records.

Screencap from ‘Neural Beatbox‘ by Qosmo, Ltd. on Youtube

– SECTOR tutorial on Youtube channel The Sound Test Room
– Sentris extended gameplay plus interview on Giant Bomb
– Jordan Rudess introducing JR Hexatone on his Youtube channel
– FUGUE Machine tutorial on Youtube channel The Sound Test Room
– Gestrument Pro demo on Youtube channel The Sound Test Room
– New Path 1.1 promo video on The Zooo’s Youtube channel

5.9 Augmented Music

Reality Jockey Ltd. pioneered augmented music in 2008 with their smartphone app RJDJ. It picks up microphone and sensor input, to generate music that reacts and adapts to the listener’s environment. RJDJ’s interactive compositions (scenes) are programmed in the visual programming language Pure Data. RJDJ is not on the market anymore, but its successor ‘Hear‘ is available on the App Store.

Screencap from Youtube video “RjDj – The mind twisting hearing sensation”

Check out this video of the amazing RJDJ scene “WorldQuantizer” by Roman Haefeli. It samples sounds and creates rhythmic loops in real-time. It’s likely that we will see the market for artistic augmented reality apps grow when mainstream AR products like Apple Glasses will hit the mainstream.

5.10 Interactive Musical Installations

When it comes to large (public) interactive installations, there are many examples feature interactive music or sound, like the simple but engaging Piano Stairs (VW & DDB, 2009), the body tracking avant-garde noises of Device Unknown (Sem Shimla, 2018), the immersive, brainwave (EEG) controlled lasers and sounds of On Your Wavelength (Marcus Lyall, 2015).

Screencap from Youtube video of Piano Staircase (2009) by Volkswagen and DDB Stockholm

Screencap from Youtube video of ‘Device Unknown’ (2018) by Sem Shimla

Screencap from promo video (Vimeo) of Marcus Lyall’s ‘On Your Wavelength’ (2015)

As an example of visual improvisation: in the floor-projected installation Healing Pool (2008), people can collectively manipulate the (regenerative) visual world.

Screencap from video of ‘Healing Pool’ (2008) by Brian Knep, via his Youtube channel

But it’s rare to find an installation that combines collaborative interaction (social game design) + music/sound improvisation. Something that comes close is The Music Box Village (2018): a collection of architectural sound toys that can be used by artists and audience to jam with otherworldly (textural) sounds. 

“The Music Box Village is created and built by New Orleans Airlift, an artist-driven non-profit organization, as one of a range of large-scale community art projects. (……) a place where play, imagination, experimentation, collaboration, community and hard work come together as artist-made, interactive β€œmusical houses.”” –

6.0 Proposed Challenges 🚧

From the information gathered by this blog post, I propose the following challenges for designing (non-linear) music games:

– Melody/harmony games inspired by rhythm games. Moving away from the on-screen rhythmic cues, towards actively listening (perhaps to randomly generated sequences), challenging players to respond with suitable notes and timings.

– Rhythm games with a stronger focus on synchronising with grooves, aiming to instil neural rhythmic synchronisation (rhythmic “feel”). Research has shown that almost none of the current rhythm games achieve this. Perhaps it can be achieved through continuous engagement with repetitive patterns.

– Music games that are equally as suitable for beginners, hardcore (high-score) players, and a third group: freestylers (DDR is a successful example of this).

– Edutainment games that teach players how musical elements like rhythm, harmony, melody actually work; so they can use this to become more musical and learn how to improvise.

– Masaya Matsuura’s idea: exploring post-human virtuosity, for example by giving supernatural influence over a virtual pianist. Perhaps related to inventing ways to conduct expressively, in stead of performing every single note. A form of cyborgian micro-conducting is present in Wii Music Jam (in the shape of hyper fast tremolo picking, for example).

– Music games that stimulate the players to improvise more, for example through an intelligent system judges how well players improvise (this is already implemented in Parappa 2).

– Music games in which computers improvise along with the player, complementing the player. Together, or turn-based (with a rhythmic clock).

Apple’s ‘Drummer‘ in GarageBand for iPhone

– Music games with endless variations of music by implementing procedural (generative/algorithmic) music systems.

– Music games revolving around emergent gameplay, by for example letting players build their own instruments by using physics-based playgrounds. Or by navigating through unrelated genres in crazy but coherent ways.

– Music games to improve social contact (incl. games for the elderly, like the Tovertafel).

– Music games crossed-over with sports games. And rhythm games connected to fitness, for example by using weights as controllers.

– Music games embracing unrestrained body movement, about losing control in movement, chaos theories. Primal playful behaviour. See also: contact improvisation.

– Music game elements in serious electronic instruments.

– Music games integrated in architecture / public space / social environments.

– Music games suitable for large groups of people, in which individual expressivity and social awareness are both continuously respected.

– Open source music games allowing for co-design, or other co-design methods based allowing co-creators to build upon certain design pillars.

Leave a Reply

Your email address will not be published. Required fields are marked *