Proposing New Challenges for Music Game Design – Towards Improvisation

Photo by Lysander Yuen on Unsplash (free license)

Since 2008, I’ve been passionate about exploring the intersections of music composition, sound design and game design. Studying Composition & Sound Design for Adaptive Systems at the HKU University of the Arts, brought me in contact with a lot of game designers.

In 2011, I was part of an interdisciplinary team who designed the multi-player music installation Thresholder. On a 10x10M floor-projected network of lines, players could engage in a mixture of social improvisation, individual expressivity, physical movement and architectural spatiality. I’m currently working on reviving this project. Simultaneously, I’d like to develop new ideas for music games, and share some of my thoughts about this process. Could improvisation be key?

I penned a basic overview of non-linear (improvisation-related) game design, music game design, and musical improvisation. From that analysis, I proposed a set of ideas for (improvisation-based) music games.

Skills or Tactics
In game design, non-linearity can be used in various (intertwined) ways, with various levels of complexity. Even a game as basic as Pac Man (1980) finds a lot of its strength in the fact that players can roam through the maze in the directions of their choice, which in turn triggers the enemy ghosts to find various paths as well. This game mechanic gives players the incentive to keep replaying the game, not only motivated to improve their hand-eye coordination, but also to come up with new, more clever tactics.

Emergent Gameplay
Of course, games of various kinds combine agility with tactics, e.g. sports games and first person shooters. Yet, few games revolve specifically around emergent gameplay: the concept of giving players the ability to develop or discover (improvise) tactics that were unforeseen by the designers of the game.

This kind of gameplay can be achieved by allowing players to combine and construct objects, and to give these objects – plus their environment – highly variable properties, by for example subjecting them to the laws of physics, such as gravity, viscosity, fragility and flexibility. Examples of such games are: puzzle games The Incredible Machine (1993), Scribblenauts (2009) and the highly complex game Infinifactory (2015), as well as role playing action game Deus Ex (2000).

In 2013, thirteen years after he designed Deux Ex, Warren Spector said:

“Plans must be devised by the player; it’s not about how clever you are [as a game designer]. (…) Games that don’t offer quote-unquote ‘real choices’ might be just fine,” he suggests. “Some people may want a game that’s all about squeezing a virtual trigger, or moving forward like a shark, or solving a puzzle that shows more about how clever the designer is, rather than how clever they are. I’m just more interested in emergence than in scripted adventures… and I believe once players get a taste of that kind of game, it’s very hard for them to go back”(via Gamasutra)

Sandbox Theatre
Games like Garry’s Mod (2004) and Minecraft (2011) – don’t even rely on specific problems that need to be solved. Instead, their sandbox design stimulates improvisation, creativity and artistry by allowing players to express themselves (socially) through inventively (and improvisingly) playing with objects, physics, camera perspective, NPCs (non playable characters) – as a sort of on-the-fly level editing experience.

Procedural Generation
Randomisation offers other forms of non-linearity. It can add an element of surprise to the game, thereby giving it a higher replay value. Algorithms (sets of instructions for the computer to follow) are called procedural when they are employed to generate endlessly evolving patterns. Some games tap into a random moment of the procedural pattern’s evolution, and then use it to evolve for example random landscapes.

Procedular level generation was first used in the Apple II game Beneath Apple Manor (1978), which also happened to be the first commercially available role playing game (RPG). Cutting-edge games like Horizon Zero Dawn (2017) for the Playstation 4 use procedural placement to organically simulate nature.

Open Worlds
Open world games like The Legend Of Zelda (1986) and Red Dead Redemption 2 (2018), allow players to roam around and tackle quests in various (non-linear) orders.

Yet, there isn’t a clear consensus on the definition of an open world. I believe this is because it’s a spectrum: non-linearity and linearity can be combined in various ways and degrees:

▫️ Non-linear storytelling: the amount of variation a game offers in terms of completing challenges in different orders, and how much effect this has on changes in the story: in how many directions can the story branch out?
▫️ Bonus play: the amount of extra variation a game offers by offering optional gameplay. For example by rewarding the players extra points for finding hidden objects or executing more difficult to perform moves.
▫️ Procedural generation: which elements are randomised and how random does the outcome feel?
▫️ Sandbox elements: in what ways are the players able to leave their artistic mark?

Adaptive Music
In many contemporary high-end games, the music seamlessly adapts itself to the player’s current situation, in terms of: health, location, activities within that location (relaxed vs threatening), room acoustics (small toilet vs big church), etc. This kind of adaptive music can work in (at least) three ways:

A. Horizontally: by chopping a composition into loops that can be re-sequenced.
B. Vertically: by adding or removing certain instruments from the aforementioned loops, also known as re-orchestration.
C. Modulation: by adding direct flexibility to musical parameters (timbre, intensity, speed, etc.) of certain instruments.

The first game to use a combined horizontal-vertical adaptive music system, with smooth transitions, was the point-and-click adventure Monkey Island 2 (1991).

The sound design of Tetris Effect (2018, with VR support) pairs every single action (movement, rotation, etc.) with a corresponding sound effect (with I believe an element of randomisation), thereby blurring the lines between sound effects and music. GlitchHiker (2011) game I co-created at Global Game Jam, also used interactive music in such as way (instant glitching) that additional sound effects weren’t necessary.

Musical Improvisation as a Byproduct
Otocky (1987) was a very musical game for the Nintendo Famicom (NES) game console. In this side-scrolling-shoot-em-up the player was able to shoot in 8 directions, each of which represented all 7 notes of a (western music) minor/major scale, plus the next octave’s root note.

This meant that you would simultaneously shoot goals and improvise melodies on top of the background music. Your melodies would automatically be quantised: the timing of each note would be slightly nudged to align it with the rhythmic grid of the background music.

Procedular Music and A.I.
Portal 2 (2011) took it a step further by using real-time procedurally generated music: every action (such as shooting) instantaneously generates a unique bit of music.

Recently the first tech demos of an AI-driven music system for the real-time generation of adaptive music were created by Melodrive (2018). 

Music Games – Pre-History

What about music games? A beautiful example is Mozart’s Dice Game from 1787 (play online), for which he wrote 272 measures that can be arranged in any random order (non-linearly). In the video game world, most music games are neither non-linear nor improvisatory; but there exceptions and innovations, we’ll get to to them soon. First I have to mention an invention that might have kickstarted the whole music video game genre.


A Japanese invention from 1971. Initially it didn’t include a television screen — see Daisuke Inoue’s Juke 8) — but as it spread across Asia in the 1980s, karaoke transformed into a karaoke box, also known as KTV (karaoke television): private, sound-proof karaoke rooms with on-screen lyrics. Although karaoke binds its players to relatively linear ruleset — singing the songs as accurately as possible — it does give players room to express themselves creatively; after all, a little improvisation or weirdness can add to the fun and social interaction. 

When using more recent Japanese karaoke systems, players can score skill-points, based on various elements of singing. A spin-off of this scoring system appeared in Karaoke Joysound (2008) for Nintendo Wii. Playstation game Singstar (2004) visualised the melody as it approached, and gave its players a live-indication of their accuracy.

Karaoke Joysound (Wii)

Music Games: Call-and-Response Rhythm Games

After Karaoke, Japan once again gave birth to an interactive music idea that would become hugely popular: rhythm games. Most commonly, the objective of these 1990’s games was to follow the rhythmic cues that scrolled across the screen.

Peculiarly, the first rhythm game, PaRappa the Rapper (1996) for Playstation 1, didn’t exactly work like that. It did show a stream (sequence) of notes, but it mainly challenged the players to repeat the teacher’s rhythms; to copy him in a call-and-response way. Another thing that made PaRappa’s game mechanic quite unique, was its emphasis on improvisation: by playing extra notes in the rhythmic grid, the player could “beat” the teacher and get a chance to perform on the “cool mode” freestyle stage.

In 2011, fifteen years after he created PaRappa, designer Masaya Matsuura, explained to gaming website Kotaku how much room there still was (and is) for innovation in music game design: 

“Strictly speaking I do not believe that ‘Music Games’ as a genre really exists yet. We just have ‘Rhythm Games’. We can’t really grow the genre until we have some games that explore areas of music other than just rhythm. I really want us to help overcome this deficiency. I’d like to for example do something that features extremely accurate musical performance animation. As an example, if we could zoom in and see lifelike fingerwork in an animated pianist, the opportunities afforded to uncover and develop new methods of playing beyond the capabilities of humans would be a crucial step in musical evolution. The ability of ‘games’ to allow us to take part in such advances is crucial.

Jazz musician Bobby McFerrin plays a melodic game with his audience in below video. He appeals to our intuitive understanding of the pentatonic (ancient 5-note) musical scale. Could we translate this to a video game? Why not, right?

Dance Games

In 1997, just one year after PaRappa came out, the rhythm virus spread from the Playstation to the Arcade Hall, thanks to Konami’s DJ simulator Beatmania. No longer was there any call-and-response game mechanic; the gameplay was all about bashing buttons in sync with the scrolling notes. 1998 marked another revolution: Dance Dance Revolution (DDR), the first rhythm game with a dance controller.

The game didn’t award its players with any extra points for improvising, yet that didn’t stop a ‘DDR freestyle’ scene from emerging. Freestylers are not focused on the game’s expert levels or extreme scores; they instead prefer to come up with new choreographies and/or improvisations. Thus DDR became a platform and reached cult status.

Later games, like Dance Central (2010) and Dance Evolution / Dance Masters (2010), challenged the players to dance with their full bodies. Hand and foot gestures appear on the screen right before they have to be executed. These games use the Kinect 3D-sensor for full body tracking. DANCERUSH (2008) continued in that tradition.

Other Rhythm Games

Many other rhythm games would follow: rail-shooter Rez (2001), “traditional” Japanese drum game Taiko no Tatsujin (2001), puzzle game Lumines (2004), Guitar Hero (2005), countless mobile games, motion controlled games like Wii Music (2009) which included a Custom Jam mode for improv, “piano” game Chunithm (2015) and VR games like Beat Saber (2018).

Frequency (2001), for Playstation 2, featured a remix mode that allowed up to 4 players to improvise collaborative compositions online. Vib-Ribbon (1999) was the first rhythm game to feature an audio-based level generator; it could create new levels based on any audio CD you would insert. 

Music Games controlled by Real Instruments

In recent years we’ve seen innovative games that can be controlled by improvising on real (acoustic or electronic) instruments, like Cello Fortress (2012): an action game in which a cellist defends a fortress from 4 gamepad-players who try to destroy it. And SoundSelf (2012): a psychedelic, therapeutic VR experience controlled by chanting (singing). 

Music Education Games

Still focused on hitting the right note on the right cue, but somewhat less rhythm oriented are music-education games, as they allow you to practice on real instruments. Examples are Rocksmith (2011) for electric guitar and Synthesia (2006) for MIDI keyboard. Robert Nichols hacked Rocksmith to make it work with violin as well. 

Sound Toys

Sound toys are arguably similar to sandbox games, or they’re no games at all. Examples are SimTunes (1996), Electroplankton (2005), Björk’s Biophilia (2011), the physics-based Musyc (2013), as well as remix apps ninjaJAMM (2013) and Playground (2017). Table-top inspired ReacTable (2005) can be scaled large enough for multi-player improvisation. 

Playful Sequencers

Sentris (2014) is possibly the first real / advanced hybrid of a puzzle game and a sequencer (software to write and produce music with). Non-game influenced sequencers have also become more playful and non-linear, see for example stochastic sequencer SECTOR (2014), multi-playhead sequencer Fugue Machine (2015) and the complex but playful grid-sequencers JR Hexatone (2009) and New Path (2017). Just released, iPad app Gestrument (2018), promises to become a highly customisable interactive music engine/instrument that can be controlled by eye tracking, movements, audio tracking and more. 

Augmented Music Apps

RJDJ (2008) is a fascinating augmented reality music app / sound toy; it can make music out of any sound it picks up around the user, plus it can use smartphone sensors to change parameters of the music. RDJD’s interactive compositions (scenes) are programmed in the visual programming language Pure Data

Interactive Music Installations

When it comes to large (public) interactive installations, there are many examples of interactive music or sound, like the simple but funny Piano Stairs (2009), or the body tracking avant-garde noises of Device Unknown (2018), or the immersive, brainwave (EEG) controlled lasers and sounds of On Your Wavelength (2015). Visual art can be created by groups of people on floor-projected installations like Healing Pool (2008). But it’s rare to find one that combines collaborative interaction (social game design) + music/sound improvisation. Something that comes close is The Music Box Village (2018), a collection of architectural sound toys that can be used by artists and audience to jam with otherworldly (texture based) sounds. 


In musical improvisation it’s crucial for all players to be aware of each other, only then one can choose how to respond in one of the following ways:
▫️ Silence: giving space to other musicians to place the focus on their output.
▫️ Synchronisation: complementing one or more of the currently playing elements, move the music forward in its current direction through variations or dynamic changes.
▫️ Recontextualization: transforming one or more of the currently playing elements by playing something around them that gives them a completely different meaning.
▫️ Disruption: playing something that completely disturbs current atmosphere, thereby giving space for a drastic change of collective course.

In many traditions, group improvisation takes place on a micro level: a direct (physical) change in dynamics, melody, harmony, rhythms or choice of instrumentation (timbre). Improvisation occurs when an instrumentalist or a collective spontaneously alters any of those five elements. But that does not mean that non of those elements may be pre-set. To give an example, jazz improv is traditionally based on the harmonies and rhythms of its well-known repertoire known as jazz standards. And in case a drum circle, there is of course no melody or harmony present whatsoever. Thanks to music technologies – such as recording media, sequencers, samplers, loopers and algorithmic music generators – improvisation can also be done on a more macro level: by manipulating larger chunks of music ahead of their playback. In any case, all music, improvised or composed, can move between the 2 core-axes of music: “relaxed VS stressed” VS “stable VS chaotic”.

The momentary state of the individual instruments can be measured in the same way. Some examples:

▫️ Relaxed + Stable = a church choir singing a lengthy harmonic chord
▫️ Relaxed + Chaotic = a hectic jazz drum solo played with very soft brushes
▫️ Stressed + Stable = a demonic sounding organ chord droning on
▫️ Stressed + Chaotic = a fast death metal guitar solo at its climax

Apple’s Garageband uses a similar concept for intuitively composing drums:

The Challenges
From all the knowledge I gathered so far, I defined a set of challenges for designing music games in 2019 (under construction):

▫️ Melody/harmony games inspired by rhythm games. Moving away from the on-screen rhythmic cues, towards actually listening (to random generated sequences) with the challenge to respond with suitable notes and timings. 

▫️ Rhythm games with a stronger focus on groove and swing, achieve neural rhythmic synchronisation through repetitive patterns. Research has shown that none of the current rhythm games achieve this. 

▫️ Music games that are equally as suitable for beginners, hardcore (high-score) players, and a third group: freestylers (DDR is a succesfull example of this).

▫️ Entertainment-educational crossover games, that teach players how musical elements like rhythm, harmony, melody actually work; so they can use this to become more musical and learn how to improvise.

▫️ Masaya Matsuura’s idea: exploring post-human virtuosity, for example by giving supernatural influence over a virtual pianist. Perhaps this also relates to inventing ways to conduct expressively, in stead of performing every single note.

▫️ Music games that stimulate the players to improvise more, for example through an AI system that knows how well a player is improvising.

▫️ Music games with endless variation of music by implementing procedural (generative/algorithmic) music systems.

▫️ Music games revolving around emergent gameplay, by for example letting players build their own instruments by using physics-based playgrounds. Or through playful sequencers (combinations of games and composition software)

▫️ Music games to improve social contact (incl. games for the elderly, like the Tovertafel).

▫️Music games crossed over with sports games. And connecting rhythm games with fitness, for example by using weights as controllers.

▫️ Music game elements in serious electronic instruments.

▫️ Music games integrated in architecture / public space.

▫️ Music games suitable for large groups of people.

▫️ Open source music games allowing for co-design.

▫️ ….what are your ideas and opinions?

Published by Rutger Muller

- please visit for my portfolio -

Leave a comment

Your email address will not be published. Required fields are marked *