From Audio Chaos to Actual Music

🎶

This is part of a ongoing series called Building A Music Engine

It documents the process of me smashing my head against the keyboard to build a game called LETSGO

It’s gotten long enough to break into several sections:

‣

Building A Music Engine entries

So there is a couple things I seek to accomplish now that the CreateMotif function is created.

Much of those things are clearly evident in the latest demo:

More audio chaos than music.

The reason is pretty simple- there’s only 2 musical strategies available, so it keeps creating motifs over and over again.

What I need is more musical strategies, strategies that consume and manipulate a created motif.

Technically I built the ability to do this in the data structure of ComposerData in Generative Music Without LLMs, but it would be a looping thing that kind of smells.

And writing the CreateMotif function was fairly difficult due to fitting all the different data structures together.

So I’m thinking there is the opportunity to add a layer of abstraction to the MusicComposer.

A Naïve Approach to Music Composition

Currently, the Composer runs a complicated tick function to trigger whether or not it generates music, writing musical bars and determining which musical strategy is most appropriate.

It does not have much nuance. It’s only considering the next two bars of music, filling the next two bars with whatever strategy seems most appropriate.

What I need is something that sits on top of this, something that manages a long view of the musical composition.

• It understands the structure of music: intro, bridge, chorus, verse, etc.
• It understands that the appropriateness of musical strategies is different in each section.
• It should make managing musical strategies easier
• Right now the game crashes due to a weird bug related to how strategies are being managed.
• It should be unconcerned with the actual notes and data structures the rest of the game requires to make the actual sounds.

I will need to design an object that satisfies the above requirements, build that object, refactor the MusicComposer to use it, then start implementing musical strategies using the new system.

Easy peasy, right?

Requirement Refinement

It should be unconcerned with the actual notes and data structures the rest of the game requires to make actual sound.

Lets dive into this.

In the game, the player can set tonic key, major/minor third, etc.

Which means off the rip the composer does not have knowledge of what keys it can use.

Thankfully, in music theory don’t need specific keys in order to generate musical structures.

Consider the common jazz progression: ii V I (minor second chord, major fifth chord, tonic chord)

This works whether the I chord is in the key of Db or F# or any other of the 12 notes dividing culturally western music.

So, I will need an object that works with music in the same way, defining the musical structures without specific keys.

In a similar vein, one of the insights I already built into the Composer is that it only uses the chromatic scale (a scale containing all 12 notes).

The chromatic scale doesn’t sound all that musical (or, at least, is very easy to make sound unmusical), but all other scales are a subset of the chromatic- all scales are the chromatic scale, just with missing notes.

What I wanted with the CreateMotif function was to generate a musical idea. That idea got ground down into the requirements of FInstrumentSchedule and FComposerData.

I want something lighter. I want something like a seed idea [ii V I], a simple structure that cascades and evolves throughout the musical composition.

Something even simpler than that - [2nd 5th 1st] , or simply [2, 5, 1] as a dead simple array of notes. A seed idea.

That simple idea is then given an order of operations.

So, we need an object that Generates Seed Idea, whose output is something like [2, 5, 1]

Start With What You Have

One lesson I continually have to relearn is that when you’re refactoring, starting with what you currently have is often more important than what your end goal is.

Above, I went into some detail with the end goal.

I shall now go into some detail with where the code currently stands.

The music composer basically runs a tick function that:

• Sets what bar in the future will be composed for
• Chooses the most appropriate musical strategy
• Generates a FInstrumentSchedule for the bar defined
• Adds that schedule to a list of instrument schedules to play

The appropriateness and schedule generation is done through a MusicalStrategy object, a stateless UObject overriding an IMusicStrategy interface defining what functions are needed.

State is passed through each function through a MusicComposerState object, which contains all the data the Composer needs shared.

With the change I intend, I will changing how bars are managed.

• Intro will be 4 bars, 4 bar verse, 2 bar chorus, 2 bar bridge, etc.
• It will then order the sections intro, verse, chorus, verse, chorus, bridge, etc.
• Which sets the bar numbers - 0-4, 5-9, 10-12, 13-18, 19-21, etc.

With the change I intend, managing MusicalStrategies will be changed.

• Intro will have a set of musical strategies with appropriateness scores different from other sections
• Intro might actually contain a map of appropriateness to strategy, as opposed to strategy generating an appropriateness score?

Each section will have a set of FInstrumentSchedule, which allows for duplication/repetition of sections.

I think I’d like the music conductor to just consume the list of sections, and the instruments will play the appropriate schedules.

Which brings up a point. Instruments are tied to the instrument type - A long running “Bass” Instrument Actor.

Should Instruments be shorter lived? For instance, have the conductor create new instruments for the duration of each section?

Might be worth considering the trade-offs. I feel like getting the InstrumentSchedule data into the Instrument is much easier to reason about if the Conductor creates Instruments per section.

Putting It All Together

• Need an object that generates abstract musical ideas
• Need a new UObject or struct that contains a set of musical sections (intro, verse, etc.).
• Need to have each section have some knowledge of each instrument.
• Consumes abstract musical ideas
• Determines strategy appropriateness
• Outputs instrument schedules per instrument for the section
• Composer manages song sections
• Ordering
• Triggering section operations
• Sending sections to Conductor to play