Music Composer Data Design

It’s the Data Stupid, the DATA

Ok, so after free jazz coding a bit, I finally think I have the piece of the puzzle that sticks it all together.

It’s the data.

Consider the following object:

// There's an expectation all values are nullable 
USTRUCT()
struct FComposerData
{
	GENERATED_BODY()

	int NumBarsToCompose;
	FLetsGoGeneratedScale Scale;
	int OctaveMin = 1;
	int OctaveMax = 5;
	FInstrumentSchedule InstrumentSchedule;
	IMusicCompositionStrategy* CompositionStrategy;
	int ComposerDataObjectIndex;
};

Here I have build a struct representing the data a Composer needs to do its job- it’s incomplete, but it’s the glue that will be passed around to each other object that needs to consume what the composer is creating.

The Composer, broadly, needs to create InstrumentSchedules to send to some Instruments.

Instrument Schedules contain the Sound that will be played, and the beat it will be played on.

The Composer will hold a set of ComposerData.

In this set, some of the InstrumentSchedules will be currently playing, others will be pending.

For instance, imagine a Bass instrument playing a pedal point of [ Bb, Bb, Bb, Bb ]

The composer created a ComposerData object that contains the schedule for the Bass.

However, it only wants the Bass to play this for say, 4 bars.

After that, it will want to essentially replace the InstrumentSchedule with a motif:

[ Bb, Db, F, Ab ] - a walk up a B minor 7th scale.

So, given that scenario, what do we know we need?

FMusicalScale containing the Tonic, and Notes of the B minor scale
An FInstrumentSchedule containing the Sound wave of each note and the beat to play on

This schedule has a set of bars to play- so it would have an array of 4 [ Bb, Bb, Bb, Bb ]
The schedule also contains the Sound to play

It doesn’t actually send Bb , it sends {Note}_{Instrument}_{OctaveNumber}.wav
This means ComposerData needs to know the Instruments and the mapping to the actual wav file.

We mentioned this is a Bass instrument that will be playing.

This infers a type of instrument, and/or a range of Octaves that an instrument will play

A NumBars for the number of bars to play this instrument schedule

Actually, InstrumentSchedule is structured to have a set of bars to play already
What is needed is which bar to start playing this instrument.
With these two data points, the bar this will finish on can be derived.

So what do we do in terms of the evolution from pedal point to motif?

Somewhere the composer needs to determine: Bar 1-4 do a pedal point, then 5-8 “evolve” to motif.

Which means the Composer needs to hold a set of instruments and their evolutions:

// Bass 
{
	Instrument: SynthThing
	OctaveRange: [1..2]
	Schedules: [
		{ 
			schedule: [ [ Bb, Bb, Bb, Bb ] ], 
			bars: 4 
			startAtBar: 1
		},
		{
			// 2 bar schedule
			schedule: [ [ Bb, Db, F, Ab ], [Ab, F, Db, Bb] ]
			bars: 2 // would play 4 total bars
			startAtBar: 5
		} 
	]
}

Potentially here, the schedule described above doesn’t necessarily need to be an InstrumentSchedule.

It could be a related data object specifically for the Composer, the intent being a valid InstrumentSchedule can be derived from this object, when it is being sent to an Instrument to play.

OTOH, that might be an unnecessary abstraction. The InstrumentSchedule is structured in a way that works with the rest of the program. So like, get with the program amirite?

The real reason though is if I want to do something like chords, this is already easy to do in InstrumentSchedule. Ab at beat 1, F at beat 1, Db at beat 1, etc.

There’s also something to be said for the fact that I’m calling this a bass. I have a drum instrument and a single sampled synth instrument configured in the game. Technically the bass could be my “simplesynth” at Octaves 1-2 and lead at octaves 3-5.

Which leads to this question of what instruments are serving what purpose? Do I want to structure the rules of say, 4-part harmony?

Eesh. This is a lot to think about. But I gotta keep going, if only because I think Im on the right track.

So.

ComposerData has Instrument and Octave range. Easy enough.

Instrument + Octave + Note gives you SoundData (how the .wav file is represented in my code).

Note is derived from Scale, Scale is updated dynamically based on gameplay events.

Notes are selected via a MusicCompositionStrategy - PedalPoint, create Motif, modify Motif, etc.

A composition strategy takes a ComposerData and returns a InstrumentSchedule?

Yes, because Composer Data has the instrument, the octave range, etc.

ComposerData also has a set of InstrumentSchedules, so Strategies have access to the existing InstrumentSchedules, useful if you want to “evolve” an existing schedule.

So Composer loops through its ComposerData’s, checking if there is enough schedules for next, say, 2 bars. If not, create some bars for that part.

💡

There might be another meta layer on top of this responsible for turning on or off parts, setting up Bridge vs. Chorus vs. Verse, etc.

Actually let’s talk about that meta layer for a minute.

Essentially, there is another process in the Composer that is creating the structure of the song. It’s mapping song sections into the instruction to create/modify ComposerData’s. It will be the thing that says “The Intro will be 8 bars, then a Verse for 8, then a Chorus for 2, then a Bridge for 2, etc.”

Then a looping function creates the into 8 bars, then the verse, etc.

So, I create this structure:

UENUM()
enum ESongSection
{
	None,
	Intro,
	Chorus,
	Bridge,
	VerseSection, // Verse has a conflict with some CoreEngine namespace
};

USTRUCT()
struct FSongSections
{
	GENERATED_BODY()

	TEnumAsByte<ESongSection> SongSection;
	int SectionLengthInBars;

	FSongSections(): SongSection(None), SectionLengthInBars(0) {}
	FSongSections(const ESongSection InSection, const int InLength): SongSection(InSection), SectionLengthInBars(InLength) {}
};

Establishes that song sections exist, and are in a struct that has how many bars for that section.

Now I’m thinking to add a set of valid Music Composition Strategies- Intro has “SetPedalPoint” and maybe “CreateMotif”, where Chorus would contain… other strategies…

I dunno. This all seems kinda sus. Does endlessly generative music have set song sections like this? This feels like a random abstraction

Off the rails and in the Data

Ok, Lets bring it back. The thing we know is important is the NumBars. We also know what bar it currently is in time.

We know that we need some kind of function that creates FComposerData.

We know we need to pass ComposerData to the Strategy.

We don’t know if the ComposerData is valid for the Strategy.

If the Scale doesn’t have a Tonic, PedalPoint fails.

So, what if we added a function to the IMusicCompositionStrategy to assert is valid?

It makes sense that Composer would own a set of CompositionStrategies.

Create and hold references to all

When its ready, it will choose a Strategy and call the Apply function to get an instrument schedule from the strategy.

As part of that “readiness, choose strategy” logic, it can query each Strategy it has, with the data it current has, and get a subset of valid strategies.

Then just choose at random.

Eventually I want some kind of weighting system that will provide a better solution than at random.

But thats not really a IsValid… that’s more, is appropriate. And I still need some sort of appropriateness algo as setting a pedal point, and then setting a pedal point, and then setting a pedal point… not that hard if just choosing at random. Not appropriate.

So hol up. There may be a strategy here to Create a Bass. Or a lead.

Imagine we have Validity and Appropriateness all figured out. At some point the Composer chooses “Hey I need a bass.”

That’s a strategy.

Not a complete strategy though.

If this is the content of our ComposerData:

	Scale Scale; 

	FInstrumentData InstrumentData;
	int OctaveMin = 1;
	int OctaveMax = 5;

	TArray<FInstrumentSchedule> Schedules;

Then “Give me a bass” essentially returns InstrumentData + OctaveRange.

What is interesting about this is a pedal point for a bass is a lot more appropriate than for say a lead/soprano type instrument/part.

{
 	FInstrumentData = BassInstrument;
	int OctaveMin = 1;
	int OctaveMax = 2;
	
	StrategyWeights = {
		PedalPoint = .8,
		Chords     = .4,
		RespondToCall = 1.2
	}
}

So here we have a mapping of different weights to some arbitrary appropriateness value.

If we loop through our ComposerData’s, we have data to determine appropriateness of strategies for this ComposerData.

We still need something to represent NumBars in here though.

Now InstrumentSchedule itself defines this like:

// Two bars long
[ Bb, Bb, Bb, Bb ],
[ Bb, Ab, F,  Bb ]

But when we’re creating the InstrumentSchedule, we need to pass in the number of bars to create.

{	
	BarsPerStrategy = {
		PedalPoint = 4,
		Chords     = 2,
		RespondToCall = 1
	}
}

So if we choose a pedalpoint, the Strategy knows how many bars of InstrumentSchedule to create.

{
	// What part does this play in the composition? 
 	FInstrumentData = BassInstrument;
	int OctaveMin   = 1;
	int OctaveMax   = 2;
	
	// What strategies are appropriate for this part? 
	StrategyWeights = {
		PedalPoint    = .8,
		Chords        = .4,
		RespondToCall = 1.2
	}
	
	// How many bars per part?
	BarsPerStrategy = {
		PedalPoint    = 4,
		Chords        = 2,
		RespondToCall = 1
	}
	
	// On which beat do the instruments play each schedule? 
	WhenPlaySchedules = {
		PedalPoint = 4, // start on bar 4
		Chords1    = 8, // "replaces" pedalpoint
		Chords2    = 10 // "replaces" Chords1
	}
}

If the composer is on bar 9, it can determine that this Data only has bars to 12, it can determine to create more schedules.

The “create a bass” strategy though would look something like:

 	FInstrumentData = BassInstrument;
	int OctaveMin   = 1;
	int OctaveMax   = 2;
	
	// Default strategy weighting
	StrategyWeights = {
		PedalPoint    = 0.8,
		Chords        = 0.4,
		RespondToCall = 0.0
	}
	
	BarsPerStrategy = {}
	WhenPlaySchedules = {}

The composer can easily determine that this bass needs new schedules, uses the weights and has a 66% chance of selecting PedalPoint.

 	FInstrumentData = BassInstrument;
	int OctaveMin   = 1;
	int OctaveMax   = 2;
	
	// Weights update on pedal point selection
	StrategyWeights = {
		PedalPoint    = 0.2, // Appropriateness reduced
		Chords        = 0.8, // appropriateness increased
		RespondToCall = 0.0  // Special case... needs external schedule
	}
	
	// By selecting PedalPoint, we define NumBars to create
	BarsPerStrategy = {
		PedalPoint = 4 
	}
	
	// On select pedal point, we also define when to start 
	WhenPlaySchedules = {
		PedalPoint = 2 
	}

There is a catch, caught in RespondToCall.

A big part of music is the interaction with the other musicians/parts/instruments within a composition.

In terms of a bass part, I can’t imagine a bass that doesn’t take input from, and provide input to, the drums. As a default case, these two should be locked in, synced up, and grooving in the pocket.

I’m going to break here I think, let this angle cook in my subconscious.

It’s called Drum & Bass for a reason

Ok so specifically for the bass, it generally wants to follow the kick of the drum. It doesn’t have to, it can do whatever it damn pleases, but those raised in the western musical tradition are going to appreciate a drum and bass working together.

Now we can assume a kick drum playing on the 1 and the 3.

And we can assume a kick drums is getting its instructions from the Composer.

Right now, its not, StartDrums Phase selects a predefined drum pattern at random.

What’s interesting here is the fact that there are predefined drum patterns.

The code has knowledge of a Basic 1-3 rock beat. Or samba, or jazz swing. These things are codified as InstrumentSchedules already.

And I’m saying that the bass needs access to a selected InstrumentSchedule.

 	FInstrumentData = BassInstrument;
	int OctaveMin   = 1;
	int OctaveMax   = 2;
	
	AllOtherInstrumentSchedule = {
		DrumKick*,
		DrumSnare*,
		LeadGuitar*
	}

That would be a naïve implementation.

I think what would be better would be RelevantInstrumentSchedules for… relevant… instrument… schedules.

The reasoning being the bassist doesn’t necessarily care about the Snare.

That being said, it might be better to also assign a weighting of how much it cares:

	RelevantInstrumentSchedules = {
		DrumKick = 0.8,
		DrumSnare = 0.2,
		LeadGuitar* = 0.5
	}

Ahhh but here’s the thing. It’s something I had forgotten to flesh out. There is a difference between which note you play and when you play it.

	PercussionInstrumentSchedules = {
		DrumKick = 0.8,
		DrumSnare = 0.2
	}
	
	MelodicInstrumentSchedules = {
		LeadGuitar = 8.0
	}

Now the lead guitar can give the bassist important percussive information. Remember this:

 	FInstrumentData = BassInstrument;
	int OctaveMin   = 1;
	int OctaveMax   = 2;
	
	Strategies = {
		RespondToCall = {
			InstrumentSchedules = {
				LeadGuitar = 1.0, 
			},
			Wheight = 8.0,
			StartAtBar = 4, // In theory the ~beat after the lead guitar makes a "Call" 
			BarsToPlay = 1, 
		},
		PedalPoint = {
			InstrumentSchedules = {} // Not listening for input from anyone 
			Weight = 2.0,
			StartAtBar = 4,
			BarsToPlay = 4
		}
	}

So here I’ve done a couple things, I’ve collapsed WhenPlaySchedules and BarsToPlay sections in the pseudo-data above into a single strategy object.

I’ve also add weighted InstrumentSchedules as a member of that object.

Now, Call and Response might actually run a bit differently. It might make more sense for this kind of strategy to take 2 instruments, Caller = guitar, Responder = Bass, and inject the appropriate instrument schedules.

But I think this makes sense in a general format.

Strategies of Pure Music

There is a slight issue.

InstrumentSchedules are essentially

[ 1.0, { Sound = Db_SynthThing_Octave1, Volume = 1.0 } ],
[ 3.0, { Sound = Bb_SynthThing_Octave1, Volume = 1.0 } ]

Where it’s playing Db, Bb on the 1-3 beats.

This is perfect for instruments, providing all the information they need to play the sounds at the correct time without having to derive any nonsense.

Not great if you want to create a Motif, and have the guitar and bass play the motif at the same time.

For something like “Create Motif”, the perfect data object would be something like

[ Db, Bb, F, Db ] // I, V, III, I

So how do I represent this data and feed it into the ComposerData object?

Well what is interesting about this data is the entire suite of logic it infers exists.

Something needs to create this data, something that knows about music theory in pure music theory terms.

And I already have large sections of that figured out - I have arrays of Notes built into scales of many different nodes.

So creating this is actually fairly simple- hooking it into ComposerData is whats needed.

Scale Scale = [ C, D, E, F, G, A, B ]
Scale[] Motifs = [
//  I  I  V  III
	[ C, C, G, E ], 
	[ C, B, A, G ],
	[ C, F, A, G ] 
]
Chords = [
//       I            I            V           III
	[ [ C, E, G ], [ C, E, G ], [ G, B, D ], [ E, G#, B ] ]
]

So things start off from the scale

From the scale we can have motifs

From motifs we can have chords.

Or have that reversed- it doesn’t matter what comes first, as long as what comes next considers what came before it.

A chord itself is a collection of notes to be played at the same time.

An InstrumentSchedule will flatten those chords out into three sounds at beat 1, beat 2, beat 3, etc.

InstrumentSchedules have no concept of musicality- it has no knowledge of notes.

Nor should it, drums convert into InstrumentSchedules and they have no notes (unless I did some gangster shit and put A = Snare, Ab = Kick, B = HiHatClosed).

Actually now that I think about it that’s essentially what Ableton does for MIDI drums. But not really, thats mapping a midi controller piano roll to a drum roll. There’s no reason to assume that internally it treats kick = some arbitrary note.

 	FInstrumentData = BassInstrument;
	int OctaveMin   = 1;
	int OctaveMax   = 2;
	
	Scale = [ C, D, E, F, G, A, B ]
	
	Strategies = {
		RespondToCall = {
			InstrumentScheduleData = {
				{ 
					Instrument = LeadGuitar,
					AppropriatenessForStrat = 1.0,
				  MusicalStructures = { [ C, C, G, E ] }
			  }
			},
			AppropriatenessToChooseStrat = 8.0,
			StartAtBar = 4,  
			BarsToPlay = 1, 
		},
		DoubleRootOfChord = {
			InstrumentScheduleData = {
				{
					Instrument = ChordPiano 
					AppropriatenessForStrat = 1.0,
					MusicalStructures = { 
						[ 
							[ C, E, G ], 
							[ C, E, G ], 
							[ G, B, D ], 
							[ E, G#, B ] 
						]
				}
			},
			AppropriatenessToChooseStrat = 2.0,
			StartAtBar = 4,
			BarsToPlay = 4
		}
	}

So, we have some sort of MusicalStructure associated with an InstrumentSchedule - coupling the sounds an instrument needs with the musical data required by the composer into a InstrumentScheduleData object. A StrategyData object contains a set of ISD’s along with the data for how this strategy would fit in the structure of the song.

It strikes me that this just defines the data set for choosing what to play next. The above object is missing what has already been chosen to play.

I’m wondering if this is just simply adding the InstrumentSchedules to a set in the root of data object. More I’m wondering if there’s metadata information we’d need for this set of data. Like, would I want to record that InstrumentSchedule [Db, Db, Db, Db] was created from the pedal point strategy?

It strikes me as probably definitely. We don’t necessarily need to store the entirety of a PossibleStrategy:

		ChosenStrategies = [
			{
				InstrumentSchedule = { [ C, C, C, C ] }
				StartAtBar = 4,  
				BarsToPlay = 2,
				Strategy = PedalPoint
			},
			{
				InstrumentSchedule = { [ C, D, G, E ] }
				StartAtBar = 6,  
				BarsToPlay = 4,
				Strategy = PlayMotif
			},
		]
		
		PossibleStrategies = [
			PlayMotif = {
				AppropriatenessToChooseStrat = 8.0,
				InstrumentInput = {
					{ 
						Instrument = LeadGuitar,
						AppropriatenessForStrat = 1.0,
					  ChosenStrategies = { 
					  		InstrumentSchedule = { [ C, D, G, E ] }
								StartAtBar = 6,  
								BarsToPlay = 4,
					   }
				  }
				},
			},
		]

It’s actually better if we just sent a reference to the LeadGuitar ComposerData.

	PossibleStrategies = [
			PlayMotif = {
				AppropriatenessToChooseStrat = 8.0,
				InstrumentInput = {
					{ 
						ComposerData = LeadGuitar*,
						AppropriatenessForStrat = 1.0,
				  },
				  {
					  ComposerData = ChordSynth*,
					  AppropriatenessForStrat = 0.0
				  },
				},
			},

With the reference we can simply retrieve the data we want from an input instrument.

Try to pull it all together.

Ok, so we’re building a ComposerData object to dynamically create music for Instruments.

It is representative of a single instrument.

It has a set InstrumentData objects, wrapped in a data object with ancillary information like when to play, and how many times to play it.

It has a set of StrategyData objects, defining different approaches for creating music, along with data related to choose each strat, and a set of ComposerData for other instruments that can be used by the Strategy.

Let’s quickly take a think about object lifetime:

There’s a Phase to start the Composer.

There’s an event listener to update the Scale on updates to the GameState.

There is a reference to the MainClock so the Composer knows what the current bar is.

Every bar, the Composer:

Checks the set of ComposerData’s
If there are no objects, trigger some CreationStrategy
If there are objects, check if there are more than 2 bars worth of data for those instruments to play
If not, trigger an UpdateStrategy

For now, I’m not too worried about end-of-lifetime for any of these objects. Eventually we’ll likely want a strategy to kill the bassist, but that’s unnecessary at this point.

Ok, I think I’m ready to jump back into code.

 	FInstrumentData = BassInstrument;
	int OctaveMin   = 1;
	int OctaveMax   = 2;
	
	Scale = [ C, D, E, F, G, A, B ]
	
	ChosenStrategies = [
		{
			InstrumentSchedule = { [ C, C, C, C ] }
			StartAtBar = 4,  
			BarsToPlay = 2,
			Strategy = PedalPoint
		},
		{
			InstrumentSchedule = { [ C, D, G, E ] }
			StartAtBar = 6,  
			BarsToPlay = 4,
			Strategy = PlayMotif
		},
	]
	
	PossibleStrategies = [
		PlayMotif = {
			AppropriatenessToChooseStrat = 8.0,
			InstrumentInput = {
				{ 
					ComposerData = LeadGuitar*,
					AppropriatenessForStrat = 1.0,
			  },
			  {
				  ComposerData = ChordSynth*,
				  AppropriatenessForStrat = 0.0
			  },
			},
		},

Action Name	Action State	Eligible	Repeatable?	Eligible Phase
Set Tonic	Complete	False	False	Intro
Set Third	Currently Active	True	False	Intro
Set Mode	Pending	False	False	Intro
Bass Drop	Pending	False	True	Bridge
BPM Switch	Pending	True	True	Bridge, Chorus, Outro

Strategy	Description	Requirements
Pedal Point	Tonic per beat	Requires tonic
Create Motif	Create 1-2 bar melody	Requires scale*
Motif Variation	Change notes of melody	Requires motif
Motif Augmentation	Extend/Contract melody	Requires motif
Create Chord Progression	Establish ii-V-I in key
Modulate Chord Prog	ex. backdoor ii-V-I	Requires chords