Possibilities for implementing single-note expression

Currently, MuseScore plays notes using note-on velocity to represent dynamics. This method, however, doesn't allow for dynamic changes during a note as one would hear with instruments such as strings, brass and woodwinds. To accomplish such expression, "MIDI continuous control changes" (abbreviated e.g., CC11) are typically used. The General MIDI (GM) standard reserves certain controllers for this purpose. Though many modern synths and samplers do not follow this standard, you will typically find at least loose adherence to the longstanding controller assignments. Controllers commonly used for expression include:

• CC1: Modulation
  • This is the MIDI CC used by the modulation wheel present on most keyboards.
  • GM standard behavior assigns CC1 to control vibrato amount, though it is frequently used to control myriad other instrument parameters including filter cutoff and note expression.
  • SoundFonts default to the GM standard behavior (vibrato), but SFZ instruments have no default assignment.
  • CC1 is commonly used by orchestral libraries such as Garritan Personal Orchestra to control note expression due to the wide availability of the modulation wheel.
• CC2: Breath Controller
  • Originally intended for use with a breath controller in which blowing harder produces higher MIDI control values.
  • Has no predetermined effect in GM standard, but is commonly assigned to control note expression.
  • Neither SoundFonts nor SFZ instruments assign a default function to CC2.
  • I have never seen a SoundFont configured to use CC2 (except some I have created myself).
  • Some orchestral libraries assign CC2 to control note expression by default.
• CC4: Foot Controller
  • Same as CC2, but for use with an expression foot pedal.
• CC11: Expression
  • GM standard behavior is as a secondary volume control for the MIDI channel, calculated as a percentage of the volume as set by CC7: Volume.
  • Many modern synths & samplers do not assign a default behavior to CC11.
  • SoundFonts have CC11 controlling the volume by default, but SFZ instruments do not.
  • In some orchestral libraries, CC11 is used for note expression, while in others, it is used as a basic volume control with other controllers (CC1, CC2) being used to control expression instead.
• Polyphonic Aftertouch
  • Aftertouch is a feature on some keyboards where additional pressure can be applied to a key after it has been struck. Doing so will send MIDI events based on the amount of pressure.
  • MIDI events are separate for each key, so simultaneous notes can be expressed independently.
  • GM standard has no default effect, but aftertouch is frequently assigned to control vibrato or note expression.
  • In MuseScore, neither Fluid nor Zerberus support polyphonic aftertouch, though support was added in 2017 to upstream FluidSynth.
• Channel Aftertouch
  • Same as polyphonic aftertouch, but lacking per-key expression as it affects the entire MIDI channel.
  • SoundFonts have channel aftertouch mapped to vibrato amount by default.

From this list, we can ascertain that:

• Polyphonic Aftertouch is the only way to attain polyphonic note expression (e.g., different dynamics within a chord), but is not supported by either of MuseScore's synths, and not commonly supported outside of MuseScore either.
• CC1, CC11 and Channel Aftertouch have default behaviors in SoundFonts but not SFZ files.
• CC2 & CC4 have no default behavior in either SoundFonts or SFZ files
• Commercial orchestral libraries commonly use CC1, CC2, or CC11 to control note expression.

Both the SoundFont and SFZ formats allow any MIDI CC to be changed from the default behavior and assigned to control any parameter of the instrument. A well-programmed expressive instrument will configure the expression CC to not only affect an instrument's volume, but also its tonality. Typically, synth parameters and sample switching are used to create a more realistic sound across an instrument's dynamic range. Playing a fortissimo trumpet sample at a low volume, for example, does not correctly sound like a trumpet playing pianissimo.

So, when I refer to "expression", I am not just speaking of volume. I make this distinction because some orchestral libraries will use, say, CC1 to control expression and CC11 to control volume, with the audible result between the two being very different.

Possible Implementations

Here are a few different ways we could implement intra-note expression within MuseScore, along with pros and cons for each:

1. CC2 for expression with note-on-velocity sent as normal

Expressive instruments in MuseScore_General would be configured to change volume/tone via CC2, with sample switching controlled by note-on-velocity. Non-expressive instruments (pianos, etc.) would have their dynamics controlled by note-on-velocity only.

MuseScore would send matching dynamics via note-on velocity and CC2 simultaneously (e.g., velocity=38, CC2=38), so both expressive and non-expressive instruments in MuseScore_General would just work without any intervention from the user.

Since CC2 is rarely used in SoundFonts and has no default behavior in any major sampler, sending CC2 events would only control instruments that are configured to support it. If a user wants to have expressive control in an instrument that is not shipped with MuseScore, they would need to set CC2 to control note attenuation and unmap velocity from note attenuation. For SoundFonts, this can be accomplished with modulators using a SoundFont editor.

Pros:

• The use of both CC2 and velocity would provide the maximum control--and therefore, realism--for expressive instruments.
• Both expressive and non-expressive instruments will handle dynamics as expected, with no further intervention from the user.
• Will produce a smooth volume transition in and out of a hairpin.
• Supported by commercial Orchestral Libraries, as most already support CC2 control for expression or allow user configuration to support any CC number.
• Will not create adverse behavior in user SoundFonts or SFZ files, simply falling back to velocity-based control where CC2 is not supported.

Cons:

• Instruments must be manually configured to support this method of expression.
• Expressive instruments will not produce dynamics if expression mode is turned off in MuseScore.
• Expressive instruments only: Consecutive dynamic marks not connected by a hairpin will sometimes cause an audible volume "bump" on transition to the second dynamic. This is caused by the sudden change in CC2 level while the previous note is still in its release phase. There are two ways to address this issue:
  • True fix (internal SoundFont renderer only):
    • Make Fluid voices in release phase ignore further CC2 events.
    • Make sure that the note release event occurs prior to the adjustment of CC2 level.
  • Workaround (works for internal and external synths):
    • When going from a louder to a softer dynamic:
      • Smoothly scale CC2 between the two dynamics 0.25 seconds prior to the second dynamic.
    • When going from a softer to a louder dynamic:
      • Smoothly scale CC2 between the two dynamics 0.25 seconds prior to the second dynamic.
      • Any notes that end at the start of a new dynamic marking should be released 0.25 seconds prior. If the note starts at 0.5 seconds or less before the new dynamic, it should play for half of its value instead.

2. CC11 for expression with fixed note-on-velocity

With this expression method, note velocity would always be 127, with CC11 controlling expression. This expression method would need to be enabled or disabled in the mixer, since it is not compatible with non-expressive instruments. The appropriate mode would be chosen when an instrument is added to the score.

Pros:

• Would work with all SoundFonts (SF2, SF3) by default.
• Will produce a smooth volume transition in and out of a hairpin.
• Expressive instruments will still produce dynamics if expression mode is turned off in MuseScore.
• Supported by commercial Orchestral Libraries, as most already support CC11 control for expression or allow user configuration to support any CC number.

Cons:

• The use of fixed velocity would provide less control of an instrument's expressive parameters and would not allow sample switches (PP vs. FF samples) in the default sounds.
• Dynamics won't work with non-expressive instruments due to the fixed velocity, so user would need to enable/disable this mode in the mixer.
• SFZ and external instruments need to be manually configured to work with this method.
• Consecutive dynamic marks not connected by a hairpin will sometimes cause an audible volume "bump" on transition to the second dynamic. This is caused by the sudden change in CC11 level while the previous note is still in its release phase. Please see method #1 above for a suggested fix.

3. CC11 for expression relative to velocity

This is the method developed by hpfmn in 2016 and improved upon by James Thistlewood. It is a rather ingenious way to achieve intra-note expression for synths/samplers where both note-on-velocity and CC11 affect note volume equally (as is the default case for SoundFonts).

The way it works, roughly, is this: Dynamics are controlled primarily by note-on-velocity with CC11 set to full volume (127). When a hairpin is encountered, notes throughout the hairpin duration are played at the maximum velocity required by any point within the hairpin/note span, with CC11 scaling either up to or down from 127 to create a smooth cresc./dim.

This expression method would need to be enabled or disabled in the mixer, since it is not compatible with non-expressive instruments. The appropriate mode would be chosen when an instrument is added to the score.

Pros:

• The use of both CC11 and velocity would provide the maximum control--and therefore, realism--for expressive instruments.
• Would work with all SoundFonts (SF2, SF3) by default.
• Consecutive dynamic markings without a connecting hairpin will sound the most natural with this method.
• Expressive instruments will still produce dynamics if expression mode is turned off in MuseScore.

Cons:

• This method only works by default with SoundFonts, and is not supported by SFZ or any other samplers/orchestral libraries (to my knowledge--though it may be possible to configure some to support this method).
• This method would need to be disabled in the mixer for non-expressive instruments.
• Consecutive dynamic marks connected by a hairpin will sometimes cause an audible volume "bump" at the beginning and/or end of the hairpin. This is caused by the sudden change in CC11 level while the previous note is still in its release phase. This can be fixed by:
  • Make Fluid voices in release phase ignore further CC11 events.
  • Make sure that the note release event occurs prior to the adjustment of CC11 level.

Suggested Implementation

Method #1 (CC2 for expression with note-on-velocity sent as normal) is the strongest option, in my opinion. While it doesn't support expression in all SoundFonts by default, I believe the compatibility with high quality SFZ and VST sample libraries is more valuable for creating quality recordings than supporting the mass of subpar SoundFonts out there.

Method #2 (CC11 for expression with fixed note-on-velocity) is the weakest of the three, and should only be considered as a selectable option for additional sample library compatibility.

Method #3 (CC11 for expression relative to velocity) has certain flexibilities that neither #1 nor #2 possess, but it only works with the internal SoundFont engine, and produces audible "bumps" around hairpins.

Besides going with Method #1, I would also suggest adding the following control options for each channel in the mixer:

• Expression Control: Off, CC1, CC2, CC4, CC11 (default = CC2)
• Note Velocity: "Normal" or "Fixed" (user can choose fixed velocity, default = 127)

This would give users the most control for working with external sounds. For example, setting "Expression Control" to CC11 and "Note Velocity" to "fixed" would allow the user to easily use expression with any SoundFont (this is identical to Method #2 above).

Test SoundFont

I have created a SoundFont for testing any expression method implemented in MuseScore in the future. Download here. It contains the following presets, all based on the preset "020:048 Violins Fast" from MuseScore_General 0.1.3:

• 000: default SF2 - This preset features the default SoundFont behavior, and can be used with expression methods #2 and #3, with #3 sounding superior on note attacks.
  • Velocity: volume, tone, sample switching and bow attack speed
  • CC1: vibrato
  • CC2: none
  • CC11: volume
• 001: vel+CC11 (hpfmn) - This preset is optimized for hpfmn's method (#3), with the both volume and tone being adjusted by CC11, and the dynamic levels between velocity and CC11 perfectly matched.
  • Velocity: volume, tone, sample switching and bow attack speed
  • CC1: vibrato
  • CC2: none
  • CC11: volume & tone
• 002: CC1 exp (Garritan) - this preset emulates the default expressive control used by Garritan Personal Orchestra (CC1).
  • Velocity: sample switching and bow attack speed
  • CC1: expression (volume & tone)
  • CC2: none
  • CC11: none
• 003: CC2 expression - This preset is compatible with expression method #1.
  • Velocity: sample switching and bow attack speed
  • CC1: vibrato
  • CC2: expression (volume & tone)
  • CC11: volume
• 004: simulated SFZ - This preset emulates default SFZ file behavior, with CC1, CC2 and CC11 having no effect on the sound.
  • Velocity: volume, tone, sample switching and bow attack speed
  • CC1: none
  • CC2: none
  • CC11: none