Possibilities for implementing single-note expression

• Dec 18, 2018 - 23:58

Currently, MuseScore plays notes using note-on velocity to represent dynamics. This method, however, doesn't allow for dynamic changes during a note as one would hear with instruments such as strings, brass and woodwinds. To accomplish such expression, "MIDI continuous control changes" (abbreviated e.g., CC11) are typically used. The General MIDI (GM) standard reserves certain controllers for this purpose. Though many modern synths and samplers do not follow this standard, you will typically find at least loose adherence to the longstanding controller assignments. Controllers commonly used for expression include:

  • CC1: Modulation
    • This is the MIDI CC used by the modulation wheel present on most keyboards.
    • GM standard behavior assigns CC1 to control vibrato amount, though it is frequently used to control myriad other instrument parameters including filter cutoff and note expression.
    • SoundFonts default to the GM standard behavior (vibrato), but SFZ instruments have no default assignment.
    • CC1 is commonly used by orchestral libraries such as Garritan Personal Orchestra to control note expression due to the wide availability of the modulation wheel.
  • CC2: Breath Controller
    • Originally intended for use with a breath controller in which blowing harder produces higher MIDI control values.
    • Has no predetermined effect in GM standard, but is commonly assigned to control note expression.
    • Neither SoundFonts nor SFZ instruments assign a default function to CC2.
    • I have never seen a SoundFont configured to use CC2 (except some I have created myself).
    • Some orchestral libraries assign CC2 to control note expression by default.
  • CC4: Foot Controller
  • CC11: Expression
    • GM standard behavior is as a secondary volume control for the MIDI channel, calculated as a percentage of the volume as set by CC7: Volume.
    • Many modern synths & samplers do not assign a default behavior to CC11.
    • SoundFonts have CC11 controlling the volume by default, but SFZ instruments do not.
    • In some orchestral libraries, CC11 is used for note expression, while in others, it is used as a basic volume control with other controllers (CC1, CC2) being used to control expression instead.
  • Polyphonic Aftertouch
    • Aftertouch is a feature on some keyboards where additional pressure can be applied to a key after it has been struck. Doing so will send MIDI events based on the amount of pressure.
    • MIDI events are separate for each key, so simultaneous notes can be expressed independently.
    • GM standard has no default effect, but aftertouch is frequently assigned to control vibrato or note expression.
    • In MuseScore, neither Fluid nor Zerberus support polyphonic aftertouch, though support was added in 2017 to upstream FluidSynth.
  • Channel Aftertouch
    • Same as polyphonic aftertouch, but lacking per-key expression as it affects the entire MIDI channel.
    • SoundFonts have channel aftertouch mapped to vibrato amount by default.

From this list, we can ascertain that:

  • Polyphonic Aftertouch is the only way to attain polyphonic note expression (e.g., different dynamics within a chord), but is not supported by either of MuseScore's synths, and not commonly supported outside of MuseScore either.
  • CC1, CC11 and Channel Aftertouch have default behaviors in SoundFonts but not SFZ files.
  • CC2 & CC4 have no default behavior in either SoundFonts or SFZ files
  • Commercial orchestral libraries commonly use CC1, CC2, or CC11 to control note expression.

Both the SoundFont and SFZ formats allow any MIDI CC to be changed from the default behavior and assigned to control any parameter of the instrument. A well-programmed expressive instrument will configure the expression CC to not only affect an instrument's volume, but also its tonality. Typically, synth parameters and sample switching are used to create a more realistic sound across an instrument's dynamic range. Playing a fortissimo trumpet sample at a low volume, for example, does not correctly sound like a trumpet playing pianissimo.

So, when I refer to "expression", I am not just speaking of volume. I make this distinction because some orchestral libraries will use, say, CC1 to control expression and CC11 to control volume, with the audible result between the two being very different.

Possible Implementations

Here are a few different ways we could implement intra-note expression within MuseScore, along with pros and cons for each:

1. CC2 for expression with note-on-velocity sent as normal

Expressive instruments in MuseScore_General would be configured to change volume/tone via CC2, with sample switching controlled by note-on-velocity. Non-expressive instruments (pianos, etc.) would have their dynamics controlled by note-on-velocity only.

MuseScore would send matching dynamics via note-on velocity and CC2 simultaneously (e.g., velocity=38, CC2=38), so both expressive and non-expressive instruments in MuseScore_General would just work without any intervention from the user.

Since CC2 is rarely used in SoundFonts and has no default behavior in any major sampler, sending CC2 events would only control instruments that are configured to support it. If a user wants to have expressive control in an instrument that is not shipped with MuseScore, they would need to set CC2 to control note attenuation and unmap velocity from note attenuation. For SoundFonts, this can be accomplished with modulators using a SoundFont editor.

Pros:

  • The use of both CC2 and velocity would provide the maximum control--and therefore, realism--for expressive instruments.
  • Both expressive and non-expressive instruments will handle dynamics as expected, with no further intervention from the user.
  • Will produce a smooth volume transition in and out of a hairpin.
  • Supported by commercial Orchestral Libraries, as most already support CC2 control for expression or allow user configuration to support any CC number.
  • Will not create adverse behavior in user SoundFonts or SFZ files, simply falling back to velocity-based control where CC2 is not supported.

Cons:

  • Instruments must be manually configured to support this method of expression.
  • Expressive instruments will not produce dynamics if expression mode is turned off in MuseScore.
  • Expressive instruments only: Consecutive dynamic marks not connected by a hairpin will sometimes cause an audible volume "bump" on transition to the second dynamic. This is caused by the sudden change in CC2 level while the previous note is still in its release phase. There are two ways to address this issue:
    • True fix (internal SoundFont renderer only):
      • Make Fluid voices in release phase ignore further CC2 events.
      • Make sure that the note release event occurs prior to the adjustment of CC2 level.
    • Workaround (works for internal and external synths):
      • When going from a louder to a softer dynamic:
        • Smoothly scale CC2 between the two dynamics 0.25 seconds prior to the second dynamic.
      • When going from a softer to a louder dynamic:
        • Smoothly scale CC2 between the two dynamics 0.25 seconds prior to the second dynamic.
        • Any notes that end at the start of a new dynamic marking should be released 0.25 seconds prior. If the note starts at 0.5 seconds or less before the new dynamic, it should play for half of its value instead.

2. CC11 for expression with fixed note-on-velocity

With this expression method, note velocity would always be 127, with CC11 controlling expression. This expression method would need to be enabled or disabled in the mixer, since it is not compatible with non-expressive instruments. The appropriate mode would be chosen when an instrument is added to the score.

Pros:

  • Would work with all SoundFonts (SF2, SF3) by default.
  • Will produce a smooth volume transition in and out of a hairpin.
  • Expressive instruments will still produce dynamics if expression mode is turned off in MuseScore.
  • Supported by commercial Orchestral Libraries, as most already support CC11 control for expression or allow user configuration to support any CC number.

Cons:

  • The use of fixed velocity would provide less control of an instrument's expressive parameters and would not allow sample switches (PP vs. FF samples) in the default sounds.
  • Dynamics won't work with non-expressive instruments due to the fixed velocity, so user would need to enable/disable this mode in the mixer.
  • SFZ and external instruments need to be manually configured to work with this method.
  • Consecutive dynamic marks not connected by a hairpin will sometimes cause an audible volume "bump" on transition to the second dynamic. This is caused by the sudden change in CC11 level while the previous note is still in its release phase. Please see method #1 above for a suggested fix.

3. CC11 for expression relative to velocity

This is the method developed by hpfmn in 2016 and improved upon by James Thistlewood. It is a rather ingenious way to achieve intra-note expression for synths/samplers where both note-on-velocity and CC11 affect note volume equally (as is the default case for SoundFonts).

The way it works, roughly, is this: Dynamics are controlled primarily by note-on-velocity with CC11 set to full volume (127). When a hairpin is encountered, notes throughout the hairpin duration are played at the maximum velocity required by any point within the hairpin/note span, with CC11 scaling either up to or down from 127 to create a smooth cresc./dim.

This expression method would need to be enabled or disabled in the mixer, since it is not compatible with non-expressive instruments. The appropriate mode would be chosen when an instrument is added to the score.

Pros:

  • The use of both CC11 and velocity would provide the maximum control--and therefore, realism--for expressive instruments.
  • Would work with all SoundFonts (SF2, SF3) by default.
  • Consecutive dynamic markings without a connecting hairpin will sound the most natural with this method.
  • Expressive instruments will still produce dynamics if expression mode is turned off in MuseScore.

Cons:

  • This method only works by default with SoundFonts, and is not supported by SFZ or any other samplers/orchestral libraries (to my knowledge--though it may be possible to configure some to support this method).
  • This method would need to be disabled in the mixer for non-expressive instruments.
  • Consecutive dynamic marks connected by a hairpin will sometimes cause an audible volume "bump" at the beginning and/or end of the hairpin. This is caused by the sudden change in CC11 level while the previous note is still in its release phase. This can be fixed by:
    • Make Fluid voices in release phase ignore further CC11 events.
    • Make sure that the note release event occurs prior to the adjustment of CC11 level.

Suggested Implementation

Method #1 (CC2 for expression with note-on-velocity sent as normal) is the strongest option, in my opinion. While it doesn't support expression in all SoundFonts by default, I believe the compatibility with high quality SFZ and VST sample libraries is more valuable for creating quality recordings than supporting the mass of subpar SoundFonts out there.

Method #2 (CC11 for expression with fixed note-on-velocity) is the weakest of the three, and should only be considered as a selectable option for additional sample library compatibility.

Method #3 (CC11 for expression relative to velocity) has certain flexibilities that neither #1 nor #2 possess, but it only works with the internal SoundFont engine, and produces audible "bumps" around hairpins.

Besides going with Method #1, I would also suggest adding the following control options for each channel in the mixer:

  • Expression Control: Off, CC1, CC2, CC4, CC11 (default = CC2)
  • Note Velocity: "Normal" or "Fixed" (user can choose fixed velocity, default = 127)

This would give users the most control for working with external sounds. For example, setting "Expression Control" to CC11 and "Note Velocity" to "fixed" would allow the user to easily use expression with any SoundFont (this is identical to Method #2 above).

Test SoundFont

I have created a SoundFont for testing any expression method implemented in MuseScore in the future. Download here. It contains the following presets, all based on the preset "020:048 Violins Fast" from MuseScore_General 0.1.3:

  • 000: default SF2 - This preset features the default SoundFont behavior, and can be used with expression methods #2 and #3, with #3 sounding superior on note attacks.
    • Velocity: volume, tone, sample switching and bow attack speed
    • CC1: vibrato
    • CC2: none
    • CC11: volume
  • 001: vel+CC11 (hpfmn) - This preset is optimized for hpfmn's method (#3), with the both volume and tone being adjusted by CC11, and the dynamic levels between velocity and CC11 perfectly matched.
    • Velocity: volume, tone, sample switching and bow attack speed
    • CC1: vibrato
    • CC2: none
    • CC11: volume & tone
  • 002: CC1 exp (Garritan) - this preset emulates the default expressive control used by Garritan Personal Orchestra (CC1).
    • Velocity: sample switching and bow attack speed
    • CC1: expression (volume & tone)
    • CC2: none
    • CC11: none
  • 003: CC2 expression - This preset is compatible with expression method #1.
    • Velocity: sample switching and bow attack speed
    • CC1: vibrato
    • CC2: expression (volume & tone)
    • CC11: volume
  • 004: simulated SFZ - This preset emulates default SFZ file behavior, with CC1, CC2 and CC11 having no effect on the sound.
    • Velocity: volume, tone, sample switching and bow attack speed
    • CC1: none
    • CC2: none
    • CC11: none

Comments

This is brilliant! Thank you so much for doing this.

A few notes:

Method #3 has more cons: due to the way velocity and CC11 are mixed, it takes a lot of processing power to work out how to mix them. This makes midi rendering times for scores increase a lot with size. The only solution to this would be to cache the segment minimum velocity available to use, and regenerate this cache when a segment is updated. But that's just an idea.

Also, the 'volume bump' problem seems fixable, but I doubt it's anything like an easy fix.

And a few questions:

  • Does the version of Fluid that we use support CC2 events? What about CC1 events? CC4? etc.

  • So your suggested implementation isn't really CC2, but instead to support ALL CCs (that are relevant)?

Other than that, thanks for this!

This would be the best way of allowing single-note crescendos/decrescendos, and other temporal changes.
I love all this, but i'm thinking the problem with implementation may be that MuseScore operates off of note events, whereas MIDI implementation requires time-based operation.

This wish is on everyone's Christmas list (everyone who uses MS for more than printing notation).

Good luck! ;)

I would suggest a variant of Method #1, in which velocity and CC are orthogonal to each other and control different aspects of the note articulation:

  • Velocity controls the attack quality, and only the attack quality (low velocity for slurred notes, medium velocity for normal notes and high velocity for accented notes)
  • CC controls the dynamics and only the dynamics (low CC for piano and high CC for forte)

In this way, note expression for sustained instruments is not a one-dimensional quality, but a two-dimensional quality. This is ideally also true for instruments that gain a sustained quality due to tremolo (for example, a cymbal tremolo or drum roll should be treated as a sustained instrument for this purpose). This method of control then allows for a clean separation of concerns, for example:

velo = 30, CC = 40: slurred piano note
velo = 60, CC = 40: tenuto piano note
velo = 80, CC = 40: normal piano note
velo = 100, CC = 40: accented or staccato piano note
velo = 120, CC = 40: heavily accented or staccatissimo piano note
velo = 80, CC = 80: normal forte note
velo = 100, CC = 80: accented or staccato forte note
velo = 120, CC = 80: heavily accented or staccatissimo forte note
velo = 30, CC = 120: slurred fortissimo note
velo = 120, CC = 120: heavily accented fortissimo or staccatissimo fortissimo note

etc

In this system, a high velocity together with a short duration would indicate that a staccaissimo/sautille sample should be used, and this would be true at any dynamics level.

On the other hand, a high velocity together with a long note duration would simply indicate that the initial attack is very prominent, like a fortepiano or even sforzato, and again this would be true at any dynamics level.

A low velocity indicates that the attack of the note is weak, and this should be used during a slurred phrase so that the notes can seamlessly blend together.

For simplicity, the non-sustained percussive instruments could retain the traditional meaning of the note-on velocity. So for example, the piano or harp would still control dynamics via velocity the old fashioned way, since no change is needed here.

I hope my explanation makes sense.

In reply to by s.chriscollins

I figured that this would be the best option for changing any instrument's volume, unless there is a way to change the volume of the soundfont instrument itself. IOW, if there is no way to change the INSTRUMENT volume, the next best thing is to manipulate the volume of the audio channel it occupies.
Hmmm...now that i think about it...will a MIDI CC implementation even be able to do that?

In reply to by s.chriscollins

After re-reading the OP, i better understand the author's intention, but instead of using CC7, he 's using CC2 or 11, which i thought were for different controls.
If this can be implemented to the main build, i'm guessing we need to then tweak the MuseScore soundfont to accommodate these features.

I'm also pleasantly surprised that no naysayers have chimed in to remind us that "MuseScore is not a playback application — it's notation software!" =D
I've always hoped that after our esteemed devs have nailed down the finer details of standard music notation, that they would then concentrate on implementing serious playback and instrumentation features. This is a big step in that direction!

In reply to by harbinger

The "MuseScore is a notation software" cop out has annoyed me many times when a feature was not implemented properly, but no one ever said that someone who wanted to write the code to implement playback is forbidden. I would love to see this happen quickly, but there one thing that must happen for Anatoly to merge the feature. It can't cause problems with backward compatibility to 2.x or 3.0 files or forward compatibility with 3.0.

The person who created this thread, I believe, is a MuseScore employee whose purpose is to improve playback. He's the one who created MDL. This has actually been a much sought after feature that Anatoly has in the past made clear he is in favor of.

In reply to by s.chriscollins

The reason to do this with a velo/cc combination, instead of two separate cc's, is that this control scheme is being implemented by existing orchestral libraries because it lends itself to live performance (left hand on the modulation wheel, right hand playing the keyboard). If MuseScore could be programmed to use the same control scheme (at least as a user preference), it could instantly gain the ability to drive these existing high quality expressive instruments.

I think Garritan was the first to totally commit everything to this control scheme, but it was not unheard of before (going under the names of DXF, dynamic crossfade), and is also now offered in the free VPO (virtual playing orchestra) as an alternative sfz programming over the same existing samples (they call it the performance patches). In my opinion, a soundfont designed for MuseScore as a 'modern default' should follow the same model.

It seems to me that a simple solution for controlling volume in MuseScore would be to use the MIDI volume controller (controller 7) as long as this is implemented on a per channel basis, so for example the volume level of the strings could be increased or decreased relative to the trumpets. This is probably supported by all MIDI sequencers and DAWs. This should adjust the channel volume up and down in cooperation with velocity. I.e. volume levels from different velocities would be proportionally scaled as MIDI volume is adjusted up and down.

To me the only disadvantage of this approach is that most MIDI sequencers use MIDI volume for track mixing. I.e. You make the whole track louder or software relative to other tracks by increasing or decreasing MIDI volume (controller 7). If controller 7 is embedded in the track data this would override the overall track volume setting. But a simple workaround for this would be that when exporting MIDI file from MuseScore have an option to convert Controller 7 to any other controller that the user wants to use. That way they can convert the continuous controller to whatever their MIDI sequencer or instrument library is using for expression.

Of course, what I would really want to see would be a continuous controller such as Modwheel (controller 1) tied to EQ depth (maybe a Low Pass Filter) so it does some timber change more like what a real instrument would do. But maybe that's asking too much of a program like MuseScore.

Do you still have an unanswered question? Please log in first to post your question.