GSoC 2016 - Week 2 - First attempt at Semi-Realtime MIDI

This was my second week working on note entry with MuseScore for Google Summer of Code. Having figured out MuseScore’s existing implementation of note entry last week, this week I was able to make real progress with the new modes. In fact, I think it’s time for another video!

This week’s summary:

1. Rhythm input mode nearly complete!
2. Basic implementation of auto and manual semi-realtime modes
3. Changes to the audio sequencer to allow metronome ticks in note entry mode

Still to do:

1. Some usability questions for rhythm input mode
2. Performance improvements needed for automatic semi-realtime mode
3. Simplification of rhythms and voice extraction in semi-realtime modes

Semi-Realtime modes

The bulk of my work over the last week has been on my semi-realtime modes for entering notes by playing the piece naturally on the keyboard. I managed to implement basic input for both the automatic and manual version of this mode. Entering notes in these modes currently results in a series of tied notes. These tied notes can be combined by exporting the file in the MIDI format and then re-importing it into MuseScore, and one of my tasks for the coming weeks will be to make this process automatic.

Preview of note entry in Semi-Realtime mode before simplification of rhythms and voice extraction.

I have found that the manual version of semi-realtime is quite effective for entering notes. In the manual version, the sustain pedal is used to tap beats and the user simply plays in time with the beats. Pressing a midi note enters a note with the currently selected duration (e.g. a crotchet or “quarter note”), and then each press of the sustain pedal ties on another note of the same duration. The user sets the pace with the pedal, so they are free to speed up or slow down depending on the difficulty of the piece.

In the automatic version, the “beats” happen at regular intervals like a metronome. I currently have it set up so that the beats do not start until the user user holds down the sustain pedal, and they continue for as long as the pedal is held. Unfortunately, there appears to be some lag due to the recalculation of the score layout on every beat, so in practice the intervals are not very regular. This means that the automatic version of semi-realtime is not very usable at the moment, but I think that this will not be an issue once the new layout optimisations are in place. In a worst-case scenario, automatic realtime entry could be done in a special view with fixed bar lengths to avoid any recalculations during note entry.

Somewhat to my surprise, the most difficult aspect of implementing semi-realtime mode so far has been getting MuseScore to play an audible "tick" sound on the beat. My plan was to borrow the metronome sounds that are used in playback mode, but this proved to be quite a challenge! The metronome is not a MIDI instrument as I first assumed, but it is actually made up of hard-coded raw audio samples that were never intended to be used outside of playback mode. In playback mode the sequencer intercepts the MIDI metronome events and adds the raw samples directly to the audio stream before passing the note events to the MIDI synthesizer. Outside of playback mode the sequencer is bypassed and MIDI events (e.g. Note ON when the user adds a new note in note input mode) are sent straight to the synthesizer, so there is no opportunity to add the metronome sound. I managed to get the sequencer involved again in note entry mode so that the metronome sound is available, but I had to work out how to manage various memory locks designed to prevent multiple threads from modifying the audio stream at the same time.

Once I had the metronome working I began experimenting with the tick sounds. There are two tick sounds with slightly different pitches: "tick" and "tack". In playback mode the ticks are determined by the time signature, with a tick for the first beat of the measure and a tack for the remaining beats. For example 4/4 sounds like this: "TICK... Tack... Tack... Tack... TICK... Tack... Tack... Tack... TICK... Tack... Tack... Tack...", etc. In Semi-Realtime note entry mode the user can choose the length of each beat, and might select something smaller than the actual beat. In my current implementation I use the volume of the "tacks" to indicate which beats are "real" and which are "extra". For example, if the user chooses to enter quavers (eighth-notes) then there will now be 8 beats - 4 real and 4 extra. The extra beats are played more quietly than the real beats, like this: "TICK... tack... Tack... tack... Tack... tack... Tack... tack... TICK... tack... Tack... tack... Tack... tack... Tack... tack..." etc.

Rhythm input mode

This is a new mode for MuseScore where pressing numeric keys enters notes of different duration (quarter notes, eighth notes, etc.) but all of the same pitch (middle C). The pitches are then set in Repitch mode. I introduced Rhythm mode last week and made an early attempt at implementing it. This week I managed to sort out entering rests and dotted notes, so that’s rhythm input mode essentially finished! However, there are still a few things to consider:

• Providing audible feedback about what the user just entered
• Rests and dotted notes - room for improvement?

It would be helpful for users if they didn’t have to look away from the score they are transcribing to know that they entered the right rhythm. This is where audible feedback comes in. Normally MuseScore plays the pitch of note, but clearly that is not useful here. As I said last week, Rhythm input was inspired by a similar mode from Denemo - another music notation program. Denemo uses percussion sounds to indicate the note duration, so it would probably make sense for MuseScore to do something similar. Currently the synthesizer can only deal with MIDI instruments that are actually in the score, so I will have to create some kind of dummy percussion instrument to play rhythm noises, or alternatively I could use hard-coded samples as is done for the metronome.

I have Rhythm mode set so that pressing the period key (.) toggles dotted rhythms on or off, and pressing the zero key (0”) toggles between entering notes or rests. The question is, should this change only last for the next note/rest entered, or should it be permanent until the button is pressed again? Also, might some users like the option to press and hold the button while entering dotted notes/rests and then release it when they want to return to entering normal notes? Making the toggle only last for one note is easy, but the press-and-hold method will require more work because MuseScore’s shortcut mechanism can currently only detect when a key is initially pressed, not whether it is still being pressed or whether it has been released. However, having such an ability may prove to be useful for other modes too.

Personal blog post: http://peterjonas.net/blog/musescore/gsoc-week-2-first-attempt-at-semi-…