GSoC 2016 - Week 3 - Realtime vs Semi-Realtime

This was my third week working on note entry with MuseScore for Google Summer of Code. Having managed to get the basic realtime note entry working last week, this week I started the task of analysing the user's input to make it look more like actual music.

This week’s summary:

1. Simplifying note durations in semi-realtime mode
   a) Attempted via MIDI import - not much luck so far
   b) Achieved within MuseScore
2. More ideas for Realtime and Semi-Realtime modes

Still to do:

1. Voice extraction
2. Realtime MIDI (raw input via MIDI import)
3. Performance and usability enhancements from last week

Simplifying note durations

Last week I managed to achieve a realtime result that visually resembled how an imported MIDI file used to look back in MuseScore 1.3 - a series of short notes or rests, with longer notes formed from multiple short notes tied together. MuseScore’s MIDI import algorithm has improved dramatically since 1.3, thanks mainly to the efforts of trig-ger, and now imported MIDI files look much more like sheet music. My original plan was to borrow trig-gers’s algorithm for simplifying note durations and extracting voices, but this has proven to be more difficult than I first assumed. To be useful for realtime input, MuseScore’s existing renderMidi and importMidi functions would have to be adapted to work on part of a score, rather than the full score. This is not straightforward because things like the time signature and key signature are only set at the beginning of a MIDI file. Problems also arise due to information being lost in the conversion to the MIDI format. Going via the MIDI format is not impossible, but I decided to try to keep within MuseScore’s native format. The advantage of using this method is that the code will be useful for other purposes, such as note adjustments following a time signature change.

Simplifying the durations wasn't easy even within MuseScore's native format due to the need to keep track of ties, etc, but I managed to get it working in the end. The next step will be to separate the different voices to deal with overlapping notes of unequal length.

Note entry in Semi-Realtime mode before and after simplification of durations. Voice separation not yet implemented.

Semi-Realtime vs Realtime

I have done some more research into realtime note entry in other music notation programs. Most of the programs offering realtime note entry do so by recording the note_on and note_off MIDI timings and saving them for later analysis. The raw timings are adjusted to follow the click rate and then sent through an algorithm similar to MuseScore’s algorithm for importing MIDI files. This method has a number of advantages, but it has a severe disadvantage in that it is not possible to display anything until the end of each measure’s worth of input. This means that the user cannot see what they have played until the end of the measure, at which point it is too late to change.

In my semi-realtime approach, the MIDI timings are not recorded. Instead, new notes are added to the score immediately on each metronome click based on whichever keys are currently pressed. This means that the user can see what they have played immediately; they don’t have to wait until the end of a measure. In fact, there is even a preview of the notes just before the click so the user can switch to a different note if they accidentally pressed the wrong key. However, this method does have some disadvantages compared to recording the raw MIDI values:

1. The quantisation level (shortest note) cannot be changed retrospectively.
2. It is not possible to enter triplets.
3. The raw values cannot be used for playback.

The last point is an interesting one. Apparently Finale gives the option to save the raw MIDI information and it use it for future playback rather than calculating a MIDI information from what is written in the score. The idea is to give the performance a "human" feel, so it doesn't sound like it is being played by a machine. I personally would rather have the exact computer performance, but I can see how that could be useful for playing custom articulations and ornamentation. This would probably be better achieved by recording the ornaments individually rather than recording the whole score, and I think that is outside the scope of my project (for the time being, at least!).

It's not clear how accurate a fully realtime note entry mode would be, but I think it is worth experimenting with for the sake of triplets. I can see this as being a separate mode to semi-realtime, where semi-realtime adds notes straight to the score and realtime takes you via the MIDI import algorithm.