GSoC 2016 - Week 11 - Automatic Real-time & PortMidi

Posted 8 years ago

This was my 11th week working on note entry with MuseScore for Google Summer of Code. Most of this week was spent making small improvements here and there and solving a few issues with automatic real-time mode.

This week’s summary:

  • Workaround for PortMidi bug on Windows and Mac
  • Metronome unit tests
  • Instant note entry in automatic Real-time mode

Still to do:

  • New rhythm simplification and voice extraction implementation
  • Delayed preview for automatic real-time
  • Test user feedback

PortMidi

PortMidi is a library that MuseScore uses to communicate with MIDI hardware and software on Windows and Mac. (On Linux MuseScore uses ALSA instead because PortMidi has a relatively limited feature set due to the need to support many Midi implementations across multiple operating systems.) Since my development system is Ubuntu Linux I had not noticed a bug in my new automatic real-time mode that only affects PortMidi.

MuseScore runs on a number of different operating systems, and each operating system has its own method of communicating with MIDI devices. Rather than having to support each method individually, MuseScore uses the PortMidi software library to provide a common MIDI interface on the Windows and Mac platforms. (On Linux MuseScore can use either ALSA or JACK, both of which provide more advanced capabilities than PortMidi - PortMidi’s features are limited precisely because it has to support multiple platforms).

Since my development system is Ubuntu Linux I had not noticed a bug that only affects PortMidi where attempting to enter a single note in automatic real-time mode resulted in an endless series of tied notes. The problem was caused by the fact that PortMidi (unlike ALSA and JACK) is unable to “inform” MuseScore of new MIDI input - it is up to MuseScore to “ask” whether there have been any new input events. My original implementation of auto mode involved a synchronous loop that would send a click, wait for a short period listening for MIDI input, and then send the next click. This function worked with ALSA, but with PortMidi it was blocking MuseScore’s periodic requests for new MIDI events. To prevent this issue I had to replace the synchronous loop with an asynchronous system of timers using Qt’s system of signals and slots.

Unfortunately, switching from the synchronous loop to the asynchronous calls made the metronome clicks more sporadic and has decreased the accuracy of automatic real-time mode at higher input tempos. It may be that the only way to alleviate this is to delay adding any notes to the score until the end of each measure so that the time spent in signal-blocking functions is minimised.

Instant note entry in automatic real-time mode

A positive consequence of using timers is that gives more control over how to enter the function from outside. By default there is a 750 ms delay between clicks (80 clicks-per-minute). The question is, when you first press a key should a click be triggered immediately or after 750 ms? I originally implemented it so that the delay comes first, giving the user time to prepare for the first click or even to release the key and cancel the note. This is still the case if the user presses the “realtime-advance” shortcut key, but not if the user pressed a note key. The result of this is that a user entering a semi-breve (whole note) counts beats like this:

Press … Two ... Three … Four …. Release

Instead of…

Press … One … Two … Three … Four …. Release

This behaviour appeared to me to be the most intuitive. A side effect of this is that it is now possible to enter a rest by pressing and immediately releasing a note key. This is because the key press immediately triggers a click, but there is a very small delay of 100 ms between the click being sent and the advancement actually occurring. This is intentional to catch any notes that the user might enter slightly late, but it means that if the key is released in this time then a rest will be entered instead. I could correct this “bug”, but I think it is actually quite useful to be able to enter a rest with the MIDI keyboard.

Comments

• May 18, 2017 - 02:33

Sorry for bumping, but could you clarify, when you say you used timers and signals and slots, does that mean you set up a separate thread for listening to PortMidi input? If so, could you point me where in the code you do this?

Are you aware that you could setup a separate thread dedicated only to receiving portmidi messages? For example as done here, which is invoked every millisecond, so it can know with about millisecond resolution when notes arrive: https://github.com/philandstuff/portmidi/blob/master/pm_test/midithread…

Also does your code take into account the precise latency of MuseScore's internal synth (since musescore is very slow to sound notes, a human who plays with the midi will always be off by that latency), and if so where is that in your code? I ask, because sometimes I've notice glitches using your input on PortAudio where I pressed the a key according to when I heard the click but it doesn't go into the right spot.

Reply

• May 18, 2017 - 03:06

Apparently [[https://doc.qt.io/qt-5/qtimer.html#timerType-prop|the default timer type is Qt::CoarseTimer]], which according to https://doc.qt.io/qt-5/qtimer.html#accuracy-and-timer-resolution is:

For Qt::PreciseTimer, QTimer will try to keep the accurance at 1 millisecond. Precise timers will also never time out earlier than expected.

For Qt::CoarseTimer and Qt::VeryCoarseTimer types, QTimer may wake up earlier than expected, within the margins for those types: 5% of the interval for Qt::CoarseTimer and 500 ms for Qt::VeryCoarseTimer.

That is from the intial commit 5 years ago. Maybe it is time to upgrade this timer to a Precise 1 ms timer, since we are now doing real-time midi input.

Reply

• May 18, 2017 - 03:10

even with your comment: "My original implementation of auto mode involved a synchronous loop that would send a click, wait for a short period listening for MIDI input, and then send the next click. This function worked with ALSA, but with PortMidi it was blocking MuseScore’s periodic requests for new MIDI events. To prevent this issue I had to replace the synchronous loop with an asynchronous system of timers using Qt’s system of signals and slots." that means with the timer as it we are still limited to 21 ms input note resolution maximum.

Reply

• May 18, 2017 - 03:14

I think that timer was initially set as 20ms Coarse timer because there was no concern about being able to receive midi notes in realtime. But I think real time necessitates this be updated to 1ms precise. The problem occurs when the user is inputting a note near the beat quantization edge, so this 21ms inaccuracy can be magnified, since the note may be mistakenly put in the wrong quantum.

Reply

• May 18, 2017 - 04:00

regarding my squesion about accounting for realtime latency, it seems you have preferences variable initialized to realtimeDelay = 750 and are using that in ScoreView::midiNoteReceived() for "realtime advance"...I haven't looked closer into that. FYI, what I'm wondering about it whether the latency has been set exactly. Also there maybe a use for a little .ui which gets the user to press keyboard when hear notes, and the measured delay will be saved as a roundtrip latency, and that lantency could be used to offset notes taking into account the latency measured from the user's hardware setup.

Reply

• May 18, 2017 - 04:10

and I notice I see in ScoreView::triggerCmdRealtimeAdvance() you are applying 100ms timer to account for:

  // The user will want to press notes "on the beat" and not before the beat, so wait a
  // little in case midi input event is received just after realtime-advance was called.

To me I think this hardcoding this delay is a bit hacky...I personally would prefer a calibration to figure out what this delay would be. I suppose this is a reason why I experience the timing glitch.

Reply

• May 18, 2017 - 13:38

This 100ms timer is there mainly to account for the user pressing the key late rather than to account for delays due to MuseScore's synth, but I suppose it would help with that too.

Calibration might be nice. I didn't want to add too many things to the preferences though, and then there is the issue of people wanting different delays for manual vs automatic realtime.

Also, I already added an option called "Realtime delay" to the preferences, which means something else. (It's the time between clicks. I could have used the inverse and called it "input tempo", but it's not really a tempo because it is measured in notes-per-minute, not beats-per-minute).

Reply

• May 18, 2017 - 14:03

ok, thanks for clarifying.

Re, calibration, I can understand your points about not wanting to overwhelm the user with preferences.

Regarding the name "Realtime delay"...that should technically be called a "period" according to what you are describing it. I guess i'm a little late to suggest a name change, but what you describe sounds like a "Realtime Sampling Period".

Reply

• May 18, 2017 - 14:48

That's getting a bit technical. Maybe "Realtime click period".

The 100ms delay could be called "click offset", "advance delay" or even something more explicit like "delay between a click being sounded and the score advancing".

Reply

• May 18, 2017 - 04:19

I'm also noticing that in ScoreView::midiNoteReceived(), incomming midi notes are enqueued into the main, but there is no timing information about when MuseScore initially received this message that gets stored in this queued event. It seems to me that first thing when receiveing a midi note from driver, ought to save a timestamp, that way any unpredicatble delays from the queueing and the reading don't add even more noise to the quantization.

Reply