Deleting soundfont from Fluid Synthesizer while in processing event.type of CTRL_PROGRAM may cause segfault
Repro steps: Open the synth window and go to fluid synth tab. While loading a big score, delete all existing soundfonts.
Result: SegFault in FluidS::Preset::loadSamples thread:
1 FluidS::Preset::loadSamples sfont.cpp 151 0x73dde5
2 FluidS::Channel::setPreset chan.cpp 224 0x73d15e
3 FluidS::Fluid::program_change fluid.cpp 344 0x735fc2
4 FluidS::Fluid::play fluid.cpp 181 0x7355f8
5 Ms::MasterSynthesizer::play msynthesizer.cpp 92 0x72cb39
6 Ms::Seq::putEvent seq.cpp 1429 0x4dc14a
7 Ms::Seq::processMessages seq.cpp 579 0x4d8ed2
8 Ms::Seq::process seq.cpp 745 0x4d9860
9 Ms::paCallback pa.cpp 50 0x681c0b
10 NonAdaptingProcess 0x65344e3d
11 PaUtil_EndBufferProcessing 0x6534602f
12 ProcessingThreadProc *4 0x653520c3
13 KERNEL32!BaseThreadInitThunk 0x762b62c4
14 ntdll!RtlSubscribeWnfStateChangeNotification 0x77a00fd9
15 ntdll!RtlSubscribeWnfStateChangeNotification 0x77a00fa4
16 ??
because it is trying to access data from a FluidS::Zone which has been deleted by another thread:
1 ntdll!RtlCompareMemoryUlong 0x77a226d0
2 ntdll!WinSqmEventWrite 0x77a32e2a
3 ?? 0x2a1dc118
4 ntdll!RtlFreeHeap 0x779e8bd8
5 ntdll!RtlpNtSetValueKey 0x77a7fa72
6 ntdll!RtlFreeHeap 0x779e995d
7 ntdll!RtlFreeHeap 0x779e8bd8
8 msvcrt!free 0x745377c5
9 ?? 0x22d80000
10 FluidS::Zone::~Zone sfont.cpp 409 0x73eec5
11 FluidS::Instrument::~Instrument sfont.cpp 436 0x73f0b1
12 FluidS::SFont::~SFont sfont.cpp 69 0x73d7f8
13 FluidS::SFont::~SFont sfont.cpp 71 0x73d89d
14 FluidS::Fluid::sfunload fluid.cpp 730 0x73768f
15 FluidS::Fluid::removeSoundFont fluid.cpp 675 0x737480
16 FluidGui::soundFontDeleteClicked fluidgui.cpp 152 0x746b1f
17 FluidGui::qt_static_metacall moc_fluidgui.cpp 182 0x749b29
18 ZN11QMetaObject8activateEP7QObjectiiPPv 0x68c03046
19 ZN15QAbstractButton7toggledEb 0x226c2d1f
20 ?? 0x26d79080
...
These two threads need to be mutually-exclusive in this situation. I notice that Fluid::removeSoundFont() does protect itself with a mutex.lock():
bool Fluid::removeSoundFont(const QString& s) { mutex.lock(); foreach(Voice* v, activeVoices) v->off(); SFont* sf = get_sfont_by_name(s); sfunload(sf->id()); mutex.unlock(); return true; }
But for some reason someone commented out the mutex lock in Preset::loadSamples():
void Preset::loadSamples() { // sfont->synth->mutex.lock(); if (_global_zone && _global_zone->instrument) { Instrument* i = _global_zone->instrument; if (i->global_zone && i->global_zone->sample) i->global_zone->sample->load(); foreach(Zone* iz, i->zones) iz->sample->load(); } foreach(Zone* z, zones) { Instrument* i = z->instrument; if (i->global_zone && i->global_zone->sample) i->global_zone->sample->load(); foreach(Zone* iz, i->zones) iz->sample->load(); } // sfont->synth->mutex.unlock(); }
According to https://github.com/musescore/MuseScore/blame/master/fluid/sfont.cpp that commenting out was done prior to werner's initial git commit. Without knowing the prior git history, I cannot say why that mutex lock was removed.
Comments
I have reproduced on my a second try. Note that the Fluid Synth window will not permit you to delete while the score is loading. But right after the score loads, it will allow you to press delete. It seems that the first or earliest Seq::process sends out some initial messages such as CTRL_PROGRAM which I guess are indicating to fluid to reconfigure for the new score's instrumentation. Anyway, I'm going to uncomment that mutex lock and see if it works...
BTW, I'm on nightly 3.0 Windows with PulseAudio output.
I believe the rational for commenting out the mutex.lock was because Seq::process is supposed to be in a realtime-process.
Scratch idea in comment #1 about uncommenting that mutex lock...doing so allows me to produce a deadlock situation.
I'm thinking better way would be to prohibit deletion of soundfonts while a score is sending out these initial CTRL_PROGRAM messages.
I can prevent the segfault by changing the mutex from a lock into tryLock():
if(!sfont->synth->mutex.tryLock())
return;
hmm...that solution in #6 was not quite the fix. Although everything seems to work (i.e. can't hear anything during playback because the soundfont was removed), the problem is that after I re-add the soundfont, my I hear static noise for those previously initialized instruments (presumably because it thinks it had loaded a soundfont and didn't re-initialize the soundfont for those instruments/zones on the reload, so was left playing random computer memory...and I end up being able to produce a segfault later on because that instrument/zone's data is now all garbage.)
I'm thinking instead of just returning from loadSamples on failure to tryLock, I think I also need to make sure that the data structure returns to an uninitialized state.
I think I need to produce a boolean result from loadSamples, and then if there was a failure loading, then don't set _preset = p in Channel::setPreset().
maybe should I propagate a false return all the way up? I would to change all the callers and their callers from void returns into bool returns.
Argh...this is getting rather complicated. I'm at the point now where I just want to disable the adding/deleting soundfonts while these CTRL_PROGRAM messages are being processed.
Actually, this bug occurs on 2.1 release too: 871c8ce
The interesting thing is that hot deleting the last soundfont while playback is engaged somehow works fine...the playback cursor looks like it hangs, but then jumps to where it should be based on the time that was needed to delete the soundfont.
Well, I've spent 3 hours thinking about this but haven't found a perfect solution and am officially giving up. :(
My best attempt was #6:
https://github.com/ericfont/MuseScore/commit/af724bdcef5c64cb06de15088f…
but that resulted in problems down the line when I re-added the soundfont.
Someone more knowledge about needs to figure out a solution to make Preset::loadSamples() behave properly when the soundfonts are being added/deleted. Either error code propagation or initialization of the zones/instruments would be necessary if tryLock wasn't successful.