Improved playback architecture

• Apr 1, 2016 - 02:40

I'd like to propose a change to the playback architecture. It would enable a lot of new features, including several that have been requested recently on other threads. I'll first describe it at a high level, and then if people think it sounds like a good idea, I can go into more detail about how it might work.

The idea is to create a new abstraction layer for translating score elements into midi commands. It would let you have a file describing the features and behavior of a particular synthesizer. Then you could load that file and everything would "just work". In particular, it would describe three aspects of playback: instruments, articulations, and dynamics.

MuseScore has built in support for GM soundfonts, so if you use one it automatically selects the right sound for every instrument in the Mixer. This feature would let it do the same for non-GM soundfonts. It would also work for external midi synthesizers. Just load one file and it configures everything for you, including sending Program Change messages as needed.

Currently, MuseScore supports a very small set of articulations. For example, stringed instruments have three: normal (whatever that means), pizzicato, and tremolo. But most commercial orchestral synthesizers have a lot of others: marcato, spiccato, detache, etc., each with or without mutes, not to mention distinguishing between up bows, down bows, and slurs. This would let you use those.

Lots of mechanisms can be used for selecting articulations. With GM, each articulation is a different program, so you select them with Program Change messages. But a lot of synthesizers use keyswitching, where you press keys outside the normal range of the instrument to select articulations. Or they may use a midi controller for that purpose. Sometimes they use velocity to smoothly blend from legato to marcato. Slurs can be indicated with the sustain pedal, or by setting the velocity to 0, or in other ways. So the file needs to describe what articulations are available, and also how to activate them.

Dynamics can also be indicated in multiple ways. GM synthesizers use velocity for that purpose, but others use the expression controller, or the mod wheel, or other mechanisms. And this can even vary between instruments within a single synthesizer. So percussion instruments might base dynamics on velocity, but stringed instruments might base it on a controller.

So my idea is that you would load a file describing a set of instruments, and the available articulations for each one, and the mechanics of how to play them, and then MuseScore would take care of everything for you. Does this sound like a good idea? If so, I've thought a lot about how it might be implemented, and I can provide details on my ideas.


Comments

MuseScore would take care of everything for you. Does this sound like a good idea?
Do we really need to answer this question :)

Show us some code! You can find MuseScore source code here https://github.com/musescore/MuseScore and information to compile on your platform here https://musescore.org/en/developers-handbook/compilation

Note that what you describe is a little bit implemented in instruments.xml, of course it's more tied to GM but you can defined which program an instrument will play on, the gate time of articulations, the velocity change of articulations. You can also add any MIDI message for text and articulations. Nothing is done for Velocity vs Expression but proper crescendo playback is a popular request so I guess we will do something around this soon.

In reply to by Nicolas

Show us some code!

Yes, it does come down to that, doesn't it? :) I'm certainly willing to help implement it, but I don't want to commit to doing the entire thing myself. In particular, I can create the abstraction layer, but integrating it into the rest of the code might be better left to the core developers. It will involve a lot of small but significant changes to existing code, and also affect the UI in lots of small ways. That seems like something that should be done by someone who knows it much better than I do.

Anyway, there's a lot of design that needs to be done first. I'll start by writing up a much more detailed proposal and post that. Is this forum the best place for technical discussions?

In reply to by peastman

Probably the best plan would be to open a feature request in the issue tracker. That way it is focussed more on the MuseScore Development community who may be better able to understand the discussion once it delves deeper into the technical aspect of implementing this.

GM, by the way, is incapable of implementing articulations etc in the manner you describe. Patch names are limited to the GM sound set, and there is no provision for alternative sounds within a Patch Number.

It would be possible in GM2 which has a much expanded instrument structure, including alternative sounds within a Patch Number using Bank Select messages.

In order to implement this, however, we would need to develop a system of classifying which Banks would produce which variant of the sound. Yamaha's XG GM extensions certainly define this across their banks, and I expect the Roland GS system does as well, although IME the Roland system is not as well documented as the Yamaha.

The key to this IMO is to get the MidiAction dialogue working properly. I understand that igevorse started work on this as part of his GSOC project, but that so far none of his PRs have been merged, so maybe looking at his code would be a good place to start.

In reply to by peastman

I agree. Although the "Tehcnology Preview" forum here isn't a bad place to discuss high level design (how it might look to the user), since people other than developers might well have opinions on that. Once it gets into implementation details, though, probably better for the mailing list.

In reply to by peastman

The developer mailing list doesn't seem to get used as much as it should.

But it is as good a place as any.

I see this initiative as very much a joint effort between the progamming team, sound designers developing for MuseScore such as String Contrabass and myself, and the users who require advanced playback but also with the technical knowledge to precisely specify what they require.

It has been clear for some time to anyone following the forums that MuseScore needs to go up a level in terms of playback, but we need to get the design right before we start to implement it.

Do you still have an unanswered question? Please log in first to post your question.