MusicXML import and export do not support non-standard key signatures

• Sep 17, 2014 - 15:58
Type
Functional
Severity
S4 - Minor
Status
closed
Project

We are using Musescore to write and edit Turkish makam music, which includes microtonal accidentials and key signatures.
MusicXML supports these key signatures (http://www.musicxml.com/UserManuals/MusicXML/MusicXML.htm#EL-MusicXML-k… under "Non-traditional") except musescore cannot import or export these.

We will work on a patch as part of the CompMusic project.


Comments

Thanks for the comment. I tested saving an mscz file (in 1.3) after placing it on the score and it saved and opened fine. Is this something that's changed in 2.0?

Yes. Custom key sigs are currently not working, as described in that issue. By "custom", I mean, ones you create yourself in the key signature dialog, choosing your own arrangement of accidentals. You can create these in 2.0, but when you try to add to your score, they add as C major.

Alastairp,

you have assigned this issue to yourself. Does that imply you are a programmer and are implementing this feature with the intention to provide a patch for integration into MuseScore ?

If this is the case, will you meet the December 22 deadline for beta 2 ?

If this is not the case, please unassign yourself and I will take over.

Regards, Leon.

Hi,
Thanks for the note. This was being worked on by a student in our lab, but I don't think we could commit to anything by the 22nd. I've unassigned myself, it'd be great if you could take it on!

For the record,
1/ I met alastairp and his colleagues IRL today and asked them to send a couple of files they are generating with their own tool. Just to make sure that we are handling microtonal key signatures the same way since apparently there is no indication of which symbols to represent eg a 1,5 sharp.

2/ It seems musicxml2ly supports microtonal keysig.
https://github.com/music-suite/musicxml2/blob/master/test/musicxml/13d-…
Capture d'écran 2014-12-14 21.41.18.png

Attachment Size
Capture d'écran 2014-12-14 21.41.18.png 15.37 KB

As I am severely time-limited tonight, just a few quick remarks:

The sample score attached by alastairp does not pass schema validation (key-alter elements missing in the key, many elements out of order, empty accidental elements).

It would help a lot if someone could attach the required mapping between MusicXML accidental names (as defined in http://www.musicxml.com/UserManuals/MusicXML/MusicXML.htm#ST-MusicXML-a…) and MuseScore symbol names (as used in the master palette create custom key signature), plus the corresponding key-alter values (which are unfortunately required by MusicXML but apparently not well standardized).

Given this information completing the non-traditional key signature implementation is trivial.

The Lilypond example only contains the Tartini accidentals, which do seems standardized (apart from some minor differences in symbols used).

To be continued.

@Leon Vinken:
you can have a look here:
https://github.com/musescore/MuseScore/blob/master/mscore/importxml.cpp…
given the MusicXML documentation I tried to match the microtonal value from MuseScore with the microtonal values from MusicXML (some were already in place).

Then I tried to implement a function which tried to guess the microtonal value from the "alter" tag in simple cases:
https://github.com/musescore/MuseScore/blob/master/mscore/importxml.cpp…
I based these cases on the MusicXML example from the Lilypond MusicXML unofficial test suite.
I think alastairp could be of great help here (i.e. mapping between "alter" and microtonal accidental), and these function can be reviewed (so that issue #17508: [MusicXML] Not all accidentals imported can also be improved).

@ABL, thanks. I did not know about that. Note that the reverse transformation (MuseScore enum Accidental::Type to MusicXML accidental name) is also needed in exportxml.cpp (currently implemented as a switch). Same goes for mapping to and from accidental type and alteration value. To keep all those in sync, it is probably best to move the into a single table.

I'll rework and re-use your table accordingly and depend on more knowledgeable people to supply the missing alteration values.

Hi all,
Thanks for the notes about the failing validation. We've had an internal talk and it turns out we were missing a few parts because we didn't know which approach to take. We've made a decision and will fix our conversion scripts. We'll make sure we actually run the validation before we send an updated version!

Hi everyone,

I'm Sertan Senturk, working in the same group with alastairp . We have checked the default intervals given in the mus2 and I am attaching the mapping from the MusicXML and MuseScore symbols and the corresponding key-alter.

Please inform us if there is a problem with mappings or there are further things to clarify.
Cheers,

Attachment Size
musicxml_musescore_alter_mapping.txt 442 bytes

Btw we also have a slight concern. Do you need these values to put a default microtonal interval if is not defined in the MusicXML sheet, or do you plan to you use the a single for a given accidental.

In the second case, it means all the microtonal scores will sound "makamish," which might not be desirable to a lot of people. Ideally people should be able to adjust the manually. In fact even we need to change the mappings in certain situations (e.g. different scales constitute different theoretical intervals).

I guess it is not possible to make all these changes before the deadline, but I just wanted point out something that might be problematic in the future...

Indeed, even though I am no expert on microtonal music, it seems simply using your table would work only for the specific case of makam and not for other types of microtonal music. I will need input from the experts on what exactly is needed in MuseScore.

With respect to the deadline, it is up to the core developers to decide when beta 2 is created, what goes in and what can still be changed after beta 2. I do expect we will try to get a certain level of microtonal support in MuseScore 2.0, but I am not yet sure what exactly it will be.

It's pretty sure that we will not have playback for beta2. So let's stay on the graphical level for now.

The only data points for MusicXML import/export we have so far are Lilypond and the couple of thousands scores provided by Sertan

As far as I know we currently have support for any accidental in the music and for custom key signatures. Both with some limitations, like they do not transpose. It would be great if we could have MusicXML import/export for these too.

MusicXML import of accidental is based on a guess on the alter value, considering half steps only and then overwritten based on the value of the accidental tag.

For key signature import, I would propose we do the same. Alastairp example only contains key-accidental and we already have a mapping. Lilypond example use key-alter value only and 0.5 steps.

Regarding export of accidentals, we export accidental based on the symbol and alter value based on an hardcoded 0.5 steps mapping. We could also do the same for key signature. We could also ignore key-alter like in example of Sertan but it seems that we would then create invalid MusicXML since the key-alter element is mandatory (?).

The mapping table implemented by @ABL in importxml.cpp matches exactly with the table supplied by Sertan Senturk, so importing/exporting correct key-accidental values is no issue.

With respect to the spec: key-step and key-alter are required, key-accidental is optional. This is unfortunate, as it implies we must also export key-alter. The difficulty is that there is no single correct mapping from symbol to alteration. E.g. in Tartini-Couper notation quarter-sharp typically means 0.5, while in makam music (according to Sertan's table) it is 0.235.

On import we could ignore the key-alter and prefer key-accidental (if present), but on export we'll have to a key-alter value.

For the time being, I think the best solution is to add key-accidental to the export, but leave out key-alter for the microtonal accidentals (as the correct value is unknown anyway). This is, strictly speaking, invalid MusicXML but will import correctly after warning for the missing values. It will only affect files with microtonal accidentals and is compatible with Sertans example.

On import we can use key-accidental to determine the microtonal accidentals to import. Once again, this is compatible with Sertans example.

Will implement this during the weekend.

Note that I feel the structural solution would be to allow changing the microtonal accidental to alteration value in the GUI. This enables both correct playback and MusicXML export.

Exporting 0.5 steps is correct only for Tartini / Couper notation, it is not correct for makam music. Alternatively, sing Sertans table redefines sharp, flat and the Tartini accidentals, breaking pitch for everyone else.

In order to do this correctly, we really need a GUI to define the pitch change for any given accidental.

If we really want to export known incorrect alteration values, the safest strategy IMHO would be to export 0.5 steps for Tartini / Couper and 0 for the others.

Notes:
Whether this is acceptable or not depends on how the receiving applications handle this and which applications are considered the most important. For MuseScore the key-alter value is not used anyway.

With respect to key-alter: In principle I really object to exporting invalid MusicXML, but in this case it probably only leads to a few schema validation errors without further harm. It does convey the message: we simply do not know the correct value.

Yes, that's what I also thought. The order is not specified in the MusicXML spec anyway, but Finale NotePad displays them (from left to right) in file order.

Note that I would appreciate any comments (especially about compatibility) from @alaisterp and friends.

There is obviously one thing that we miss... if an accidental is "at the key sig" it shouldn't appear again in the score. I guess we will need some more code in core to make that work.

I think we would prefer to see musescore export valid XML, even if that means that we cannot support makam music until some sort of dialog is created to set the alter values. So my vote is to write 0.5 if that is correct for some types of music.

Please don't consider our XML as valid for our purposes - after talking to some of the other researchers we didn't validate the XML. This means that any errors are because we created bad XML, not because we had a reason to go against the schema. In the new year we will be fixing our conversion tool.

I'll confirm the key signature order. It may be easier for us to change so we write from left-to-right as finale does.

As of 2.0 beta 2, MuseScore exports 0.5 steps for Tartini / Couper accidentals and 0 for the others (such as those used in makam music).

With respect to comment #26, just to be clear about it: I am not going to make that code change as I do not know anything about that part of MuseScore.

Please let me know if any more changes are required in the MusicXML import / export. If not, I assume we can close this issue.

Just checking: I haven't seen any feedback on this for some time and would like to hear, especially from @alatairp and friends, if the MusicXML import/export of non-standard key signatures now works as expected. In that case I'd like to close the issue.

Note that I believe everything that could be done in the scope of MuseScore 2.0 has been done, so we may have to close this issue anyway and open a new feature request for required changes post 2.0.

Hi Leon
Sorry for the delay, we've just had a look at current master.
For now this is good enough for us. Of course, we would like to see the ability to set custom key-alter values, but we understand that this won't fit in the scope of 2.0.

Thanks for the work!