GSoC 2018: Score Comparison Tool
Hi all,
I am Dmitri Ovodok, and this topic is intended for technical discussions related to my score comparison tool project for GSoC 2018.
The goal of the project is a creation of the tool which will be able to show user differences between two scores in textual way. A work on visual flavor of the tool will also be started if time allows.
The project includes three main parts:
1. Making changes to MSCX file format to make it more suitable for automatic textual comparison tools.
2. Making a tool built in MuseScore that will compare scores and show the differences in textual form. Data on differences will probably be obtained using existing textual comparison tools.
3. Starting a work on visual comparison tool (optional, depends on the amount of available time).
More details will be added in subsequent posts.
A blog post introducing this project: https://musescore.org/en/user/1794009/blog/2018/04/29/gsoc-2018-score-c…
My Github fork of MuseScore: https://github.com/dmitrio95/MuseScore
Comments
The first topic which I wanted to discuss is about changes which will be made to MSCX file format. As stated in the starting post, the primary goal of such changes is making MSCX format more suitable for usage with automatic textual differences tools, which will both make subsequent work on the project easier and allow advanced users to use those textual differences tools and other tools that make use of textual differences (e.g. version control systems).
One of the main problems making automatic comparison of MSCX files difficult is an absolute numbering of some elements throughout the file. This leads to large diffs between two files with only a few changes between them (like deleting/addition of slur) due to renumbering of lots of other elements not related to those changes. While in some cases (like measures) numbers can be just omitted without any other changes, in most of other cases (like slurs, beams etc.) numbers are really used for identification purposes. One of the possible options to solve these IDs problem is using IDs chosen either by random (preserving them during load/save cycle) or based on the hash of the content of the file. Still I think that both these options will lead to other problems like introducing a dependence of the resulting file on the order in which changes are made. So I wanted to share my thoughts on this problem. While these thoughts are are based on very brief investigation of MSCX format and should still be reconsidered, some general ideas may still be useful.
My proposal is based on some more semantic sort of identification of elements. If a human needs to identify any slur of beam on the score he would say something like "The slur from that C note to the A note in the next measure" or "the beam which cover the next two notes". Sorry if my musical English is not so correct :) Anyway the idea is that we should make more use of semantic properties of the score element which will allow to uniquely identify them based on local state of the score only or even avoid the identification problem completely (as some information on the location of the element is available from the nesting of the corresponding XML tags in MSCX file). We may also use local identifiers based on some properties of the element which are less probable to change when making changes to the score and reference elements using that element's relative position and id.
Some example of what am I thinking about:
Current state:
Then we will not need to identify this element anywhere else.
Please let me know your opinions on these options.
In reply to The first topic which I… by dmitrio95
I fail to see how
slur_beatxxwould be any clearer than simplyxxfor slur-identification.The downside of storing the "end" together with the "start" is that you have to always update the start-node with that information when the end changes. That is not only when it is moved explicitly, but also when a measure is inserted or deleted in between the start and end-points. Edits that otherwise have nothing to do with the slur in question.
Erasing a measure between start and end for example in the current situation would not change the slur element at all; but in your proposal it does (because of the relative position info). I fail to see how that makes comparing a score easier.
In reply to I fail to see how slur… by jeetee
My understanding is it has nothing to do with the "clarity" of the identifiers, but the "persistance". That is, a slur whose id encodes something about its semantics (eg staff, measure, beat) will keep the same id regardless of other changes to the file. Whereas in our current numeric scheme, adding a slur earlier in the score might very well change the id's of all subsequent slurs, making a meaningful "diff" problematic. It's actually not quite as bad as that - some id's probably do persist - but in general, yeah, it's a problem.
In reply to My understanding is it has… by Marc Sabatella
So actually the only change required is to stop renumbering the slur id's on each save, but have an internal counter (32bit should get you somewhere ;) ) that indicates the x-th created since score origin.
I do agree with below that using ref for references to elements is a real improvement.
In reply to So actually the only change… by jeetee
Well, that could work to some extent but isn't foolproof either. Consider, deleting a slur but then adding one back at exactly the same spot in exactly the same way would now show a diff. And it's not just slurs or even just spanners (lines) that have these numeric id's - in the presence of linked parts, practically all elements have them. So basically, any operation that involved replacing a passage with an otherwise-identical passage would show a diff.
In reply to Well, that could work to… by Marc Sabatella
What Marc said.
Additionally, that would make it impossible to compare independently created scores, which is a task I also need on the occasion, and which, with a bit of upfront design, can be easily solved in one go.
In reply to So actually the only change… by jeetee
I suppose any solution based on artificial preserving of identifiers during load/save cycle will suffer from dependence of result on the order in which changes are made, so even files containing equal scores may be largely different. I believe it is better for resulting file to depend only on the score content but not on history of its creation.
Concerning the mentioned downside of the relative referencing, it really seems to produce unwanted differences in the result after some kind of changes are made near the considered element. However I should admit that this is still an improvement since the relative referencing will suffer only from local changes in the score, but not from changes made to some distant parts of the score. This will make diff output to be more meaningful both for human perception and for automatic analysis. It is clear however that this solution is not perfect and it is better to consider other options too. Perhaps the discussed below solution with global meaningful identifiers can be developed to work better, but this still needs to be thought about.
In reply to I suppose any solution based… by dmitrio95
Just brainfarting here:
First build a list of the now-persistent id's of elements of a given type (let's roll with the spanners example for now, then poke holes into the theory with other elements later).
* Start off by comparing that list for both scores; id's appearing in both scores are considered the same element, therefor they haven't been added or deleted; but could've changed anchor.
* For each deleted element, lookup if there is a matching element in the added list (compare anchor points); if so the element was replaced with an identical element (=> indicate as "possible change")
* For each matched element, compare the anchor points: if they are identical → happy times; if they're not → difficult times :-p
In a first round, simply state they have changed; in an advanced round, try to figure out "why" they've changed. Has a measure been inserted/deleted and is that the real cause of the new anchor point? Then this element itself didn't change (so yes, relative information of an anchor point within its measure seems to have advantages here, although this can be reconstructed and doesn't necessarily require a change in the score format)
In reply to Just brainfarting here:… by jeetee
If you wish for a solution in which one has to use the new score diff tool… that would be acceptable.
I’m striving for a bit further: while creating the new tool, the MSCX format is changed already anyway, so why not change it in a way that makes a simple “git diff” also more legible?
As I said, I regularily work with standard Unix text and XML processing tools on MSCX files. And even shellscripts.
In reply to If you wish for a solution… by mirabilos
My viewpoint indeed started off as changing the format as little as possible and better interpreting it, in the (probably idle) hope that it would work on all existing 2.x scores already and wouldn't have to wait for 3.0…
But if the powers that be are open to changing the file format, then by all means; take all we know and all we want into consideration and make it our best format so far :)
In reply to My viewpoint indeed started… by jeetee
My global identifiers approach does not necessarily¹ require changing the file format.
¹) Replacing references’ id= by ref= does, but this change is independent of the actual scope of this task, but if the file format is going to be changed anyway it’s a good idea nevertheless.
In reply to If you wish for a solution… by mirabilos
I agree it would be wonderful if regular diff tools worked well. But if that ends up being too difficult to accomplish, I'd happily sacrifice that in favor of a working MuseScore-specific tool.
In reply to The first topic which I… by dmitrio95
I massively advice against local identifiers: I regularily process
.mscxfiles with regular XML tools, and for these,idattributes must be unique / distinct.Instead, I suggest you look at using a harmonised naming schema. The resulting IDs will be a tad longer, but that shouldn’t matter. (While there, make all things that “use” a defined item use something other than “id”, e.g. “ref”, for the attribute name.) Something like (here, both definition and use are in the same measure, but they need not be, as IDs are global):
(Of course, using the beat schema would be even more stable. Something to consider though is having multiple Slurs, etc. in the same beat, e.g. on 32ths in the same beat in a 4/4 Takt.)
Furthermore, doing so allows people to easily use an XPath expression like
//[@id="measure3_slur1"]to select the Slur definition, when they know the reference.In reply to I massively advice against… by mirabilos
I agree it's best they are unique. But hopefully an encoding scheme can be devised that ensure that - an id that represents something about staff, track, measure, tick and then appends a numeric identifier as well if there happen to be more than one.
Such a scheme will indeed still show large differences if a section of music is deleted, thus throwing off measure numbers, unless the diff tool is sufficiently clever. Even if not, though, I'd consider that an OK price to pay. I'd rather have larger diffs in the relatively uncommon case where measures are inserted or removed than in the more common case where slurs are added or deleted.
I've given this a lot of thought, and I've decided to recommend @dmitrio95's initial suggestion "Describing all the needed properties of the element just in the place where it first appears", albeit with a few modifications.
To paraphrase @dmitrio95's proposal, there should be no
<endSpanner>element. Instead, the spanner start should specify, in terms of measures and beats, how far it is to the end of the spanner. This removes the need for an ID, thereby enabling diffing and merging any two files, regardless of whether they share a common history, and with any tool, not just one specifically designed for MuseScore files.With one exception, @dmitrio95's method allows measures to be added and removed anywhere in the file, and it would even work with meterless scores (scores with a single measure of unlimited length). The exception is that no measures can be added or removed within a spanner), because as @jeetee mentioned, this would mean that the spanner's length attribute will need to be updated. This won't be a problem when editing within MuseScore, but it could be a problem for shell scripts and
git merge. However, I think that with a few modifications to the method we can overcome even this problem.We still need an
<endSpanner>elementWe don't need to give the spanner an ID because we can still match the start to the end using the stored length (in measures and beats), but it is still useful to have an
<endSpanner>tag as a marker.Consider these points:
<endSpanner>marker.)<endSpanner>markers, any element could be part of a spanner<endSpanner>tag can be used to reconnect broken spanners<endSpanner>will no longer occur in the expected place.<Spanner>and<endSpanner>remaining unconnected, MuseScore knows they must go together.<endSpanner>tags are to their expected positions.The
<endSpanner>needs to refer back to the startSo having an
<endSpanner>tag allows us to recover from most situations where measures are inserted without updating spanner lengths. However, it could be that one of the inserted measures itself contains a spanner, and this spanner happens to have an<endSpanner>tag in the expected place for a spanner outside the new measures. Or perhaps some measures are removed, and now the length lines up with a pre-existing spanner outside the range of removed measures. In these cases there will be at least two spanners with lengths pointing to the same<endSpanner>, and MuseScore won't know which is the right one. Here the problem can be solved by saying "always trust the shorter spanner", but this may not generally be true.The most reliable way to ensure that the spanners are correctly matched is to have the
<endSpanner>give the distance back to the<startSpanner>. If a<startSpanner>and<endSpanner>both point to each other then you know for sure that they belong together. If extra measures are added and the start and end no longer point to each other, you can still pair them up using the fact that the length is the same for each. (So now the length acts a bit like an ID, but only after all other spanners have been connected that could be, and relative positions must still be taken into account).The
<endSpanner>could specify all properties of the spannerWe can make it even easier to pair up spanner endpoints if all properties of the spanner (horizontal and vertical offset, node positions, dash type, colour, etc.) are provided at both ends. Of course, this doesn't help if all spanners use the defaults, but if things like hairpins overlap then there is a good chance that at least one of them will have an offset. Specifying all the properties also makes it easy for human readers to match up the spanner endpoints, and it makes it easier a write a shell scripts to "delete all red spanners" without having to look backwards to see whether a particular spanner was red.
Syntax
I recommend the following (or equivalent) syntax for all spanners:
Note: Using
<Spanner>for everything, and not specific names like<Tie>, makes it easy for a shell script to remove all spanners from the file. Adding the "kind" attribute allows particular categories of spanner to be identified. It is important that "kind" is mentioned on the same line of the file as "Spanner", as this makes it much easier for shell scripts to identify (e.g. usinggrep).If both ends of the spanner are in the same measure (or we have a meterless score) then
measureswill be 0 andfractionswill simply be the distance to the other endpoint. If the endpoints are in different measures then you need to take into account where the current endpoint is situated within its measure, as well as where the other endpoint is situated (i.e.fractions-startdoes not have to equalfractions-end).In reply to I've given this a lot of… by shoogle
I like everything you have suggested about spanners. There are some problems with this. The amount of data each spanner uses has suddenly multiplied. What will this do to the size of a symphonic score? They're already large enough. Even with the new algorithm being worked on the prevent too much recalculating, I'm concerned about what this will do to the speed of MuseScore. Perhaps a default could be defined and anything not changed from the default would not need to be explicitly stated.
Default:
Kind and type would always need to be stated so there are no defaults
Layout would of course be the default layout if the user does not change anything. (I haven't seen version 3, but I would like to see user definable numbers rather than only visual feedback for most of the edit boxes on a hairpin, line or slur).
Measures would be 0
Fractions would be = to 1 beat and only stated if different.
Track would only be stated if different than the current one
Voice would only be stated if different than the start voice
Element is already optional in your discussion.
In reply to I like everything you have… by mike320
You raise a valid point @mike320, but the issue is not as big as it might appear.
The OpenScore Edition of Mozart's Symphony No. 41 is 157 pages long. As a compressed MSCZ file it is 4.6 MB, but 4.3 MB of that is just for the cover image! When compressed, the actual musical content of the symphony is only 300 kB, and this will not significantly change as a result of the new spanner syntax. (In fact, MSCZ files may even get a tiny bit smaller with the new syntax, because spanner elements will have more in common and will therefore be more compressible.)
Of course, if you need to work with the uncompressed MSCX file then that is going to be much larger. In this case the MSCX file by itself is 9.4 MB, and it has 364450 lines of code. The file contains 3696 spanners (2786 slurs, 880 ties, 25 hairpins and 5 other). Let's assume each of these takes up 5 lines of MSCX code for the start tag, and 1 line for the closing
<endSpanner>. That's 22176 lines of code just for spanners, or 6% of the total.Let's assume that, after this GSoC project, each spanner now takes up 10 lines for the start tag, and another 10 lines for the end tag. This means spanners now take up 73920 lines of code, an increase of 51744 lines, bringing the total number of lines in the file to 416194. So spanners now occupy 17% of the file, and the length of the file has increased by 14%. Assuming the size also increases by 14%, that means the MSCX file goes from 9.4 MB to 10.7 MB.
A 14% increase isn't a huge amount in the grand scheme of things, and this can be reduced by using sensible defaults as you suggested (though I think the default value for
fractionsshould be zero, since grace notes slurred to a real note occupy zero beats). Also, we should consider the fact that other changes made as a result of this project, such as not storing IDs or measure numbers, will actually reduce the size of the file.So in most cases, MSCX files will increase in size by no more than 15%, while MSCZ files will remain tiny. When it comes to the time taken to save or load a score, this should be significantly improved in MuseScore 3 due to the new layout engine, and the fact that joined albums remain as separate scores internally.
In reply to You raise a valid point … by shoogle
My only concern about the size is the amount of time it takes currently to redraw the score. It already gets quite sluggish very quickly. I haven't tried the Mozart example, but I suspect I would have entered it in 2 or 3 files, then joined them to a single file for upload due to the length of the score. My concern is that the gains being made by the new layout engine will be eliminated by a larger hairpin definition. There are no doubt other things (I'm thinking auto avoidance) that will slow down the layout and we need to guard against slowing down MuseScore even more for users such as myself who have sluggish response when entering large scores such as symphonies.
In reply to My only concern about the… by mike320
An understandable concern, but actually, it should be a non-issue for two very important reasons.
The first reason is that the new layout engine is not just faster overall, it's that it does something very specific: it does not lay out the entire score on every change, but only the part of the score that actually changed. So it totally should not matter if the score is 20% larger, or even 200% larger - the time to make a change is proportional to the size of the change, not the size of the score. At least in theory; in practice there are some operations that do still require a full relayout.
But then we get to the second reason: what maters is not the size of the score on disk, but the szie of the score in memory. Once MuseScore loads a score, it works with its own internal representation, which is not based on a system of text-based tags but rather on data structures that are represented in a much more efficient format. And the proposed change to the file format would not necessarily require any change in that internal format. So a more verbose file format would not cause the score to take up more space in memory. And in any event, to the extent that a full relayout of the score is proportional to the size of the score, it isn't actually about the amount of memory taken up, but on the sheer number of elements to be laid out. A single slur is still a single slur and takes exactly the same amount of time to lay out whether it takes up 100 bytes of memory or 9000.
So really, we are talking about disk space only, and as noted, the actual difference should be negligible in practice.
In reply to I've given this a lot of… by shoogle
I don’t think all this redundancy is good, especially the
«element»Note A4 quarter«/element»part also makes it hard to change (e.g. transpose the entire score) the end element.In reply to I don’t think all this… by mirabilos
It seems to me that what you mention would in fact be redundant. The only thing that should be identified is if it is a note or rest. If it's a note in a chord, perhaps the notes in a chord could be numbered from the bottom (normal entry order), so a singe quarter note would be simply identified as "Note 1" or "rest" since all of the other specifications are identified elsewhere. If it's a quarter note in voice 3 there is no other note it could be. This is the only note that could be found on that beat, in that measure in that voice. It would not be at all ambiguous and make transposition and changing the duration easier.
In reply to I don’t think all this… by mirabilos
<element>Note A4 quarter</element>was just filler text to give you an idea what would go inside the<element>tag. The actual syntax would be the same as for a note elsewhere in the score (possibly minus a few subtags if they are not needed). It wouldn't be any harder to transpose since the syntax is the same, though I accept that it is one extra thing to take care of.Anyway, it may not be necessary to include the element tag, or at least not all of it. It will be up to @dmitrio95 to determine how much information is needed to uniquely identify the companion endpoint in various editing scenarios.
In reply to <element>Note A4 quarter<… by shoogle
I understood that, my point was that including the actual note (length, pitch, …) would be counter-productive.
In reply to I've given this a lot of… by shoogle
I have started implementing some of the discussed changes so I think I can post an update on this. First of all I would like to describe what have been implemented for now. As my modifications are based on the described by @shoogle solution I will denote mainly the differences and will try to explain why I decided to make those changes.
Note that I still did not implement a reconnection of broken spanners so these points may change as I start to do it.
Not writing the endpoint type (start/end) in the
Spannertag's attributes.First of all, it is redundant as it can be readily deduced from the presence of
prevornexttags inside theSpannertag. It also makes not so much sense in the case of beams or tuplets to which I intend to apply this way of storing too. The only possible reason for keeping this may be making the format a bit more human-readable though I don't think it would improve such a readability.Storing the spanner properties only in the starting endpoint.
In fact, I have no strong objections against adding such an information to every endpoint, I just still had no need for this. Perhaps I will need this when implementing a reconnection of broken spanners but in other cases this is not needed.
A set of parameters used for an endpoint identification.
For now I used three parameters for an endpoint identification:
In fact, I am not sure how it behaves with grace notes, perhaps another parameter will be needed to identify those.
Only differences of the mentioned parameters are stored so this prevents changing the connections representation in MSCX file while doing most of changes not related to that connection. Values that equal to the default ones are not written though defaults are chosen arbitrary now.
To make it a bit more understandable, here is an example of a Slur stored in such a format:
Concerning things I intend to implement next I can mention:
1. Using the new storage mechanism to store also beams and tuplets.
2. Implementation of reconnection of broken connectors;
3. Fixing a bit my code to work with elements parsed in Score::paste(). Not a big task but still I mistakenly thought that only Elements deal with parsing MSC format on beams/spanners level.
In fact, there are some other points I would suggest to change but I guess I'll leave them for another post.
As always, my code can be seen in my MuseScore fork on Github.
In reply to I have implemented a bit of… by dmitrio95
Looking good so far! However, you absolutely must not use ticks since we're trying to get these out of MuseScore's code. You need to use fractions instead as these are more accurate for tuplets. You also need to use measures rather than giving the total distance in terms of fractions, because this will help with diffs when measures are inserted. So if two ends of a spanner are separated by 5 beats in a 4/4 time signature, you need to give the separation as "1 measure + 1/4 fractions", not "5/4 fractions", and certainly not "2400 ticks". The measures and fractions need to be on separate lines of the file to help with the diff.
I recommend starting with only the attributes you are certain that you need (as you are doing) and set up some test cases to see how they handle challenging situations (like multiple spanners starting and ending in similar places), to see if any more information is needed.
The non-essential information could be output in a special "verbose MSCX" mode. I know that Werner added things like measure numbers and (I think) linked part IDs (
<lid>) to the MSCX format specifically to make it easier for humans to read, even though they are not strictly necessary for the file to be parsed correctly. You will have to remove these things to make the file diffable, but it would be good to have a special mode where they are still added (maybe when MuseScore is run in debug mode [mscore -d] or test mode [mscore -t]?). These optional IDs could be harmonised as @mirabilos suggested (i.e.id="measureX_fractionY_elementZ").In reply to Looking good so far! However… by shoogle
Thank you for your comments, I will make the necessary modifications then. Concerning writing the non-essential information, I can suggest using XML comments for this purpose like I have done it for measure numbers (this commit). This way we can write any useful information without affecting the file reading process at all. This may also be useful for writing a MuseScore-specific score diff tool: comments can simply be ignored when analyzing textual differences between two files, and this can simplify the analyzing process without any special workarounds like temporary turning off the debug mode, introduction of any new flags/parameters for score comparison mode etc.
In reply to Thank you for your comments,… by dmitrio95
It's a good idea, but only for the verbose files. It won't work for the standard file because we need to be able to analyse them with tools that don't understand even normal XML, let alone MSCX. The output of
git diffwill be affected by any kind of sequential information, regardless of whether it appears in an XML comment or actual XML code.When it comes to the special verbose files, the information could be included as a comment, but I'm more inclined to include it in the actual code because that makes it easier for people to use it if they need to. (They can always strip the things they don't need from the file.) However, I'd be interested to hear what other people think about this.
In reply to It's a good idea, but only… by shoogle
Of course I propose to apply this only to verbose files (written in debug mode, as for the linked commit). Concerning the actual form of writing the information, including it in the code will not make a big problem, of course, and a convenience for reading is more important. So it would be good to know which option is considered more convenient.
In reply to Of course I propose to apply… by dmitrio95
Ah sorry. Anyway, here are some tricky situations you will need to think about. Extra information will be required to solve these. Perhaps the distance between endpoints needs to be given in measures + fractions + segments, where segments is the number of non-empty segments in the current voice?
BTW, you will need to store "Staff" and "Voice" separately (i.e. not as a single "Track" variable), for similar reasons as why "Measures" and "Fractions" needed to be given separately.
I had a thought I'd like to share, a somewhat different application for what is largely the same tool. I'd like to come up with an automated way to grade a student's performance on a simple tasks, and I think a diff tool could be a part of this. For instance, the student might start out with a simple score. He is then asked to add a crescendo marking starting in measure 4 and extending to beat 3 of measure 5. He adds the crescendo, then I want to have a way of automatically determining if he did it right - in particular, that he didn't add it to measure 4 only and then drag the endpoint. To some extent we can already do this via an ordinary diff (and that is the basis of the automated test suites we use in MuseScore development) but a smarter diff tool could be really useful for this. Just something to think about.
In reply to I had a thought I'd like to… by Marc Sabatella
This is an interesting use-case. You can kind of see it as a "reverse diff". You are comparing the student's score with a known-correct version, but it's not a case of "the crescendo endpoint was moved from measure 5 to measure 4", it's actually "you need to move the crescendo endpoint from measure 4 to measure 5".
For your information, the raw MSCX diff will not be displayed in MuseScore (at least not by default, there might be an option to view it). Instead, there will be a nicely formatted (and translated!) list of changes between files:
Of course, this list will be generated based on the raw diff, potentially with a bit of advanced processing to recognise when things have been moved (rather than simply saying they were deleted from one place and then added again elsewhere) or transposed, etc.
There could be the option to reverse the language ("you need to" rather than "this happened") and expose this to the user somehow, possibly via the plugin framework. There could also be a command-line option to print the formatted diff, so your student could upload their score to your website, and then the diff is generated on the server and displayed on a webpage.
If you have any other requirements then feel free to start a new thread to discus your specific use-case further.
In reply to This is an interesting use… by shoogle
“the raw MSCX diff will not be displayed in MuseScore” seems like way too much effort at first and something that should be added later, after the raw MSCX diff was made more useful.
But in the end, as long as the raw MSCX diff does not get any less useful (since I use regular “git diff” on those files very often, and external tools just won’t cut it) I won’t complain too much. (Background: I regularily work on computers not powerful enough to run MuseScore, for which I have to open a remote desktop connection to another one, which is slaggish.)
In reply to “the raw MSCX diff will not… by mirabilos
Haha same, i wont complain too much as well.
In reply to “the raw MSCX diff will not… by mirabilos
Indeed, there won't be any work done on the UI until the raw MSCX diffs are sorted. The first half of the project is improving raw diffs, the second half is to format them nicely in the UI, and the "third half" (if there is such a thing) is to create a viewer for visual diffs (i.e. highlight added and removed elements in different colours in the score viewer itself), but that will only happen if there is time left at the end of the project after the textual diffs are sorted.
In reply to This is an interesting use… by shoogle
This is something that only just occurred to me last night, so I don't have any more developed thoughts yet. Somehow it also seems connected to the idea of a "lint" tool detect common errors, but here it's more specific of course, comparing to a known expected result but hopefully providing more usable feedback than a plain text diff currently would. Honestly, I hadn't thought about the wording of things, and was envisioning the teacher rather than the student being the one to view the output, but sure, allowing the student to self-assess is even better.
I am about to open a pull request with my changes to MSCX fileformat so they will be available for reviewing. Below is a list of those changes. Any adjustments are, of course, possible so any opinions on the proposed changes are welcome.
So, the list of the proposed changes to MSCX fileformat:
1. Do not store measure numbers
Measure numbers were not used during reading MSCX file but could produce large changes in case some measures are added or removed from score. After the proposed changes numbers will be written as comments only in debug mode:
2. Spanners identifying their endpoints by their positions
In order to avoid IDs when writing spanners a mechanism similar to the one described here was implemented. Both spanner endpoints contain information on where the other endpoint can be found, and this information is then used to connect spanner endpoints. Start point also contains a description of spanner properties. It is also possible to reconnect spanners that appeared broken after editing MSCX file outside MuseScore — this feature was not extensively tested but should probably work in most cases.
3. Beams without IDs
Some features of Beams make it possible to store them in a bit simpler way. If not explicitly written, beams are assigned to chords based on their duration, position and score context. In MSCX files explicitly written beams were assigned to chords using IDs, but given that those IDs were always defined to conform the beam assignment according to the score context and special chord properties, it is necessary only to properly assign a beam to the first chord it should contain. Within the discussed proposal
<Beam>elements are made to be stored right before the first chord they should contain, and beam IDs are completely removed as redundant:4. Tuplets without IDs
Similar to beams, tuplets are also situated within one voice and within one measure. In order to identify tuplets it is proposed to enclose chords that are within the certain tuplet to be enclosed within
<Tuplet>and<endTuplet/>tags denoting tuplet's start and end:5. Voices storage
Previously a lot of elements contained
<track>tag to denote the voice to which they belong. To avoid the repetitive use of absolute track numbers it was decided to store each voice within a<voice>tag. Voice change was previously accompanied with a presence of a<move>tag which denoted a reading position shift. With an introduction of<voice>tag it is needed more rarely. Still when it is needed the same<move>tag is used that was previously shown for spanners storage. In fact, the name of this<move>tag is derived mainly from this its application:6. Beams and slurs not storing track numbers
While the previously described changes eliminated the need for explicitly stored track number for most elements, beams still needed it if they belong to chords with non-zero staff move attribute. As beam's track is anyway defined by chords which it is attached to, it is proposed to force beams not to write track number. A similar change was done to slurs: they were previously forced to write track numbers regardless of the need for it. This enforcement was removed, as well as was removed
<track2>tag previously used by slurs and ties: second endpoint's track will anyway be set up correctly upon endpoints reconnection.7. Use common
<move>tag within Chords and Rests to denote staff moveMostly a cosmetic change:
<move>tag was used by chords and rest do denote their "staff move" property. The<move>tag format was changed to the commonly used within the other context so the format become a bit more uniform.8. Avoid links IDs (
<lid>)Elements belonging to the linked staves, or some other elements in different situations, can be linked to each other. Such a link was previously stored using linked elements' IDs which creates some problems for the file comparison purpose which was already discussed in this forum topic. To avoid them the following solution is proposed (also it is far from perfect, so fell free to propose other options too!). In each link an element from master score or the first element is considered to be "main". In fact, currently the main element should be written first within a score file but perhaps this will be changed in future. The main element is denoted by
<linkedMain/>tag, other elements have<linked>tags which can have various properties denoting the main element's position. In the most simple case if two linked staves have each element linked, those<linked>elements will be empty. Otherwise they will contain some information helping them to find the main element: the previously described<move>tag to denote main element's position and<indexDiff>element to denote a change in local index of the written element among other elements in the same score, staff, voice and position. This was intended to fulfill a requirement to have<linked>tag empty in most cases: if there are two linked staves then most of the linked elements will go in the same order within the file so their local indices will not differ. That way it is possible to avoid difficulties with highly non-linear structure of the score but it makes scores with linked elements hardly editable by hand at least in those (rare) chunks that have such<indexDiff>attributes present. For those cases I have also a thought not to disable<lid>tags: that way one can get a score with<lid>tags (in debug mode, for example), then edit it in any convenient way and convert it using MuseScore to something more or less comparable and usable with version control systems. Of course, it is not a problem if one does not intend to edit a score outside of MuseScore. Anyway, I will be pleased to hear your opinions on that. As always, some example of the proposed option:That seems to be the full list of changes to MSCX fileformat. There were also some other minor changes to code but I'll probably leave them for some other post or pull request description.
In reply to I am about to open a pull… by dmitrio95
I appreciate the work that has gone into this! Sounds like you've consider a lot of cases. Just a couple of observations:
I personally will miss the measure numbers. I use them all the time when examining an MSCX file to understand something about it. Probably this makes me unusual, and using debug mode would be my workaround, but still, I wanted to express the concern.
Be sure you handle cases where slurs cross staves / tracks. I think maybe they are unique among spanners in supporting this?
In reply to I appreciate the work that… by Marc Sabatella
I'd miss measure numbers too
In reply to I appreciate the work that… by Marc Sabatella
I understand that measure numbers (and IDs) are handy for humans reading the file. Obviously we can't include them by default or that would ruin the diff, but they will be available if you run the file through MuseScore in debug mode, and if MuseScore is unavailable for whatever reason then it is a trivial matter to write a Bash script to insert measure numbers.
IDs are a bit harder, but I will create a Python script for this purpose. The script will assign IDs to spanners by reading the
<move>element, allowing you to easily manipulate the file outside MuseScore. Once you've done with the file you run it back through the script to replace the IDs with<move>tags so it can be read by MuseScore. The IDs will be based on the harmonised naming schema proposed by @mirabilos, and the script will also add (and remove) measure numbers.By the way, I suggested to @dmitrio95 that it might be good to output IDs in debug mode as well as measure numbers, and to actually put them in the elements rather than add them as comments. This would make it possible to actually use the measure numbers when manipulating the file programatically, though it could encourage bad practice (i.e. people making a habit of saving files in debug mode). Any thoughts on this?
In reply to I appreciate the work that… by Marc Sabatella
Sorry to answer so late: I was used to e-mail notifications from the forum but this time they somehow didn't come.
Concerning slurs crossing staves and tracks, the new spanners strorage system handles that so it is possible to store such spanners regardless of whether they are only slurs or not.
Concerning measure numbers and IDs, a few notes on that:
1. It might be not too convenient to assign measure numbers (and IDs based on them) by an external script since measure numbers do not always grow monotonically. Of course, it is possible to handle that but still an extra attention should be paid to it while creating such scripts.
2. If many people prefer to see measure numbers and IDs in score files (they are indeed convenient) then perhaps we should detach their writing from the debug mode and add some new flag handling that which can be switched from GUI (in save dialog or perhaps MuseScore preferences screen)? This may be more convenient that constant usage of MuseScore in debug mode.
3. I should admit that IDs will spoil diffs only when comparing files with external tools, the tool built inside MuseScore can anyway convert scores back to the comparable format prior to performing the comparison itself. So having two modes for MSCX format one of each would include various extra IDs and measure numbers written wouldn't hurt that functionality, and perhaps for most users such a mode can even be made default. What do you think on such an option?
In reply to Sorry to answer so late: I… by dmitrio95
<move>element by default. What happens if I adjust one but not the other?Regarding (3a), there is no doubt that IDs are useful for scripts, so perhaps they should be trusted in preference to
<move>? Or maybe they should be ignored unless<move>is missing, so people have to explicitly delete<move>if they want to use IDs.In reply to I would be assigning each… by shoogle
Well, then it is probably better to disable IDs by default. Then we need to ensure that the interested people are aware of an option to turn it on :)
Concerning priority of IDs and
<move>tags, I suggested a similar solution regarding<lid>tags: the new system for storing linked elements proposed in my changes is probably not too friendly for human who would like to read or edit links so I suggested that<lid>tags should take a priority if they are present (it is not implemented in the pull request). The same thing can be done with spanners: if ID is present then the other information on spanner endpoint's position is ignored.<move>tags can even be not outputted when saving MSCX file in such a mode to avoid confusing readers of the file.P.S. Perhaps we should also call these new old IDs by some other name (
reforid/refpair?) like pointed out here.In reply to I am about to open a pull… by dmitrio95
A small update: I have recently added to my pull request a commit which makes TieSegments be written in
<TieSegment>tag instead of<SlurSegment>as it was previously done. This does not directly affect scores comparability but will make it easier to determine a correspondence of XML tags inside MSCX file to MuseScore's internalTieSegmentobjects.In reply to I am about to open a pull… by dmitrio95
The pull request I described in this thread got an update. It involves mostly code changes but has also an impact on the fileformat so I wanted to post an update here. Main changes are:
1.
<move>tag renamed to<location>.2. Staff move property of chords and rests returned to an almost old syntax.
These changes are made after this discussion for achieving more consistency between entities naming in MSCX files and in MuseScore code. After renaming
<move>tag to<location>its name does not overlap with the name of "staff move" property anymore so I returned it back to the old syntax. The only thing I changed is the name of the corresponding tag:<staffMove>is now used in this context instead of simple<move>. Hopefully this name will be more clear for XML tags corresponding to this property.Discussion of diff-related changes to the MSCX format should continue here, but Dmitri has made a new forum post to discuss designs for the comparison tool's UI.