Spatium rounding problems due to file format and UI

The core issue here is the file format, unfortunately. MuseScore stores the spatium in millimeters. Yet it stores page sizes and positions in inches. And the staff distance field in the XML is in staff space units. Storing different values in different units inevitably leads to rounding errors when converting units. This must be changed in order to solve the problem.

Internally, at run time, the spatium is in points at 360DPI. Some other values are in staff space units, right? I think points at 360DPI is the best choice for units storage in the xml file. That way even the spatium is in the same units as everything else. Otherwise staff spaces might be a plausible choice. Hypothetically it could be millimeters or inches, so long as it's one choice, not many.
IMO the fewer conversions the better, and since run-time is in points @360DPI, that's my recommendation.

So long as this makes it into the beta, it will be fine because of the need for conversion of v2 scores regardless. If it slips past then it will remain a thorn in the side of MuseScore for years longer. Please consider this critical and let me do this work! I've been trying to fix this for over 2 years now...

I am refreshing my memory on this myself, so I too am getting back up to speed on all the previous specifics I tried to tackle previously. I appreciate your input, as this topic needs more points of view. The file format is a problem because it forces the conversion of units. Page width, height and margins are the only things stored in inches in all of MuseScore. That is not sustainable, IMO. At least that much needs to change. Here again is the issue where there are examples of the the margins being corrupted by these problems:
https://musescore.org/en/node/108196#comment-487676

I don't really care what unit becomes the primary unit here. It can be millimeters if that's the consensus choice. I think it's odd that a paper-based unit would be the standard for a software application that does its own layout, but whatever is best. I understand the positive aspects of immutability too.

I believe that units should be handled in one way throughout MuseScore, to go to the core of this and set it straight. You are right there are too many issues that revolve around this same set of behaviors. The UI is a big part of it and I have been active posting ideas to the companion issue to this one here:
https://musescore.org/en/node/278887
The most recent comment goes into some detail about building on QPageSize and QPageSize::Unit, which the code now uses already. If you have any suggestion for how to consolidate this discussion I would appreciate that too. I have been away from MuseScore for a while now, only briefly back 6 months ago for a short visit.

My logic behind using the points @360dpi as the core unit was based on only having one unit. I agree that staff space is a logical choice too, but then the spatium itself is in a different unit from everything else. Then again, maybe that's the whole point. IMO, the spatium should be stored in points at 360dpi, as that's what it is internally, the default being 25pt at 360dpi. I suppose I like a whole integer as the default for the spatium.

Most of my problems with this come from the fact that for my SVG export I don't want to scale anything. SVG is designed to scale things, "scalable vector graphics". Once you diverge from the default 25pt, everything gets scaled. On the selfish front, I could be largely satisfied by a "reset spatium" button. I proposed that at one point as a possibility, but it wasn't received well, and with some good reason. I also believe that a very small percentage of users change the spatium value, and that it's important to maintain a clean default value when the user changes other values in the page setup. That is not happening now, and isn't happening in v2 either.

Here is a previous discussion about the spatium rounding issue:
https://musescore.org/en/node/272331
On the most superficial level, it's caused by the fact that the spatium value is displayed in the units selected by the user, and is rounded by the display widget. So when the user switches to inches, the spatium gets converted, using rounded values.
I'll be looking at newer code with fresh eyes, but my recollection is that a superficial solution was not plausible.

@mirabilos, can you be more specific as to which styles have this problem? I am looking inside a .mcsx file right now and I the ones I am checking are already stored in staff space units, using the latest master. I'm not doubting your issue, but there are over 100 of these style values, so it's not easy to review them all as compared to the values in the dialog.

It looks like the 144dpi units I mentioned before are no longer in use. They were for the page sizes and margins, which are now stored in inches. Am I correct about that? That is a change to the file format already. I thought MusicXML had predefined units for certain things, like page sizes, for compatibility across MusicXML reader/renderer applications. I got that impression from the old 144dpi code, which had a function called getTenthsFromInches(), and maybe a set version too. Tenths being this 144dpi unit. So it looks like one conversion has already been removed, and that these units were in inches already, only the conversion to "tenths" has changed.

There is also one unit I missed, the most mutable of all units: percent. Percentage units are used regularly in the Styles dialog. I'm not saying there is any issue with percent units in MuseScore, I'm just adding them to the list of units that are stored in the XML. Percent is a popular unit in CSS too :-)

I believe I can sum up the spatium rounding problem now:
The spatium default is 25pt @360dpi. That is the actual default in the default units.
Both in the dialog and the file, the spatium is not in points, it is in millimeters or inches.
The problem is that when it is converted to millimeters or inches, it is also rounded to 3 decimal places. It is stored as a rounded number, it is displayed as a rounded number, and the user input is limited to 3 decimal places in inches or millimeters. Note the large difference in precision between 3 decimal places in inches versus millimeters, due to millimeters being 25.4 times smaller than inches. It's all very awkward for managing a core value like the spatium, that multiplies everything else. That was part of my reasoning for pixels = points @360dpi as a page settings unit, so I could ensure the value stayed at 25, a whole integer. There was a default that made sense.

Currently the new score default is 25pt, but once you save that file the default becomes 1.764mm, which does not convert cleanly back to 25pt because it is rounded. You could say the problem is that the file's default is different from the program's default. Then if the user starts editing the spatium value or the units, further discrepancies ensue, at which point restoring the default of 25pt is not possible via the UI.

Something needs to change. Not rounding is an odd solution because it relies on floating point numbers, which cannot be used in comparisons unless rounded in some way. Which might be part of the reason for the fuzzy comparisons that are used to compare page sizes in different units as MuseScore does now via QPageSize::FuzzyOrientationMatch. I am still not seeing a superficial solution that doesn't involve changing the units of the stored spatium value, or some other non-superficial changes.

I think the main bummer is the difference in precision between 3 decimals in inches and 3 decimals in millimeters. When you switch the inches, the rounding becomes a real problem as a result.
But the issue is the rounding, regardless of the degree of problem in different circumstances. And because storing rounded numbers is problematic, it would be best to use units that can generate integer values, like points. Points, in the QPageSize::Unit enum, are fixed at 72dpi, just like MuseScore's DPI constant. That is also immutable, and can be combined with the 5x multiplier that is in place now for simple, integer conversions. You could store all the page size and margins values in points @72dpi, including the spatium. QPageSize can handle the conversions to/from inches and millimeters. MuseScore will do fewer, more precise conversions as a result.
I'm open to any real solution. This is simply the solution I see in the moment. I am certainly not all-seeing.

Also note that in my email and in previous comments I referred to the file format as being the core issue. I should have been more careful in my choice of words, as my issues are limited to page size units and the spatium. I am not intending to include all the styles that are stored in staff spaces as being problematic. That is a related set of units, but I am having no issue with them, and it looks like @mirabilos's issue is fixed in 3.0, though it would be good to verify specific style values to confirm that fix.

I have a usage question for @mirabilos and other users who change the spatium:
Do you think of the spatium in millimeters/inches, or do you think of it as a scaling factor, relative to the original size?
I ask because from an internal point-of-view, the spatium is a scale specifier. There is a default value which is effectively zero scaling, and any variation from that default result in a scaling factor being applied. Scaling in SVG, CSS, and elsewhere is a factor where zero is the default value. It's a standard paradigm for scaling.
But from a user perspective, it's the staff space, and you might naturally think of it in printed page units like everything else, not as a scaling factor applied to a default scale. If users do think of it in page layout terms, printed page units, not scaling factors, then the current interface must continue to exist. But if users treat spatium as a scaling factor, then maybe it could lose the units entirely, like the Spacing (1=tight) style value in the Style dialog's Measure tab. That is an alternate approach to resolving this issue. The file format would still change for the spatium field, but there would be no issue with rounded floating point values.

That is what I thought, but I wanted to confirm. It was a nice idea, but I didn't expect it to sail. I don't deal much with the printed page, but I know most users do.
Just out of curiosity, again - how would you feel about adding points as a unit to MuseScore? Or you might prefer didot or cicero if you're in Europe. Points are 1/72 of an inch, which is about 3x more precise than millimeters. In the US, points are the standard commercial typesetting unit, in addition to being the units for font sizes everywhere. They are also the base unit inside MuseScore (now with a 5x multiplier to provide 360dpi resolution). QPageSize supports all of those units I mentioned, and I'm curious if MuseScore users would appreciate having them available. They are not usually available in Office style desktop publishing programs, but all the pro design and publishing applications have them available. If you are doing fine typesetting in the US, you don't use inches, you use points, maybe picas. I don't know about Europe and the rest of the metric world. I just checked and Finale software supports inches, millimeters, points, picas, and centimeters, but not didot or cicero.

Points help solve this rounding issue because you can use whole integers specify the default value. 5 points is the default spatium. And if you store the spatium in points, then the default value can be converted to any other unit without rounding issues. I actually work in pixels, because I use MuseScore to generate SVG, so for me I want pixels at 360dpi - zero conversion from the internal scale/units. But simply offering points as a unit provides me with solutions to the truly pressing problems, and gives me a cleaner conversion path when specifying page sizes as a user. 5x is also a whole number conversion factor, so it's a lot cleaner than using millimeters or inches. As soon as you save a MuseScore file, the spatium is rounded to 5 decimal places in mm. Once you open the page settings dialog, that is reduced to 3 decimal places. The score's spatium is distorted from the default value at each of those stages, then additionally if you use inches.

At least you have no opposition... The opposition might be to the file format change, storing the spatium in points versus millimeters. But the more I think about it, the more I think it is the best way forward. Points are defined as an immutable unit, relative to inches. In that sense they are equivalent in nature to millimeters. They are also aligned perfectly with the internal spatium value at 5pt. It certainly solves this rounding problem completely and with finality.

Jojo - the only word I know in German is "Ausfahrt", though I appreciate the intention of the Wikipedia link :-)

5pt is the internal value's default setting, yes. The spatium value is treated in various places as a scaling factor, relative to the default value. That value is now multiplied by 5 for greater resolution internally, so if you call score->spatium() for a newly created score it returns 25.
Yes, pt is at 72dpi, just the way QPageSize::Point defines it, the same way the DPI constant in MuseScore defines it.
Floating point numbers are not as precise as they might seem. You must do "fuzzy" comparisons, round both numbers, or deal with the epsilon value:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Globa…
Comparisons against the default are important when deciding whether or not to scale things.
It's generally best to store things in a single, constant unit and always convert from that unit to other units, IMO. It's more consistent and reliable that way. Also, the fewer conversions the better. I'm glad to see the 144dpi "tenths" units go away, but I'd like to go farther.
But there might also be compatibility considerations with MusicXML. I'm not an expert on that side of things.

Also, you should look at the companion issue I've submitted about the UI of the page settings dialog, as it relates to units:
https://musescore.org/en/node/278887
There is an open question about whether or not to store the units as a global preference, a document property for each score, or a combination of the two. The current UI is unlike most other desktop publishing apps, and there are two very standard options to implement. The UI is fully entangled with this rounding issue.

Sorry, I don't understand your 288dpi math. MuseScore 3.0 draws things at 360dpi = 5 x 72dpi. Those are the units it passes to many Qt functions that draw things. I believe those are the layout units.
Yes, I am trying to find a way to provide the user control you describe. The issue is how to implement the path from the UI to the XML and back again. I have a particular issue with rounding that is solved by allowing for units, like points, that tie to the default spatium value in a whole number way. There are other rounding issues too, as you mention.
Regarding the spatium and all the page sizes and margins: storing them in points would reduce the number of conversions and provide the same, if not greater degree of precision as today.
I'm not understanding your last comment, or you're not understanding mine.

...and preset page sizes like A4 should be improved in 3.0 now that it uses QPageSize and all its definitions. But you will need to confirm any specific improvements for yourself. I use custom page sizes for all my "online" scores, and they must be considered too.
From my point of view, MuseScore's primary output is MusicXML. I don't think anything I'm describing screws up PDF printing, which is code I've looked at in file.cpp, or any other kind of printing or measurements on the screen.

Qt handles all the scaling that you describe. You must consider the need to draw on the screen too, not just printing. First and foremost, MuseScore draws on the screen at whatever resolution a user has available.
Regarding preset page sizes, here is the list that QPageSize supports:
http://doc.qt.io/qt-5/qpagesize.html#PageSizeId-enum

To be clear: the list of preset page sizes in MuseScore 3.0 is built from the enumeration in that link. That is the list of preset page sizes for MuseScore 3.0.

It's also useful to understand that MuseScore internally uses a vector style of graphics, not a pixel-based one. It draws some lines directly, and it uses fonts to draw the rest. Fonts are built on vectors, or whatever name you choose for them, not pixels. So any conversion to pixel-based graphics for screen or printed page involves rasterization. AFAIK, Qt handles all the rasterization for MuseScore. It's probably one of the main reasons for using Qt. The obvious advantage to vectors vs pixels is the ability to scale with less distortion.

When I speak of points @360dpi, that is the unit used by the vector graphics in coordination with Qt. Qt converts that to a MuseScore (Emmentaler) font at 100pt for SVG, an extremely round number. Vectors need scale, units, and a canvas size too. note that the 100pt number is based on the default spatium value

...which brings up an important detail about user control of the spatium:
When a user changes the spatium value in the page settings dialog, MuseScore scales everything in the "body" of the score, including staff/system text and measure numbers. It does not scale the title or footer elements.

Changing the spatium effectively changes font sizes too, for text as well as music. You'll still see your style settings at 12pt for staff text, but the result on the screen and printed page will be scaled by the spatium. For your scores with non-default spatium values, your printed font sizes are not equivalent to your style settings; and to determine the actual printed font size you must do the multiplication relative to the default spatium value. That's how it works in both v2 and v3 MuseScore. The spatium is not just the distance between staff lines, it is a scale factor for everything, and that scale factor is relative to the default value of 5pt @ 72dpi.

Here are some specific font size examples for @mirabilos's 3 non-default spatium values, 1.675mm, 1.75mm, and 1.8mm:

1.675 - 9pt = 8.55pt; 12pt = 11.40pt; 18pt = 17.09pt;
1.750 - 9pt = 8.93pt; 12pt = 11.91pt; 18pt = 17.86pt;
1.800 - 9pt = 9.18pt; 12pt = 12.25pt; 18pt = 18.37pt;

It would be nicer in a table, but I don't know how to do that in this comment markup. Still you can see that at your smallest spatium value, 18pt text is actually rendered and rasterized much closer to 17pt.

The QPageSize class handles page width and height for preset and custom page sizes, but does not include margins.
The QPageLayout class is related to QPageSize (by containment, not inheritance), and includes margins:
http://doc.qt.io/qt-5/qpagelayout.html

Both QPageSize and QPageLayout have methods for converting to points, aka PostScript Points, aka QPageSize::Point, at 72dpi:
http://doc.qt.io/qt-5/qpagesize.html#sizePoints
http://doc.qt.io/qt-5/qpagelayout.html#marginsPoints
MuseScore has two sets of margins, for even and odd pages. That will require two instances of QPageLayout.
All other units in the QPageSize::Unit enum are handled by these: note that points is one of those units, so the points-specific methods are a bonus
http://doc.qt.io/qt-5/qpagesize.html#size
http://doc.qt.io/qt-5/qpagelayout.html#margins-1
also note that QPageSize::Unit and QPageLayout::Unit are identical

MuseScore has 2 additional properties beyond the spatium value: Two-sided and First page number. There does not seem to be anything in Qt that manages those two properties, but they are not floating point values that require unit conversions. By using QPageLayout, the unit conversions for margins can be offloaded to Qt, as the page width and height have already been offloaded to QPageSize.
The Orientation property, portrait vs landscape, is not a real property; it is for display purposes and a way for the user to toggle-flip the width and height.

By using these two Qt classes, storing all the page settings dialog's numeric values can easily be done in points. It eliminates custom conversions MuseScore is doing now, and normalizes all these properties: page width, height, the 8 margins, and the spatium. The more I dig into this, the better QPageLayout looks. It handles additional calculations that MuseScore is doing manually at this time, such as pagePrintableWidth and the printer's page orientation, which really is a property. I am going to start cataloguing the affected code and lay this out somehow.

That's nice to know. I wasn't saying any of it was necessarily a problem, I was simply trying to clarify how things work. I did think it might be a problem to encounter an unexpected text size in the final output. If you're really expecting 18pt text, and you're getting 17.1pt text. But I don't know if that's a problem for anyone either.

The important facts are:
a) the spatium value is used in more than one place as a scaling factor,
b) that scaling factor is relative to the default value, and
c) that default value, in its natural form, is a constant in the code set to 5pt. That's how it starts, the file storage in millimeters is later.

Storing a whole number spatium value as the default eliminates the vast majority of scaling and rounding issues, and it aligns with the core MuseScore spatium units by a factor of 5, another whole number defined as a global constant. See here and two lines below:
https://github.com/musescore/MuseScore/blob/1b1912d6ede1ae7bd1e989b6623…
The SPATIUM20 definition could be simpler:
static constexpr qreal SPATIUM20 = 5.0 * DPI_F;
Some excess math in the current definition, but the results are the same.

I have created a document to consolidate all the facts and design decisions for this set of issues. It's too difficult to keep track of it by continuing to post summaries here. I have attached the document as a PDF and in Apache Open Office Writer format, .odt extension. I can convert it to MS Word if anyone requires that, but I believe Word opens .odt files properly.

The document is not complete, but it is a good start. I'm starting to code some of these things to see how they play out.
MS_PageSettings.pdf
MS_PageSettings.odt

PR is here: https://github.com/musescore/MuseScore/pull/4456
It has PDF docs attached.
This is now part of this EPIC issue: https://musescore.org/en/node/280363