R4778 XML import & line breaks

• Sep 13, 2011 - 18:56

I will focus this question in this trunk forum as this is the future version.

XML import/export has recently received many updates and is getting very clean. However, one this which I didn't notice until now was the auto-linebreaking that was happening. Issue #12437: [trunk] R4754 MusicXML issues: keysig imports in first staff only dealt with keysig export/import and through that I noticed that some component of XML was auto-inserting linebreaks.

Now, I typically do set a line break about every 4 measures in my own scores, and so does the XML import. However, my question is why does it do this? It is a manual process to remove them if someone don't want them.


Comments

The MusicXML importer does not add any linebreaks, this is done by MuseScore's layout engine, which builds systems out of the imported measures by inserting automatic linebreaks. If any linebreaks are present in the MusicXML file (as elements "print new-system=yes"), these are imported as manual linebreaks.

On export, both types of linebreaks are exported to MusicXML.

End result is that when importing a MusicXML file without linebreaks, after export it will contain linebreaks at locations defined by the MuseScore layout engine. A file already containing linebreaks may contain additional linebreaks after a roundtrip through MuseScore, if the original layout does not match the MuseScore generated layout. This typically happens with Recordare sample files. After import most measures need to have the width manually tweaked in order to match the original layout.

This behaviour has not changed in MuseScore since 1.0.

Hope this explains what is happening. If not, please let me know. Furthermore, if you can define an improved behaviour I'd be interested to hear about it. Please note that I am not aware of any fully automatic solution that will work in all cases. Changing the behaviour will require additional user input, either at every import/export or as a userdefined setting.

In reply to by schepers

So you're saying it isn't just doing the regular thing of breaking a system automatically as necessary, but is actually adding explicit breaks - those little green icons?

Btw, manual line breaks can be removed in on easy easy shot using the lone break plugin - just enter 0 for the number of measures and it will clear all existing linebreaks. As of a couple of months ago, the plugin works on either the full score or just a selectEd region.

In reply to by Marc Sabatella

Yes, MS (not the XML import according to Leon) is manually adding the green 'line break' icons. While I can see where it could be useful to some people or for some specific import types (recordare files were mentioned), I don't see the necessity of it. MS will break the staves automatically with or without the line breaks defined.

Basically, I don't want MS doing something automatically which isn't necessary, or defined in the original file, and might require me to manually change afterwards. Since my original file didn't have any breaks in it, there's nothing for MS to determine where to actually put the breaks. It appears that when MS is importing the score, and it reaches the physical end of a page stave, it just adds a break as a matter of course.

In reply to by schepers

MusicXML is not done to connect MuseScore with itself but to connect MuseScore with other scorewriters. MusicXML export/import just try to be as complete as possible knowing that MusicXML and MuseScore model can be different. For example, there is no concept of implicit and explicit line breaks in MusicXML.

When you export to MusicXML, MuseScore will add "print new-system=yes" at every new system end to let other scorewriters know that they have to create a new system at this given point. When you import such a MusicXML file into MuseScore, it will create a new system when it encounters this instruction and so it will add a green line break symbol.

FWIW, Recordare is the creator of MusicXML.

In reply to by Nicolas

This certainly does explain things better, its the export that adds the line breaks. I still wonder why the need to specify in the XML export file when a new system happens? Systems should get generated automatically as page margins are reached. In every score program I've ever used, I've never specified line breaks, or page breaks, and I wouldn't expect the export file format to try to enforce the same formatting (or lack of) when I move it to some other program.

Is there an implicit assumption that the XML should contain as much layout detail as possible so it will re-import elsewhere and still look the same?

In reply to by Nicolas

I assume MusicXML *can* be used to represent music at a higher level, though? That is, if you just wanted to represent the music at a logical level, independent of layout, by simply not including all the tags you otherwise would? If so, I could see value in having an option to export in this fashion. If MusicXML doesn't really work that way - if too much about it is tied to layoit - then I wonder if there is a better high level format out there? I know I have seen various research projects in the area of music representation systems, but I don't if there is a better standard for an actual file format.

In reply to by Marc Sabatella

Being XML, you can code more or less things in MusicXML. There is a hidden option in MuseScore to export MusicXML without measure width, notes coordinates and so on. It's mainly used for testing MusicXML input output independently of the layout ending changes.
But as a normal user, I don't really see why you wouldn't want all the information to be carried out to or from another scorewriter.

In reply to by Nicolas

"But as a normal user, I don't really see why you wouldn't want all the information to be carried out to or from another scorewriter."

Since I _didn't_ set the line breaks in this score I wouldn't expect the interchange file (XML or otherwise) to have them. I guess if they do, it can be considered a benefit. To take this one step further, why doesn't the XML export add page breaks as well, even though I didn't define them? If you do it for one thing...

(As a side-note, I just checked Finale as I am in the process of exporting my scores to MS using XML. Sure enough it does export with line breaks defined in the XML, even if I didn't lock the systems in Finale.)

In reply to by schepers

MuseScore implicitly assumes layout of the music is also important (even in an interchange format such as MusicXML). Thus it tries to read/write as many layout details as possible from/to MusicXML. It is not yet perfect, but making slow progress. Note: this also includes page breaks :-).

I'd like to know if this is purely a philosophical discussion? Or can anyone explain a valid reason to be able not to import/export the layout info from/to MusicXML? This may not be as strange as it might seem, given the difficulty in reliably import music including layout generated by another score editor. One of our competitors offers the option of ignoring all layout info on import (which means using the program supplied defaults instead for page size etc.). It might also be useful to be able to export MusicXML without layout info too, for importing into other tools that cannot handle the layout.

This could probably still be done before 2.0.

In reply to by Leon Vinken

@Leon:

"MuseScore implicitly assumes layout of the music is also important": this is to me an unjustified assumption.

In some cases, layout IS important and needs to be preserved, for instance in a 'final user' scenario: you need (or like to) use that particular score which happens to have been engraved with MuseScore, but do not have MuseScore: via MXML, you can import it into your score notator and get (a reasonable approximation of?) an immediately playable score.

In other cases, layout is in fact irrelevant, if not disturbing, and it should be possible to drop it.

1) For instance, I use MuseScore mostly for editing early music pieces; now, conventions for this kind of works may (and in fact do) change: another user may find a piece interesting, but disagrees on the my choice of time signature or may want to represent coloratio as actual black notes rather than with conventional symbols and so on. If he does not have MuseScore, he may use an MXML export to do more work on the piece.

In this cases, layout is largely irrelevant (worse: conflicting) and this hypothetical user will have to remove it as first step.

More simply, he may prefer a different paper size, or more (or less) white margins or whatever: all situations in which the MXML user is not 'very final user' but want to do more work to the piece.

2) Another scenario is the "documentation" scenario: if a piece is engraved (with MuseScore or other), a generated PDF may provide the immediate usability, while an abstract export format like MXML may provide the 'documentation for the future', to be read by other software (not only engraving software, but maybe data management software, to create repertories...) or for when MuseScore no longer exists (or no longer can reliably read that version of its own file format), and so on.

3) Lastly, even in the 'final-user scenario', transferring, via MXLM or other transfer format, a piece engraved with one music notation program to another might be not so straightforward: laying out algorithms will almost surely be different from one software to the other and details like line breaks or page breaks which were appropriate 'before' might no longer be reasonable 'after' (for instance if the destination sw tends to have a more 'airy' format, hard-broken lines may overflow and result in a single measure alone in the next system).

These are examples of context in which preserving layout information is not only not needed, but sometime disturbing. Whether the layout dropping should happen while exporting or while importing is another matter, possibly also worth discussing.

Thanks,

M.

In reply to by Miwarre

I tend to disagree. In all the cases you describe, it doesn't hurt to export as much info as possible to MusicXML. The most information the better. The actual issue it's what to do on import in MuseScore or other sofware.

1/ There is nothing about layout in your example (time signature, noteheads). As Leon said, some software packages can ignore layout information on import if needed. Another user may find useful the layout information. The more in the file the better. If a user doesn't have MuseScore, he can download it ;) it's free.

2/ For the archive scenario, it's even better. Archiving policy is be to preserved all the info you can and layout info is a big part of sheet music. Archiving it's typically a good example of MusicXML usage and it would be a pity to archive partial information.

3/ That's also why it's important to preserve as much info as possible. MusicXML export in MuseScore is improving at every release thanks to Leon. MusicXML has features to preserve font information, measure width, system width, note lacement, text offsets etc... everything needed to preserve as much as possible the semantic and the layout (and also the playback...) of a sheet music.

Implementing a "reduced" MusicXML export would disseminate less good MusicXML files and would be an error according to me. Implementing an import function ignoring all layout info could be valuable to override easily choices made by the first transcriber.

In reply to by Nicolas

My interest stems from my experiences working with a blind student. We've been using ABC fairly successfully to communicate musical ideas, to the the point where I can email homework and tests to her, and she can complete them on her laptop. Her screeneader reads the ABC to her as text, she types her answer as ABC, and emails it back to me. I feed her email into a program like abcm2ps that converts it to standard notation, and grade her work as I would anyone else's. You almost wouldn't believe such a think were possible, and it only works for us because we were both willing to learn ABC. But it's not ideal even for us, and I'm also trying to extrapolate here to other teachers who might be less inclined.

By exporting a simplified version of MusicXML that eliminated unnecessary details about the visual representation of the music, there would be a much better chance of the resulting MusicXML being able to be automatically converted to ABC, thus eliminating the need for the teacher to know ABC, and also simplifyng the ABC that the student has to read. ABC is at its heart a non-visual form of representation, although it does make allowances for the fact that people will want to use it to prepare scores using programs like abcm2ps. And I do end up taking advantage of those features so the same ABC source can be used both by my blind student (via her screenreader) and the rest of my students (via prints I make from the output of abcm2ps). But it's kind of a drag for the blind student, as the ABC is cluttered with all this formatting stuff that has no meaning for her. If I could instead prepare teaching materials using MuseScore and have it automatically converted to a formatting-less ABC representation, that would serve her needs better, plus it would be a more viable method for teachers not willing to learn ABC directly.

Of course, one could also write the logic to strip out formatting within the MusicXML to ABC converter. But the task of writing such a converter would be simplified if you knew up front you didn't even have to deal with the possibility of formatting info in the source.

So, no, perhaps not something "ordinary users" would care about, but definitely something worth having work well. So at some point I may look into that debug output option.

In reply to by Nicolas

Yes, I'm sure it is possible to create an ABC representation of full MusicXML by simply ignoring anything related to layout. I'm just thinking the parser would be easier to write if it didn't have to even eead layout info. Maybe not, though, since XML is designed to be easily parseable. But FWIW, I am pretty sure ABC 1.6 would be inadequate for anything but very simple examples, as unless I'm mistaken, it doesn't support multiple voices. The multiple voice notation in ABC 2.0 is especially interesting from this perspective because it completely divorces the idea of a "voice" from that of a "staff". A four part SATB piece is written with four voices, and then a single line at the top of the file controls whether a program like abcm2ps renders them on four staves or two. So when I talk about a layout-independent representation, that's what I am really dreaming of.

Hmm, I could probably write a plugin to output ABC directly, if I didn't care about chord symbols, lyrics, or other markings. Suggesting that I might someday get under the hood and look into implementing native ABC export.

BTW, like many blind music students, my student does not read music Braille, but I am familiar with the application you mention. I'm glad it exists. A major limitation of Braille, though, is that it is essentially read only for the blind student. Even if I get music to them in that format, there's no way for them to complete an assignment or take a test and get something back to me that I can grade. So traditionally, blind students depend on someone to sit down with them face to face and do everything orally. It's really a fascinating series of technical problems to solve if you want to avoid that. Many of the pieces of the technology that would be required are already in existence, there's a bunch of work left to do in the way of integration. Like I said, we did pretty well last year using ABC directly, but I don't think that's a viable solution for the mainstream.

Since I am the person who started this whole discussion, I want to re-iterate why I brought this topic up. MS is including in its MusicXML export file something that I didn't explicitly include in my score, namely line and page breaks (and maybe more things). When I export from MS 1.1 using XML and import into 2.0, I now have fixed green-icon line and page breaks which did not exist before.

So far, I have been told that these, and likely other things, are part of the MS score structure information and should be saved or exported in order to maintain the scores look in another program. But I say they are not since I didn't include these line or page breaks in score layout. If I didn't _explicitly_ include these elements, they should not be exported. Nitpicking, maybe, but as is evident from the ongoing discussion there's more to this topic than just my concerns.

To answer Leon's question, this is not a philosophical discussion to me. I just don't seem to be able to explain my concerns, or maybe understand yours. Are we both making assumptions or have missed including some point which will help clear this up?

In reply to by schepers

It does seem people have different notions of what MusicXML might be good for, and what the goal of export should be. I see two very different uses of MusicXML - as an abstract rendition of music independent of written layout, and as an application-independent format for sharing music complete with written layout. Lasconic's point seems to be that these are not incompatible - that including layout info in MusicXML file doesn't interfere with the usefulness of the file as an abstract rendition of the music, since programs interested in the abstract rendition can simply ignore the formatting info. On a philosophical level, I still like the idea of a truly abstract format, but on a practical one, I have to agree I don't see the harm in having layout info I don't care about in the file. And if having that info there makes the file more useful for purposes other than mine, then I can see the wisdom in insisting MuseScore always put it there.

I guess I'd want to better understand just how clean that spearation between content and formatting really is in MusicXML. As a litmus test, if one can't write a Perl script in about one page or less that can take a MusicXML file and strip out all formatting and leave the content fully intact, then I would say there still needs to be a way to get MuseScore to generate a more abstract rendition.

So, schepers - I think when Leon asks if your objection is just philosophical, what that really means is, does the presence of those line breaks actually cause problems - for example, something that produces incorrect results that are visible on paper - or does their presence just bother you? The latter strikes me as a philosophical objection only unless it actually causes problems in practice. As a great (pointy-eared) philosopher once said, "a difference which makes no difference is no difference".

So I'm kind of on the fence at this point personally. I still *like* the idea of an abstract format. But maybe MusicXML doesn't need to be it.

In reply to by Marc Sabatella

"So, schepers - I think when Leon asks if your objection is just philosophical, what that really means is, does the presence of those line breaks actually cause problems - for example, something that produces incorrect results that are visible on paper - or does their presence just bother you?"

If this is your definition of a philosophical objection then I guess thats what I have. It irks me that something unnecessary, and not defined by me, is being included in the export file. Since there is a simple way in MS to eliminate the page/line breaks, then maybe its not such a big issue. But it has certainly been an interesting discussion.

On the bright side, this process has led me to create two sample scores which contain as many musical elements as I can. I use these for testing XML import/export to and from both MS versions, as well as score compatability in v2.0 when reading a complex 1.x score. I've discovered many things which don't either export or import properly and it's going to take time to complete my tests and write-up my findings so the XML support can be improved again.

In reply to by schepers

It seems one could take this to kind of an extreme. You didn't manually add barlines, so should MusicXML export contain them? You didn't manually add staff lines, you didn't manually add ledger lines to notes above and below the staff, etc. I mean, at some level, sure, I get why it feels like line breaks are different and shouldn't be added unnecessarily even though we'd never question the wisdom of adding barlines, staff lines, and ledger lines. But when I really think about it, is it really *that* different? If MusicXML export is meant to be something that can reasonably faithfully reproduce how your score looked when you exported it, I think line breaks are part of that. Otherwise, it might end up flowing differently in another programs. So again, it comes down to, what do we think MusicXML is? Is it trying to represent musical content in an abstract way, or if is trying to reproduce the actual layout you created?

Maybe instead of thinking of MusicXML as being like a plain text format that might be rendered differently by every program that views it, perhaps it is more appropriate to think of it being like PDF, which has the goal of making sure each page looks the same on all systems on which it is viewed, not just that making sure the document contains the same content. the world already had a bunch of formats for that, but the value of a format that also locks in the formatting is pretty clear, or no one would use PDF. It's probably overused, though. With PDF, there is no middle ground, - no easy way for a PDF viewing program to choose to reflow the text to fit different display constraints or whatever. Whereas apparently some score programs do offer a reflow option on import.

Anyhow, I'm more and more coming around to the viewpoint that it should be up to the person/program doing the *importing* when he wants to honor the formatting in the original, and that is just as appropriate to bake in the line breaks in MusicXML as it is in PDF. Although I wonder is MusicXML allows for two different kind of line breaks - "soft" and "hard", or "user generated" versus "software generated". Seems to me that the right solution here, if MusicXML offers it.

In reply to by Marc Sabatella

Now Marc, don't get silly with things here. Everyone would expect all the musical elements, and the elements that the user defines in the score, to be exported as closely as possible. If I force any page layout to exactly what I want, then that's what I would expect to see upon export/import.

To take the example of an interchange format for a word processor, each musical section is like a paragraph which flows unending until the end of the piece or a section break. The way staves break on screen only happens because of physical media limitations (defined score page width). Widen the screen/page/whatever and the music will re-flow _unless_ you want it to flow differently by defining line/page breaks just like you can force in a word processor.

In reply to by schepers

and I'm sure you mean no disrespect. Hyperbolic? Maybe. :-)

While I don't use this format at this time, Marc's description of conceiving the output similarly to a .PDF makes a great deal of sense to me.

Why should I have to go back and, yet again, format my output?
If this is someone's idea of how things should look, why should it be imposed on me?

Regards,

In reply to by xavierjazz

"Silly" is not a disrespectful term, and I didn't intend it as such, but he certainly is taking things beyond the extreme. Should I include the smiley next time?

But your first and second comments are contradictive... reread them to see what I mean. "If this is someone's idea of how things should look, why should it be imposed on me?" This means that what someone does should _not_ be imposed on you, but that's not what I have been saying through this whole discussion and not what it sounded like you wanted to say either.

If someone is formatting their score so precisely with the proper line/page breaks, then I don't mind that it will be imposed on me, thats expected and I can still break the formatting. However, and this is the important point, If they are not "forcing" the formatting, then it should _not_ be imposed on me. That's my argument.

In reply to by schepers

How you want to format what you import is up to you, and you would be free to make any change you wish.
However, my expectation is that the file should basically be reproduced looking like it did when it was exported.

If you want to change the format, there are tools for that, for example the break measure plug-in.

Perhaps we can have this discussion board automatically insert a smiley.

In reply to by schepers

Yes, but did you read past my first paragraph? I realize I tend to be overly verbose, and I apologize for that, but as far as I'm concerned the real question is what I got into *after* the first paragraph - whether MusicXML should in fact be more like a word processing format like ODT or OpenXML (where you expect the formatting to come out different in each application that opens the file) or whether it should be more like PDF (where you expect the formatting to look the same no matter where you open it).

Don't get me wrong - I think there is room in the world for both types of format - an abstract interchange format like ODT or OpenXML that provides no guarantees that the score will look the same anywhere else, and a more "formatted" format like PDF that does try to achieve consistency of presentation. The problem is, in the music notation world we don't *have* both types of format. We only have MusicXML, and it is somehow trying to allow one to do both things at once.

As far as I can tell, the best way to do that is to encode the line breaks in the file but give the user of *importing* program the ability to control whether the file should be imported with formatting intact or whether the importing program should reflow things.

The only real world problem this could cause that I can see is that if you've inserted some manual breaks because they are important for readability but you are willing to let the rest flow however makes sense for the importing program and output size, there may not be a way to do that. As I also said above, this would all be solved fairly elegantly if MusicXML supported the idea of soft versus hard line breaks, but I don't know if it does. Even if it did, though, I sort of doubt it would work that well. and more than it works well in MuseScore to put a line break on some arbitrary measure without also forcing the breaks on all preceding lines. It might happen to work for the moment, but as soon as you change scale, stretch, add notes, alter page size, etc - it's quite likely the reflow that results will mean that measure you inserted a break after is going to end up on a line that is way too "short" and hence get stretch out. So in practice, there might not be much value in the soft/hard line break distinction.

Anyhow, sorry again for the length of this, but it does seem important for this to be thought through. As I'm sure Leon, Werner, and lasconic, and others have done before us - this is all probably old news to them. I was quite conflicted myself, and initially I agreed with you, but I have now completely convinced myself that the current behavior is better - put all the relevant info needed to reproduce the score with formatting intact into the MusicXML and let the *importing* program decide whether to treat the input more like ODT/OpenXML (and hence ignore the formatting) or whether to treat it more like PDF (and hence honor the formatting).

And BTW, since our responses seem to have crossed - I took no offense whatsoever at the use of the term "silly". I was obviously not seriously suggesting anyone would not want barlines by default. That was just a rhetorical device to shift thinking to my real point about content versus presentation.

In reply to by Marc Sabatella

Hmmm, content vs presentation is what this seems to be about. Read the Recordare site. MusicXML is not defined as a way to represent the score page (presentation), but mainly a way to represent the score elements (content). Yes, there is formatting options in XML to force the score to always appear a certain way. People are free to use them and I would expect any program reading the XML to honour the formatting. This means that MusicXML can handle _both_ things, a PDF-like way or the flexible way.

However, I certainly don't read anywhere that the intention behind MusicXML is to force the score presentation onto you, unless the score author intentionally does that. It seems that those who implement MusicXML export are taking the intentions behind XML too far and forcing the layout when its not defined.

MusicXML is firstly meant to be a content-driven format, with formatting if desired. Therefore if I don't define the score formatting, it should export that way, allowing reflowing & reformatting. And I don't see the need to control this ability via some user option in export/import.

I'm just as convinced I'm right about this as you are.

In reply to by schepers

As you say, "this means that MusicXML can handle _both_ things, a PDF-like way or the flexible way."

This is true *only* if the formatting information is actually present. That's why I think it absolutely makes sense for it to be there by default. It's the only way to get the maximum value out of MusicXML.

Now, for special unusual cases where you know with absolute certainty that no one anywhere will ever want formatting info in that file, and where its presence would for some as yet unexplained reason be inconvenient, sure, I'd support an *option* to create a more abstract MusicXML output that wouldn't be useful for some of things regular MusicXML is good for (the PDF-like applications), but might somehow be better for this still as yet undefined special purpose.

But this strikes me not as a bug report, but as a feature request - and frankly, a pretty low priority one at that. Try looking at it from the perspective of a user who just wants to get some work done and has no philosophical stake in the question of whether MusicXML was originally intended by Recordare to be primarily about content or presentation:

With MusicXML export as it is now - most of the necessary formatting to represent the score in a PDF-like way is present in the output - the user can take that MusicXML file and do whatever he wants with it. It contains all the information necessary for use in either an ODT-like or PDF_like manner. He gets the best of both worlds, with no additional programming effort required from anyone on the MuseScore project.

With MusicXML export as you propose it be changed to, programmers spend effort changing the implementation in a way that will have one and only one effect as far as the user is concerned: the functionality of MusicXML will be *reduced*. The file no longer contains the info needed in order for applications to use it in a PDF-like manner.. Meaning some things previously possible become impossible. All disadvantage, and as far as I can see, no advantage whatsoever.

So, how could it possibly be considered a good thing to expend programming effort to reduce functionality?

Again, as an *option* (not the default), if some people who do have strong philosophical feelings on the subject or who have discovered some sort of actual use case where the presence of the formatting info causes problems, I fully support the request to add the feature of creating more abstract MusicXML. And if soft/hard line break style exist in MusicXML, I absolutely support taking advantage of them, and that doesn't have to be an option. But I just can't support someone going to the trouble of changing the behavior of MuseScore to achieve the effect of *reducing* its functionality.

In reply to by Marc Sabatella

MusicXML is not meant to be a PDF-like snapshot of your score, unless you specify the formatting in the score by laying it out a certain way and locking it in. It's that simple. MusicXML is meant to be an interchange format, a la RTF.

As is obvious, I disagree with the present export implementation. The PDF snapshot concept is the export default when it never should be. You think the functionality of the export is reduced, but I say it will be the correct implementation: only export the elements that the user defines.

I know I am arguing against the masses, but I don't mind. It shows I am passionate about this project.

In reply to by schepers

I don't mind passion, and I have absolutely no hard feelings about any of this discussion. It's been quite stimulating, and largely a thinking aloud process for me, as I've managed to change my own mind (from agreeing with you to disagreeing :-) over the course of the thread. Obviously, the necessary input came from lasconic and Leon, but it was the process of writing up these responses that really crystallized things for me.

Hard to see how to proceed further, though, as it this point it is philosophical. You are expressing concern over what MusicXML was "meant to be"; I am expressing concern over what it *actually good for*, which turns out to be *more* than just what it was originally intended to be. If there were another more PDF-like format that could be for that purpose, I'd be back on your side in a New York minute. As it is though, I'm reluctant to give up that very useful functionality just to satisfy a philosophical purity over what was MusicXML "meant" to be.

So again, I agree, it makes philosophical sense that only user-defined elements should be exported. But given that doing it the current way causes no problems whatsoever that anyone has ever identified, and it provides extra functionality that wouldn't be possible in a more purist model, I have to be a pragmatist and not a purist here. If there were any downside - any actual problems in practice at all - to the current model, I'd be back on the fence. But right now, I just can't see this as anything but you asking me and all other users to give up functionality just to satisfy a sense of what MusicXML was "meant" to be.

Here's a very specific practical example of what you're asking me to give up:

I produce a score in MuseScore that just happens to lay out as exactly four measures per system, with five bars in the one system where that's what I wanted, the coda just happens to start with a new system, etc. It looks perfect to me as is, so I don't add any manual line breaks. Or maybe there is one or two systems where MuseScore put too many measures, so I add a line break there, and then it looks perfect, so I don't add any more breaks. In other words, I am not skipping line breaks because I don't care where they happen, I am skipping them because *they are already the way I would have wanted anyhow*. This is not hypothetical - I'm describing quite a few of my own leadsheets. I only *need* to add line breaks on a handful of systems; the rest already lay out the way I'd have chosen to anyhow.

Now I export to MusicXML and import into another program.

If MuseScore does *not* include the automatic line breaks in the MusicXML file, what are the chances it's all going to happen to layout out exactly like I wanted it in the other program, which surely uses different fonts, different spacing algorithms, etc? Pretty low.

Even though i didn't add line breaks anywhere except possibly that one line, I still considered the specific layout of that score, with the line breaks as I was seeing them, to be part of what I hoped to transfer to the other program. In the current implementation, there is information there to allow the importing program to make necessary adjustments to recreate this look if I, *the user of the importing program*, wish it. Or, if I want to have the piece reflow in the importing program, I should be able to have that option - *as a user of the importing program*.

And that's the key - it's the user of the importing program who actually needs the ability to make that choice, because is the one who has reason to care. Why should you, the exporter of the file, feel the need to make that choice for the importer of the file? With the information present, the importing user can choose between seeing what you saw or being at the mercy of his own software's layout engine. With the information absent, the importing user has no choice - he's stuck with whatever his own software's layout engine comes up with.

Anyhow, all I'm asking is you provide a similarly practical example of what *you* are missing out on because of the current implementation. Because you're asking me to give up a pretty useful piece of functionality here, and there has got to be something equally/more valuable I gain in return for me to consider this worthwhile.

In reply to by xavierjazz

I suspect this is a terminology misunderstanding between the green "line break" icon in MS and the automatic line breaks that MS does to break systems as it imports files. There's a big difference between them... one is automatic (and not part of the presentation layout) and one is user-driven. Correct me if I'm wrong, Leon.

From what I've read through this thread, it is the XML export function that is adding all the line and page breaks, whether they are user defined or not. That is what I have an issue with.

In reply to by schepers

A bit of context first. MusicXML has evolved from a format containing only the musical data (but not the layout) into a format containing both musical data and layout. It enables (but does not enforce) describing the layout in terms of page size, measure width, position of notes in measures etcetera. This includes where the system and page breaks are. MusicXML is unlike a word processing format in that the breaks do not distinguish between automatic and user-defined breaks. The format just describes the layout (more or less like pdf, although pdf is lacking any semantic info).

As the MusicXML standard does not require including any layout information (position of breaks included), our competitors each made different choices about what layout information to include in their MusicXML export. Basically any combination of different pieces of exported layout information imaginable can be found. Typically this also varies with program version and/or may be affected by user preferences.

Currently on import MuseScore converts MusicXML breaks into user-defined breaks (the "green thingies"). On export it tries to write as much layout information as possible, trying to preserve the layout. ALL (note emphasis) breaks (both automatic and user-defined) are written as MusicXML breaks. I still think this should be the default behaviour.

Having said that, I am aware of several scenarios where the layout information does not help but hurts. This typically happens when importing into a program where different layout constraints are enforced. Thus I consider it a useful feature to (optionally !) be able to export MusicXML without any layout information. On import it may also be useful to ignore any layout information present and revert to MuseScores defaults. This enables things like easily importing into a different page size and layout.

Note: simply removing the breaks doesn't help a lot, as page size and measure width still pretty much define where the breaks are.

In reply to by Leon Vinken

Leon's background was quite useful to me to understand more the question.

1) 'Philosophically'

First, I have to say that, on principle, I agree with schepers almost 100%.

We DO have a PDF-like export format, i.e. PDF itself! If an immediately usable (playable, readable) score is the main goal, export as PDF and ANYTHING will be in the score exactly as you put it in.

As Leon said, PDF lacks semantic info, so the specifics of MXML is to include this info too and the advantage of using it is for destination users needing this info. If they need this info, chances are they are going to change the musical contents (if they only need to read the score, using the PDF output is by far more convenient), which is likely to affect the layout and to make the original layout info useless if not encumbering.

I feel Marc example misleading (for once! I usually find his posts very on the mark and well thought), as it can be seen by porting it to the word processing context: you have written a textual document whose chapters by chance happen to end each at a page break; you do not NEED to add page breaks at the end of them. If you export the doc to an exchange format (say, RTF), do you expect the exporter to 'create' page breaks for you? (of course at EACH page break, not only those at chapter ends)

Marc, lasconic and possibly Leon seem to think that ANY data is relevant by itself, while data might be relevant or simply noise, depending on the context. Data like 'soft' line and page breaks are casual (they simply happened there by chance) and exporting them is likely more to reduce rather than increase the signal/noise ratio.

So, generally (or 'philosophically') speaking, I do not see a compelling reason to include ANY layout info in MXML. As layout data might be there in the more recent versions of MXML, "I consider it a useful feature to (optionally !) be able to export MusicXML WITH layout information.", to paraphrase Leon.

2) Practically

I have little hope the above will attract any agreement...

But as I am a MuseScore user and as I find most of the layout info in imported MXML files to be just noise, I think an even more useful feature could be an option to STRIP ANY layout info from MXML while importing it into MuseScore to keep only the musical data.

Or even an option the KEEP it, assuming the stripping will be the default.

Thanks,

M.

In reply to by Miwarre

"Philosophically", I do very much like the idea of a purely semantic MusicXML. I even argued for that earlier.. It's just that "practically", I see a very real need for a sort of "hybrid" format that contains both semantics *and* layout. A layout-only format is nice too, of course, but as you say, we have that: PDF.

I think the need for an export format that contain both semantics and layout is being glossed over here. And in particular, I disagree with your assertion that "if they need this info, chances are they are going to change the musical contents". I can think of plenty of applications for which this is not true, the most obvious of which is implementing a score viewer with playback facilities. You need the semantics for playback, but you want the layout info for display. Of, for that matter, just the simple case of taking a finished score created in one program and importing it into another, just to have it in that format. I most certainly would want to keep all the current layout as well as the semantics. And even when one does plan to change the musical content, many of the use cases I can think of would be such that we'd want to start from a known good layout. Eg, fixing a wrong note, substituting different chord symbols, adding fingerings or other annotations, transposing, etc. That is, you're taking a score that was already in presentable form, but customizing it in some way that doesn't necessarily require one to have to revisit layout decisions. Far from being encumbering, having the original layout info present saves a good deal of unnecessary re-work for these applications.

So like it or not, we need to ability to include layout info in a MusicXML file. And yes, that may mean that sometimes, data necessary in one context will be noise in another. That's why I'm all in favor of an option to include that data or not. Of course, I still wonder what's the harm of including it even though you might not need it, and I'm also concerned it's really the person importing who is going to be in a better position to know if the info is needed or not. Shoot your own foot if you know you won't need it, but why shoot that other person's foot too? Still, I'm as much a purist as anyone (really!), and I can see the value of having a file that contains semantic info only.

So I'm fine with the ability to produce MusicXML that way, as long there is *also* the ability to produce something more like the current hybrid. As for the question of which is the default, I guess if the option is presented as a checkbox in the export dialog itself, so it's always visible and easily changeable, then I don't care. Make the checkbox 'sticky" across invocations and I'm even happier. Suddenly this seems like a total non-issue to me.

BTW, you're right, of course, about my example not really holding water when applied to word processing . Somehow, though, I just see this being an issue more often *in practice* with music. Something to do with user expectations and the sort of situations in which one would tend to use such a facility. As I said, I'm really thinking aloud here, so not everything I write is fully baked. But the concerned I mentioned is quite real in my own use of MuscXML import (mostly from Finale to MuseScore). If layout info from my Finale original were not present, I'd be having to repeat work unnecessarily.

FWIW, one new feature in MuseScore would help with these issues for me: give me a command to "lock" the current layout explicitly. In other words, to add an explicit line break to all systems in one shot. I do this manually now, just adding line breaks to all systems once I'm happy with my layout. Not specifically for MusicXML export, but just to increase the odds that a future version of MuseScore will open my file the same way. Finale had such a command, and I used it religiously whenever I finished a score.

Of course, these line breaks only guarantees MuseScore won't try to put *more* measures on a system than I wanted, but it might still fit less. So what I'd really like is a feature that tells MuseScore, "reduce stretch as necessary to fit these measures on a system", and I think that would be an improvement for overall usability as well. But that's presumably more complex to design and implement than just adding line breaks. There might even be enough info in the plugin architecture to figure out where the lines currently break (looking at X/Y positions of notes)?

If I had a "lock" feature to turn my implicit breaks into explicit ones, I'd even be OK with a one-size-fits-all export facility that only included explicit breaks, because at least I'd have an easy way to get the layout I see into the file. Returning to the word processing analogy, this is sort of like defining a "chapter title" style that inserts a page break for you, in that it saves you from having manually insert the breaks everywhere you want them.

But really, I might still think some additional level of control over what goes into the exported MusicXML would be nice. Manual line breaks, OK, but what about dynamics that were dragged slightly to the left? I'd almost never want that info preserved, because the default position of those dynamics is unlikely to be the same in the importing application as it was in MuseScore. But that's another can of worms, I suppose.

In reply to by Marc Sabatella

I wrote several MusicXML writers (abc2MusicXML, nwc2musicxml, musicxml export in Impro-visor,...) and I wrote or currently maintain a couple of xml readers (unpublished musicxml2lilypond, ...). In a past project, I saw MusicXML files from a lot of sources. So my point of view is not philosophical but 100% practical, pragmatic and from the battlefield. I fail to see how philosophy is involved with an already defined format.

While writing all these converters, I *never* found any musicxml information to be noise. Of course, if one tries to read MusicXML, a file with a lot of info might look more noisy, but MusicXML (or any XML format) is not meant to be read or written by a human. If I didn't like the provided info, for example if I believed my layout routine was better than the one of the source, I just ignored these pieces of info and that's pretty easy to do in a XML parser, you just don't read them.
The goal of MusicXML import/export is score interchange. If you want to send information to one program to another, you must encode as much info as possible at the source side. Often, you don't know what will be the intended use of the MusicXML file.

Checking the MusicXML export facility of Finale, Sibelius, or the Dolet plugin by Recordare, inventor of the format, it's clear that this principle is applied. They did all they can, and it's sometimes complex, to put as much as possible information in the MusicXML file.

So let me finish with an answer to illustrate my point.
"what about dynamics that were dragged slightly to the left? I'd almost never want that info preserved, because the default position of those dynamics is unlikely to be the same in the importing application as it was in MuseScore"
Does it need to be your choice as the exporter? You'll probably not be the one who will import the file. Even if you are, there is little chance that your importer will consider this info if you don't want to be preserved in your use case. But someone else might need it and it's a couple of bytes, zipped, so why removing it or even make an option to don't export it? For a philosophical reason? :)

Of course, an option to ignore layout info on import makes sense !

In reply to by Nicolas

I have come to the conclusion that, and agree with, the adding of an option to ignore layout info on import would be about the only approach. Originally I was thinking of both export and import options, but the import option would handle things nicely.

I do see one problem with this, and it was mentioned by Leon. If we ignore all the layout data, what do we do about the default page size? Do we just assume the MS default (A4, Letter)? I suspect this is one piece which cannot be ignored, especially for large grand staff pieces, or the user has to select an appropriate page size before import. Are there other layout elements that need to be dealt with?

I'm not as convinced that the presentation XML data is the panacea that people here are making it out to be. MXML as an interchange does a good job handling the musical elements, but you are at the mercy of the loading program to support all the elements the MXML contains. I've yet to see any program open an XML and have it look even reasonably close to what I laid it out the original to be, much less the actual measures per line, systems per page, etc. Yes, that includes manual reposition of elements as positioning is part of the musical data.

In reply to by schepers

It is unclear to me why the single page size parameter is more important than other layout parameters during XML importing into MS.

The page size can be changed at any time; if, after a no-layout import, the user realizes a different size fits better the imported score, he can change it at will. As this is supposed to be a no-layout import, there are no other layout details (like 'hard' line breaks) interfering with such a change.

Or am I missing something?

M.

In reply to by Miwarre

I don't know if it's important thus why I brought it up. My main train of thought was a large symphonic grand staff as it would require a sheet much larger than a letter. Assuming you can successfully export such a beast into MXML, would there not be issues importing it into a letter page, or would MS happily do that but only a % would be visible until you adjust the page size?

In reply to by Nicolas

I agree that neither my example of dynamics being dragged nor page size present any particular reason for special cases in export. Put it in, let the importer decide whether or not to use it. Of course, that does mean we would really want control over the import in MuseScore. From a practical standpoint, I see three levels I could want: honor all layout, ignore all layout, and a "smart" mode that kept thing like line breaks but ignored things like manual positioning offsets. The latter mode would please me, but I could also see it being left out as too quirky.

I do still have an attachment to the idea of a completely abstract representation, but I no longer care if MuseScore produces it natively, given that it would apparently be easy to write a program to remove the layout info. Ad actually, such a program could implement the "smart" mode I mentioned above. But I do still think MuseScore should present the first two import options.

Do you still have an unanswered question? Please log in first to post your question.