Version Scores with Git

• Sep 10, 2013 - 15:43
Type
Functional
Severity
S5 - Suggestion
Status
open issues
Project

Hello everyone,

This post has two purposes, both related to the fact that I've been experiencing some difficulty with large scores in that MuseScore has a tendency to crash unexpectedly. This is usually not a problem because I can restore the session, but every once in a while it crashes during a save and my file gets completely corrupted! The size of the file is reduced to 0, and I have to start from scratch**. I very quickly learned to manually copy a backup file, but this was cumbersome, and I started making a new backup file each time, leading to bloat on my file system.

So, being a programmer, I naturally thought, "This needs Git." And so I started using Git.

The purpose of this post, then, is two-fold:

  1. To document my solution to this problem using Git so that if anyone else is having these issues and would benefit from it, then hopefully this will help them out. This may or may not be a problem unique to my system, but I figure there's at least some probability that someone out there is having similar problems.
  2. To make a feature request for native integration of version control into the MuseScore application itself.

For now I'm just versioning my scores locally on my computer. The nice thing about git is that it does not require a remote server. You could use github or bitbucket if you wish, but I have no need for that at the moment.

I'm running Windows 7 64 bit
I have downloaded and installed Tortoise Git (http://code.google.com/p/tortoisegit/)
I'm working with MuseScore 1.3, though this solution is version agnostic as far as I can tell.

I'm assuming you have some basic understanding of what version control is and how it works. If not, start here:
Getting Started: About Version Control

Essentially, for our purposes, we are using it to provide backups of our scores as protection against corruption caused by crashes during save. It could also be used to allow a team to collaborate on a project using MuseScore, though that may or may not introduce some challenges. And short of collaboration, even, version control has the added benefit of allowing you to roll back changes if you completely rewrite something and then decide you don't like it, you can always revert back to a previous commit that has the older version of the part.

In order to use version control, I do the following:

  1. Create a new folder where I will save my scores
  2. In Windows Explorer, right click on the folder and click "Git create repository here..."
    This option is added by Tortoise
  3. Open up MuseScore and start working on my score
  4. Save my score in the git folder as an Uncompressed MuseScore score.
    This is the important bit. Your files should have the .mscx extension, not the .mscz extension. This is necessary so git can do a diff of your files. It can't diff the compressed scores.
  5. Find the new file in Windows Explorer and Right-Click on it.
  6. Hover over the "TortoiseGit >" option and click "Add"
  7. Click OK
  8. Click "Commit"
    * Note: Tortoise seems to skip the staging step and treat all previously versioned files as automatically staged. I usually use git from the command line, so I'm not sure how to use a staging environment with Tortoise. If you really want that, and can figure it out, it would be a nice addition to this documentation.
  9. You'll see a message that says you need to set your name and email address.
    It'll open up the git config window for you to enter these. Just enter them at the top and then click the Global checkbox below so that it doesn't ask you every time you create a new repository.
  10. Enter a commit message and click "OK." and then click "Close"
  11. Your file should have a green check mark icon next to it.
  12. Open your score, make changes and save it.
  13. Your file should have a red Exclamation point next to it.
  14. To see what changed, right click on it and hover over "Tortoise Git >" and click "Diff".
    This will open up a window that will show you what changed. Unfortunately, it's comparing the XML source code, so you won't actually see a WYSIWYG view of how the score was changed, so this is only marginally useful at the moment.
  15. Periodically when you have a successful save, commit your changes.
    1. Right-Click on the file
      (or if you're working with multiple files, you can go up a level in Explorer and commit all changed files in the repository at once by right clicking on the folder),
    2. click "Git Commit... -> master..."
  16. Enter your commit message.
    This is a good place to log what you changed so you have a record of it.
  17. Click "OK" and "Close"

Now if you do have a corrupted score, do the following:

  1. Find the file in Windows Explorer. It will have the Red Exclamation Point
  2. Right click on it and hover over "Tortoise Git >"
  3. Click "Revert." Your file will be restored to the previously committed state.
    You will lose all progress you made since the last time you committed, but if you think about it, you've already lost it. This is just limiting the damage. It's a lot better than losing everything, eh? Especially if you're working on Movement 5 of a symphony and your file gets corrupted! Yikes. So, you will have to determine how often you commit based on how much you're willing to risk losing. If you're super paranoid, you can commit after every save. A little less so and you can commit when you've reached "milestones" in your project. It's really up to you.

So the last thing that remains is to suggest native integration as a feature request. Just using Tortoise to interface with git solves the problem of not having a backup, makes it relatively easy to make and restore backups, and limits the amount of drive space your backups take up while still preserving a detailed change log, which provides additional features. But it's still a bit of manual work to do these backups. It smooths out when you get the hang of it, but it might cause headaches for some at first.

If versioning was integrated into MuseScore natively, however, it would/could provide the following benefits:

  1. A menu/button to version your files without having to find them in Explorer.
  2. Allow the user to configure their versioning experience
    (Use different scm systems like git, hg, svn; Use different hosts, github, bitbucket,etc).
  3. Provide a WYSIWYG diff interface to show changes as musical scores rather than just xml code for easier diffing and merging.
  4. Further, MuseScore.com could provide their own github/bitbucket clone that provides additional music related features for easier collaboration with remote teamates
    (like pull requests that show the diffs as scores rather than just code).

Thanks for the read and considering this. If nothing comes of this other than helping someone else get some security and peace of mind without having to worry about losing their work, then I'll be happy.

** Yes, I know that MuseScore does somewhat do its own backup with the ".[ScoreName].mscz," files, but as far as I can tell, it only backs up when you open the file, and not every time you save, so if you have a whole day's work, for example, you could still lose a ton of progress.


Comments

This is a feature that I'm missing ever since I started using MuseScore some 4 years ago!
A first step in that direction might be to be able to make .mscx the default extension/format when saving rather the current .mscz
A fully blown implementation might (and probably should) contain the entire git repository for that score in the .mscz file.

Status (old) active postponed

Thanks for your very handy tutorial. Some things to consider though.

- MuseScore shouldn't crash. If you encounter a reproducible crash, please report it. Especially 0b files should not happen...

- MuseScore doesn't compress MSCX only for reducing their size. The MSCZ file is a container and also contains the images inserted to the score and has provision to store more info such as audio files. Making MSCX the default format is a very bad idea.

- For the vast majority of MuseScore users (and for some of the developers a year and half ago), git is nothing but an english word... It's a pity but it's the case. So in order to backup their scores I would advise my musician friends and MuseScore users to use MuseScore.com (Disclaimer, I'm one of the cofounder, and we need to make a better job of saving several versions there) or Dropbox, or a DIY replacement (Owncloud, Sparkleshare etc...)

- Regarding diff and versioning (but not backup or git), there is a bit of code in the current development version of MuseScore. It will not make it to 2.0. But if someone wants to pick this feature, he is very welcome. Note that it won't be an easy job, at all. Contact me on IRC if you need guidance #musescore on freenode.net.

I put this issue on postponed. You could report the git part to the General discussion forum where it will be more visible.

Seems to me this isn't a MuseScore issue but a much more general one. Why should MuseScore be responsible for how its files are stored? It doesn't implement its own tree-structured directory scheme; it uses what the OS provides. Seems if you want files to be automatically versioned, that should happen at the OS level where it can benefit all application. And my impression is that any number of cloud storage facilities do exactly this. For example, if you store your MSCZ files on Dropbox as suggested by lasconic, or Google Drive, or whatever the current flavor-of-the-month favorite is, don't they keep older versions around for you? Maybe not with fancy logging (no opportunity to enter a message explaining he reaosn for the change), but it seems that is a facility they could someday provide. So that's where I'd be directing my energies if I were tryng to pish for this to happen.

Well, some versioning would be very usefull, e.g. when collectivly working on a score, how else would one be able to find the differences between 2 scores, other that viewing them side by side and comparing them opticaly and listenening to them? Very cumbersome...
On .mscx files one could at least run diff against them, but much better would be a "visual diff" inside MuseScore

It is already very easy to use "normal" version control with Musescore, even with compressed files Musesccore files (.mscz), and as Marc said, there is no need to have Musescore acting as a version control system.

If you have unlimited disk space, you can just forget all that diff business and store every revision of your file in your version control system. This is extremely wasteful because even if you change a single note in a 70kiB score, your commit will increase your repository size by 70kiB anyway, instead of a few bytes. If you have unlimited space, this is not a problem at all.

Back into the real world, where you don't have unlimited space...
There would be no problem is you save your scores in .mscx format, because version control systems can diff this file pretty well. But if for some reason you don't want to save your files as .mscx, there are some ways to get meaningful diffs.

If you are using Mercurial, you can use the zipdoc extension. It was originally written to be able to diff .docx and .odt files, which are nothing more than diffs of xml files. Because a .mscz file is just a zipped .mscx file, the extension will work just as well with Musescore. This extension allows you to work with .mscz files with meaningful diffs.

If you're using another version control system there is probably a way to handle zipped files.

The only problem with this setup is the fact that musescore stores a PNG thumbnail of the score in the .mcsz file, and PNG files can't be diffed thumbnails are around 25kiB. This means that whenever you edit the file in a way that changes the first page of the score, you repository (unnecessarily) grows by at least 25kiB (compared to < 1kiB for small changes that don't affect the thumbnail).

This could be avoided if Musescore didn't store the thumbnail in the file. I can't see the use of storing thumbnails in the file, except for the window that appears when you start Musescore that shows the most recent scores, and even those thumbnails could (should?) be stored in a dedicated location and not in the file itself.

Is there something that forces us to store thumbnails in the file? If there isn't, I'll submit an feature request for making it optional.

On the other hand, It would be useful to be able to use Musescore as a *diff viewer*, like kdiff3 for text files or nbdiff for ipython notebooks, but this is probably hard to implement. This functionality is of course independent of the actual version control part, which is best handled by something like hg, git or fossil.

I vote for GEB's suggestion. Musescore doesn't need to have a built-in versioning system, but if it can compare multiple source files and highlight what's the difference, would be pretty nice.