Migrate MuseScore windows nightlies from a dedicated server to appveyor

• Apr 14, 2016 - 09:53
Reported version
3.0
Type
Functional
Severity
S4 - Minor
Status
closed
Project

Travis has benefited us a lot in the past 2-3 years but it only build Linux nightlies. It would be good to have something similar for Windows and be able to run the tests and create packages on a hosted platform. Appveyor.com looks like a good candidate.

ABL already did some investigation: https://github.com/musescore/MuseScore/compare/master...AntonioBL:appve…


Comments

Sorry that I had almost no free time in these days to continue this investigation.
What I basically found is that the main problem is the timeout they set at 60 minutes.
With the installed software the timeout was reached before finishing the compilation.
Therefore, I enabled pre-compiled headers in MinGW by using the trick (hack) described here:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=56926#c17
With this hack the compilation time fell to ~51 minutes. I also cached the "thirdparty" folder of the build folder. The compilation time fell to ~40 minutes, and also uploading to BinTray went fine.
At the moment I am fecth the dependencies (and the hacked cc1plus.exe) from my Google Drive.
I tried with ccache, but compilation time did not change much (it is still enabled in the branch at the moment). I also tried setting ccache sloppiness to "pch_defines,time_macros" (by setting env variable CCACHE_SLOPPINESS) but testing this on my laptop I found that it increased a lot first compilation time (most probably exceeding the 60min limit in the Appveyor compilation), it creates a huge ccache folder, and does not change much second compilation time.

In conclusion, I think that Nightly builds can be successfully compiled on Appveyor and uploaded similarly to what is currently done for Linux Nightly, but I don't know if we can compile both release and debug build because of the timeout.

P.S: Many thanks to ericfontainejazz for the tutorial about BinTray!

the cc1plus.exe from your google drive apparently is gcc 4.9.1, should rather be 4.9.2 (part of Qt-5.6), shouldn't it?
I tried it anyway and compared the times for a complete build of master from scratch on a 4-core machine (using -j4); with PCH and the patched compiler it took 15 minutes, without (and the original compiler) 35 minutes. Quite a significant improvement!

It is gcc 4.9.1 because I was using it before the switch to Qt5.6. In that case the hack is analogous.

I didn't like very much the fact that the dependencies are downloaded from an external zip archive, so I tried to investigate the use of msys2.
I still have to copy the needed Jack files, but all the other dependencies are correctly installed by msys2 :-)
Locally, by launching it inside msys2 terminal, it works (it gives problems about the dlls to be installed since those shipped with msys2 have different names or positions (and some additional dlls are required since msys2 Qt is probably compiled with other flags than the official Qt), but that's a trivial correction. I had to hack one of the Qt .cmake files of msys2 because it was giving an error.
Inside Appveyor, I have an error during the installation step, due to the fact that it is defining the PATH using Windows style instead of msys style (=Unix style).
See here: https://ci.appveyor.com/project/AntonioBL/musescore/build/1.0.59
related to commit ab733eb13caa6

With a couple of additional changes I can also compile the 64bit version with msys2.

Morevoer, I also tried to investigate the static build of MuseScore with msys2 static Qt5.6 build. At the moment I have a very experimental branch which is compiling MuseScore statically in 64bit and the application more or less works. I still have problems with the plugin framework, so I disabled for it; and I still have to add the additional Q_IMPORT_PLUGIN (for svg and printing, for example).
The exe file in release mode is ~86MB (but some Qt plugins and the qml part are still missing), and runs without any additional dll. (I had an error about the svg export class, since it is a duplicate of the internal Qt class; I commented one line to make it pass, but sometimes when recompiling it still gives error, I don't know why)
Here it is:
https://drive.google.com/open?id=0BxjayMZiuupOZ2JnWHdTY1FHOHM
as I said, it is still very experimental, something is probably missing and something is probably superfluous (the qml directory?).

@Jojo-Schmitz : You should search for hexadecimal sequences (these are the same also for gcc 4.9.1):
81FA00000008
81FF00000008
C744240400000008
In the 4.9.2 version I am using (not the Qt one) they can be found at addresses:
003A0D67
003A0EAF
003A0D7F
I searched for and modified them with HxD.
Hope this helps.
Ciao,
ABL

Thanks, with that I got it patched (same addresses BTW).
Full build of MuseScore from scratch in just 6 minutes, on an 8 core PC running Windoes 7 Enterprise, 64bit, with 8GB RAM, using -j8 and with PCH enabled, vs. 11 minutes without PCH.

Now the linking becomes the bottleneck, taking about 1 minute ;-)

Now 2.2 branch is also built on appveyor.
We now need to switch http://prereleases.musescore.org/windows/nightly/ to https://ftp.osuosl.org/pub/musescore-nightlies/windows/ and we are done.

In the process, I also tweaked the webpage to separate the downloads link per branch.

We now have all nightly builds generated publicly on travis (Linux/Mac and tests) and on Appveyor and all code is available on github. I hope it will bring more contribution to improve and maintain the build process.

A nice side effect of the Appveyor setup: Pull requests are also packaged and everybody should be able to download the resulting binary and test the PR !
For example, this PR https://github.com/musescore/MuseScore/pull/3364 created this binary: https://ci.appveyor.com/project/MuseScore/musescore/build/1.0.10/artifa…