json batch convert job should continue after error

• Dec 19, 2019 - 12:53
Reported version
3.3
Type
Functional
Frequency
Few
Severity
S5 - Suggestion
Reproducibility
Always
Status
active
Regression
No
Workaround
No
Project

Error:

convert ...
importMidi: bad file format
Cannot read file /Users/sheerun/Downloads/130000_Pop_Rock_Classical_Videogame_EDM_MIDI_Archive[6_19_15]/Classical Archives - The Greats (MIDI)/Rachmaninov/Prelude op32 n5 .mid:
bad format
[1] 24347 segmentation fault ./mscore -j jobs.json

Command used:

./mscore -j jobs.json

Job in file:

{
"in": "/Users/sheerun/Downloads/130000_Pop_Rock_Classical_Videogame_EDM_MIDI_Archive[6_19_15]/Classical Archives - The Greats (MIDI)/Rachmaninov/Prelude op32 n5 .mid",
"out": "/Users/sheerun/Music/Songs/Classical_Archives_-The_Greats_MIDI--Rachmaninov--Prelude_op32_n5.mxl"
},

I attach file that caues crash.

I'd expect that musescore fails to procress this file but doesn't crash, it should just proceed processing all other files.

Also if directories of out files don't exist, they are not created and files are not written. In this case it doesn't cause either crash nor error message in console. I'd expect at least error message or even better, the the target directory is automatically created.

Attachment Size
Prelude op32 n5 .mid 14.24 KB

Comments

The directory issue is separate (and minor or suggestion)

I can't reproduce the crash problem at all under Windows


C:\Program Files\MuseScore 3\bin\MuseScore3.exe" -j C:\Users\Jojo\Desktop\jobs.json
C:\Users\Jojo\Desktop>copy foo.json con
{
"in": "C:/Users/Jojo/Desktop/Prelude op32 n5 .mid",
"out": "C:/Users/Jojo/Desktop/Classical_Archives_-The_Greats_MIDI--Rachmaninov--
Prelude_op32_n5.mxl"
},
1 Datei(en) kopiert.

Nothing happens, no crash and no conversion

I'm on MacOS 10.14.6

Darwin sheerun.dev 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64

OS: macOS 10.14, Arch.: x86_64, MuseScore version (64-bit): 3.3.4.24412, revision: 7684abe

It may or may not, the point if this issue is that batch processing fails if single item fails, I don't mnid this particular file fails to be processed, but that whole batch of tasks fails to be processed because of one error

Title Segmentation fault for batch convert job json batch convert job should continue after error
Severity S3 - Major S5 - Suggestion
Status needs info active

Garbage in, garbage out...

In reply to by Jojo-Schmitz

I'd kindly ask to upgrade it from "Suggestion" level. For me this is definitely a bug, not a suggestion, because it makes batch processing unusable in my case because I have 10000 midi files to process and if 100 of them are broken I need to re-start batch processing 100 times and each time remove the file that batch processing stopped on.

Workaround Yes No

> Midi files that realy are wav files will cause this and it isn't MuseScore's fault at all. Garbage in, garbage out.

This issue is not about MuseScore crashing on single file, but crashing whole batch because of single file

> Try the batch convert plugin, that should continue on errors.

I tried it even before, it's even worse as each error in process requires me to click 2x on screen. Skipping error is not automatic there as well... Could you up the severity?

This is likely the most crippling bug I've ever had to deal with any open source software, do you have any idea how serious is it? Let me put it this way: Musescore have an exclusivity on some file formats that only MuseScore can read and handle (ie. MSCZ), it created an entire ecosystem where workers and clients exchange files created with this software on a regular basis and are so much dependant on it (think of Adobe) that they have no real alternative, how am I supposed to EVER successfully convert hundreds-of-thousands of MSCZ files to MUSICXML in a reasonable time if the batch converter feature is broken? This is literally threatening my future and career, I would love to just throw money for an alternative software that can deal with this but there isn't one because all source files are in MSCZ format. When a file causes the batch conversion process to crash, there's no way to resume automatically from where it crashed, not in a reasonable amount of time.
If I process the files with the "musescore3 -j jobs.json" command, any crash will stop the entire process, if I run a batch process like "musescore3 input[x].mscz -o output[x].musicxml" (where [x] is an incremental number) I can ignore errors but the process is extremely inefficient as MuseScore will have to completely restart over and over for every single file, it will work but it's so ridicolous that it feels like rotating a house just to screw a lightbulb.
The GUI batch conversion plugin is even worse as sometimes a file will cause it to opens a dialog that requires the user intervention (ie. choosing to ignore, retry or cancel), but will crash anyway when it hits a problematic files.
I attached all problematic files here, the archives names are self explanatory, about the symptoms: some files can be opened with MuseScore but will make the batch process crash, some will crash both MuseScore and the batch process, some files were saved in the wrong format even thought the MSCZ file format was chosen ( Don't believe me? Go to https://musescore.com/user/11430251/scores/2647696 and download this as MSCZ and try to open it, then rename it to MSCX and try again )
Hope this can be fixed once for all, it's been ignored for too long, it's making lives miserable.

download this as MSCZ and try to open it, then rename it to MSCX and try again How is renaing supposed to change the file format? (yes, this works for .musicxml and .xml, but that is the exception rather than the rule)?

In reply to by Jojo-Schmitz

On Dec 19, 2019 - 23:06 you said "Midi files that realy are wav files will cause this and it isn't MuseScore's fault at all. Garbage in, garbage out."
But I disagree with that statement, my point was proving that even the most reliable websites such as MuseScore can still host garbage files, how am I supposed to defend or know that before hand that a file is going to be garbage? (I asked Musescore to download a MSCZ file and received a fake MSCZ instead, I had to manually examine it to find it was actually a MSCX file) The point of a batch process is to minimize human interaction as much as possible, so a failback measure is required because it's just plain impossible to keep the garbage out when dealing with thousands of files, what's even the point of having a batch process at all if I need to manually check which files are garbage one by one?

No need to manually check, that json comvert will tell you...
If musescore.com gives you an mscx file vut witj an mscz extension, or a wav with mp3 extension, this is a bug and needs to get reported and fixed there.

In reply to by Jojo-Schmitz

>No need to manually check, that json comvert will tell you...
That's not what happens here, it will silently crash and do nothing at all with the remaining files.

Given a JSON file like this: [{"in": "haveproblem.mscz", "out": "haveproblem.musicxml"}, {"in": "noproblem.mscz", "out": "noproblem.musicxml"}]
If I run "musescore3 -j convert.json" nothing happens because it stops/crashes at "haveproblem.mscz"

But if I invert the order like this: [{"in": "noproblem.mscz", "out": "noproblem.musicxml"}, {"in": "haveproblem.mscz", "out": "haveproblem.musicxml"}]
And run "musescore3 -j convert.json" again, it will generate "noproblem.musicxml" but stop/crash when it hits "haveproblem.musicxml".

This nullifies the convenience of a batch process as it keeps crashing and having to manually restart after editing your json file to ignore the problematic file. I just want to start the process and not have to keep a watch on it.

In reply to by Jojo-Schmitz

This bug isn't exclusive to garbage files as we thought initially, also happens with perfectly working files that were saved with versions older than 2.0.0.
If you open the attachments with MuseScore 3, an error message appears: https://i.imgur.com/3WKA1QV.png however you can click "Ignore" and everything loads just fine.
The batch converter doesn't have the ability to answer "Ignore", thus it stops on its tracks when it encounters such files.
Turns out the problem is more serious because even not feeding it garbage isn't a solution anymore.

Attachment Size
432.mscz 9.26 KB
446.mscz 8.86 KB
592.mscz 47.65 KB