Recognition of score from image or PDF
Hey! I'm Max from internal research team. I thinking about adding feature "recognition of score from image or PDF" to MuseScore (editor).
How it may works:
- User select image or PDF in MuseScore.
- Start processing...
- User see this in score window:
- Green: recognized elements.
- Purple: recognition engine recognized that there is an element, but didn't recognize which one.
- Purple elements aren't valid symbols, they are just placeholders. In place of these placeholders, user must drag the real elements from MuseScore editor.
- When user click save file then all green elements and new elements placed from user are saved to file.
What do you think? Will be it helpfull for transcribers? Will you use it?
How it works from developer point?
It will be use Audiveris engine for recognition. When user select image file, when we run console version of Audiveris and get .omr file from here. Then we:
- read .omr file,
- create new blank score,
- put recognized elements
- draw placeholders from points
I have few unresolved issues:
- Audiveris is licenced by AGPL v3.0 which incompatible with GPL 2.0. I thinking about adding this feature as separate plugin or make cloud version of Audiveris.
- Audiveris is written on Java and have large size. I wouldn't want to increase size of MuseScore distribution.
- I need to make .omr file reader for MuseScore. Audiveris can export files to MusicXML, but some part of data may be lost.
P.S. Yes, I know that MuseScore 2 had own OMR engine. But it can recognize only staves, which is clearly not enough. Also there's no placeholders for unrecognized elements.
P.S.S. Yes, I know that musescore.com have pdf2score feature, but it works on Audiveris 4 and didn't works well. When user sees emptiness instead of the recognized elements, he thinks that something is broken. If we show: "here's should be some element, but we don't know which one" - I think it be much more user-friendly.
Comments
Do you really want to listen to a million "I tried to read this pdf and couldn't understand it" complaints that you will now inherit?
Audiveris is indeed much more powerful than one might be led to believe from the attempt at automatic conversion provided by the web site. The actual Audiveris interface has many controls but the average user doesn't want to have to install and learn a separate program. If there is some way to integrate Audiveris more closely so that you can control more about the conversion, that would be wonderful. But, I'm still not really clear from the description above how that would work in practice. Would be nice to see a video demo, if you have a working prototype or even just a mockup.
In reply to Audiveris is indeed much… by Marc Sabatella
Hoo, boy! PDF to mscz. converting is not an easy thing to do, by any means (especially with the often horrendous differences in quality of the PDF), but I'm glad that you guys are still trying, as it would be a great feature to have. However, I'm not expecting that it'll be perfected anytime soon, considering the hurdles, both known and yet to be discovered, that are repeatedly encountered along the way.
I still appreciate the effort, though. Keep up the good work!
I would love to see some interactiveness in the processing of a score. Sometimes I try to import a score in order to make it better readable for me with MuseScore. Sometimes the result works almost out of the box, but all to often the scanning program doesn't properly recognize the time signature. That is my main problem.
For me, it would already be great if I was able to correct the scanned time signature before I get the final result of the score.
By the way, the pieces I use, have usually only one or two voices, so that may explain why I'm happier with the beta scanning function than some others. I tried to find out how to get Audiveris running on its own, but it was too complicated for me to understand what the manual was trying to explain.
In reply to I would love to see some… by Wil Weten
For that level of control, I definitely recommend installing Audiveris yourself and using it directly. It provides much more sophisticated control than the automatic web service can allow. Yes, it's complicated, but well worth learning if you wish to rely on it.
In reply to For that level of control, I… by Marc Sabatella
I completely agree. if only I understood how to install the program and the necessary additional programs. If only there was a step for step manual for people who really want to learn, but who don't understand where they can find the information they need in order to be able to follow the steps of the manual.
In reply to I completely agree. if only… by Wil Weten
I just did a conversion with the standalone version of Audiveris. The recognition was without mistakes. So exported the result to MusicXML and imported to Musescore. The notes were okay, but all texts - mainly fingering - were other sorts of texts. The fingerings were converted to song-texts. The from Audiverdis exported xml-File is not readable in a normal text-editor.
Next surprising fact. I tried to import the notesheet via Musescore with the Import PDF funtion. The result of the notes were to bad for use. See the attachment. The original score was done with musescore. So the plugin for musescore crash notesheets done with musescore.
In reply to I just did a conversion with… by hasenfuss
Plugin.mscz is corrupt:
Measure 12, staff 1 incomplete. Expected: 2/4; Found: 104/192
Measure 13, staff 1 incomplete. Expected: 21/32; Found: 164/192
Measure 14, staff 1 incomplete. Expected: 17/32; Found: 116/192
Measure 40, staff 1 incomplete. Expected: 5/8; Found: 152/192
but loads when ignoring those, no crash, but quite messy and useless it seems.
Why would an XML not be readable? CR/LF issue maybe? If so use a better editor, like NotePad++
Also please attach the XML here
In reply to Plugin.mscz is corrupt:… by Jojo-Schmitz
Hier bitte.
Edit. Dir ist klar, dass Plugin.mscz via Import-Pdf erzeugt worden ist?
In reply to Hier bitte by hasenfuss
Yes, and that indeed is a bit of a problem child, hence the recommendation here to use the standalone Audiveris rather than musescore.com's PDF import.
You can edit that mxl after having extracted the xml from it, mxl is a compressed (ZIP) format
I am trying to speed up the transcribing of old manuscripts, like for example the Rost-codex (see IMSLP). If you simply don't expect an OMR to be able to "read" a handwritten score, the task becomes designing a GUI for interactively transcribing a handwritten score. Hence I made my own application where only the staffs are recognized and printed on top of the original image. Next I can click all the different music symbols in the image like barlines, crotchets, quavers, etc. The clicking order is not important. When a note is clicked in the image, a coloured dot is shown, hopefully automatically correctly placed on the stave, otherwise it can be moved using arrow keys. When finished clicking all the symbols a MusicXml is generated. The main advantage of this approach is that I can click in the original image. It would be nice if a more professional GUI could be made for such an application. I tried your integrated OMR conversion software, however it stopped processing my handwritten score. Attached is a screenshot of my app.