Recognition of score from image or PDF

• Dec 13, 2019 - 14:09

Hey! I'm Max from internal research team. I thinking about adding feature "recognition of score from image or PDF" to MuseScore (editor).

How it may works:

  1. User select image or PDF in MuseScore.
  2. Start processing...
  3. User see this in score window:
    score_recognition_example.png
  • Green: recognized elements.
  • Purple: recognition engine recognized that there is an element, but didn't recognize which one.
  1. Purple elements aren't valid symbols, they are just placeholders. In place of these placeholders, user must drag the real elements from MuseScore editor.
  2. When user click save file then all green elements and new elements placed from user are saved to file.

What do you think? Will be it helpfull for transcribers? Will you use it?

How it works from developer point?

It will be use Audiveris engine for recognition. When user select image file, when we run console version of Audiveris and get .omr file from here. Then we:
- read .omr file,
- create new blank score,
- put recognized elements
- draw placeholders from points

I have few unresolved issues:
- Audiveris is licenced by AGPL v3.0 which incompatible with GPL 2.0. I thinking about adding this feature as separate plugin or make cloud version of Audiveris.
- Audiveris is written on Java and have large size. I wouldn't want to increase size of MuseScore distribution.
- I need to make .omr file reader for MuseScore. Audiveris can export files to MusicXML, but some part of data may be lost.

P.S. Yes, I know that MuseScore 2 had own OMR engine. But it can recognize only staves, which is clearly not enough. Also there's no placeholders for unrecognized elements.
P.S.S. Yes, I know that musescore.com have pdf2score feature, but it works on Audiveris 4 and didn't works well. When user sees emptiness instead of the recognized elements, he thinks that something is broken. If we show: "here's should be some element, but we don't know which one" - I think it be much more user-friendly.


Comments

Do you really want to listen to a million "I tried to read this pdf and couldn't understand it" complaints that you will now inherit?

Audiveris is indeed much more powerful than one might be led to believe from the attempt at automatic conversion provided by the web site. The actual Audiveris interface has many controls but the average user doesn't want to have to install and learn a separate program. If there is some way to integrate Audiveris more closely so that you can control more about the conversion, that would be wonderful. But, I'm still not really clear from the description above how that would work in practice. Would be nice to see a video demo, if you have a working prototype or even just a mockup.

In reply to by Marc Sabatella

Hoo, boy! PDF to mscz. converting is not an easy thing to do, by any means (especially with the often horrendous differences in quality of the PDF), but I'm glad that you guys are still trying, as it would be a great feature to have. However, I'm not expecting that it'll be perfected anytime soon, considering the hurdles, both known and yet to be discovered, that are repeatedly encountered along the way.

I still appreciate the effort, though. Keep up the good work!

I would love to see some interactiveness in the processing of a score. Sometimes I try to import a score in order to make it better readable for me with MuseScore. Sometimes the result works almost out of the box, but all to often the scanning program doesn't properly recognize the time signature. That is my main problem.
For me, it would already be great if I was able to correct the scanned time signature before I get the final result of the score.
By the way, the pieces I use, have usually only one or two voices, so that may explain why I'm happier with the beta scanning function than some others. I tried to find out how to get Audiveris running on its own, but it was too complicated for me to understand what the manual was trying to explain.

In reply to by Marc Sabatella

I completely agree. if only I understood how to install the program and the necessary additional programs. If only there was a step for step manual for people who really want to learn, but who don't understand where they can find the information they need in order to be able to follow the steps of the manual.

In reply to by Wil Weten

I just did a conversion with the standalone version of Audiveris. The recognition was without mistakes. So exported the result to MusicXML and imported to Musescore. The notes were okay, but all texts - mainly fingering - were other sorts of texts. The fingerings were converted to song-texts. The from Audiverdis exported xml-File is not readable in a normal text-editor.

Next surprising fact. I tried to import the notesheet via Musescore with the Import PDF funtion. The result of the notes were to bad for use. See the attachment. The original score was done with musescore. So the plugin for musescore crash notesheets done with musescore.

Attachment Size
Standalone.mscz 15.6 KB
Plugin.mscz 23.12 KB

In reply to by hasenfuss

Plugin.mscz is corrupt:

Measure 12, staff 1 incomplete. Expected: 2/4; Found: 104/192
Measure 13, staff 1 incomplete. Expected: 21/32; Found: 164/192
Measure 14, staff 1 incomplete. Expected: 17/32; Found: 116/192
Measure 40, staff 1 incomplete. Expected: 5/8; Found: 152/192

but loads when ignoring those, no crash, but quite messy and useless it seems.

Why would an XML not be readable? CR/LF issue maybe? If so use a better editor, like NotePad++
Also please attach the XML here

In reply to by hasenfuss

Yes, and that indeed is a bit of a problem child, hence the recommendation here to use the standalone Audiveris rather than musescore.com's PDF import.
You can edit that mxl after having extracted the xml from it, mxl is a compressed (ZIP) format

Do you still have an unanswered question? Please log in first to post your question.