Trying to develop a plugin for piano students which can simultaneously help check if the playing finger is wrong
Hi guys,
In musescore we can add fingering mark above each note to help kids learn piano faster. I'm thinking to create a plugin which can read the finger number from camera simultaneously, then compare with the correct fingering mark we added in musescore.
I think the tricky part maybe how to track the finger number in real-time, then synchronously talk to musescore playback, here is some resources I found:
Tracking a pianist's fingers with model-based computer vision:
https://www.youtube.com/watch?v=yFxH3zIHTyU
Finger Counter using Hand Tracking | Computer Vision | OpenCV Python 2021:
https://www.youtube.com/watch?v=p5Z_GGRCI5s
Any ideas and suggestions would be helpful, thanks!
Comments
I see several tricky parts here:
1) The real-time tracking of the fingers. The example you mentioned is in Python. Transposing this into the limited javascript set of instructions available in MuseScore might be hard.
2) Controlling MuseScore playback with the API, (like stopping it, resuming it (°)) , is not possible.
So my suggestion would be to a solution build in Python were you read/import the Musescore file and do all the matching there. Check this question which is not the same as you but is related (the requester would like also a realtime analyse of the user input and compare it to a score), and the answer by Jojo-Schmitz
(°) which is, as far as I understand, what you mean by "then synchronously talk to musescore playback"