"sing your melody" - Is there MIDI and/or audio input for plugins?

• Dec 18, 2019 - 02:13

(Apologies, if this has been suggested before - please direct me there then.)

Is a MIDI input or audio input available for plugins?

I am sporadically thinking about starting tinkering with a "sing/hum your melody" type plugin. It would record audio, analyze (find pitched notes), then interactively transcribe it in cooperation with the user.

First the plugin draws the notes on a continuous time (no tempo, no measures, no beats) vs. MIDI-note-number-graph, and allow the user to pull the notes into place in pitch and in loudness/velocity and in time - with the possibility of playback of the current state - until they are satisfied with the notes sounding as they want them.

Second comes the transcription into a score. Here the user overlays tempo/beats/measures/clicktrack/whathaveyou grid on the time axis, and flexibly massages the played notes and/or the tempo/measures into the musically correct and meaningful timings and lengths; if there is a tempo change, the user can "cut" the tempo grid into sections, and drag the grid wider or narrower if needed, with some musically meaningful snapping of the tempo grid to the existing notes (in an advanced version, even gradual tempo changes could be drawn like this). Ideally, a mini-score/staff is continuously shown below to show (at least partial) feedback about how the transcribed score would look, so that the user can be satisfied with the note lengths and positioning, and that the score concisely but precisely captures their actual musical idea.

Finally they can commit the thusly transcribed piece into the underlying actual score.

There could be an option to keep the audio recording (or perhaps just the extracted pitch contour) and other settings attached to the score, so that the user can return to it to amend transcription parameters when later needed.

Useful as a practical tool for getting musical ideas more directly from instrument to score - but even more importantly as an educational tool, to very graphically help beginner musicians/composers learn how notes/pitches they have in their head and can sing or play, actually convert into rhythms and notes in a score.

For MIDI input, the pitch estimation step is omitted, but beyond that the experience would be pretty much the same.

Not a small feature for sure. But we could work through it in multiple iterations, start with an MVP (the smallest set of features that are already useful), and gradually enhance.

Comments

• Dec 18, 2019 - 18:05

Sure, I'd use it, as well as the machine that correctly predicts the outcome of horse races, and the machine that copy-pastes funds. It's imaginable in a sci-fi movie, just like HAL-9000. Clearly, it is doable in principle, just like any other AI vision, but it has nothing it needs in the MuseScore code base; MuseScore is not an imaginary AI projection (and nothing wrong with those), but a product that already has specializations, and audio input and understanding is nowhere in its repertoire or capabilities. In 100 years, maybe such programs can transcribe a recording of an orchestra to a score. Not today.

Reply

• Dec 18, 2019 - 21:32

No, I don't think so. As I said, there are experiments and academic apps that can do some aural transcription, but that is a ball of functionality that is really conceptually divorced from what MuseScore does. Just my opinion. I don't think there's much more to discuss about it. Surprise me and implement a gnignis (singing spelled backwards) plugin.

Reply

• Dec 18, 2019 - 22:07

Plugins are currently capable to communicate with external processes. If you implement recording (and maybe analyzing) audio samples in some external program it should be possible to get it working together with MuseScore plugin to enable such kind of note input.

Reply

• Sep 15, 2020 - 07:44

I studied this question some time ago since I am a practically fingerless what comes to playing a keyboards, therefore being able to hum notes would be a great help. There is already an audio to midi software named A2M which works like a virtual instrument, one can use any sound source which pitch is adjustable, guitar, piano, whistling, humming, chain saw, you name it. I collected the results to this link:

https://musescore.org/en/node/291830

The trick is reconnecting midi inputs and outputs in suitable way using loopMIDI, a nice piece of software. A remaining problem was Musescore metronome which comes to same output channel and causes a machine gun effect because of looping. I already asked before whether the developers could make it adjustable, user could choose another channel for it to prevent the problem. In long run the entire loopMIDI should be unnecessary piece because MusesScore should have adjustability in input and output channels. I recall that in some early versions this was possible but no more.

I am following this discussion a bit frustrated, for me it is absolutely irrevelevant how a musician turns her/his ideas to sheet music, I can't understand the resistance against providing such possibility as all other pieces are available in open source or free code but Musescore is not capable of using the generated midi signal properly to write notes to staves. Sort of arrogance maybe, "you have to learn writing sheet music the hard way, MuseScore is a notation program, not a toy for whistlers"?

If Musescore writers don't want to help with this matter, which to my understanding is not rocket science for a person who knows the midi module code in Musescore (I don't and don't either have time to learn the entire development environment), another possibility atop of commercial products is jniz ( www.jniz.org), developer is doing a great job. The code runs fine in some environments, although I have found some issues in my Win laptop, but developer is aware of it and tries to sort it out just now.

Reply

• Sep 15, 2020 - 16:33

MuseScore is definitely capable of notating a MIDI signal to write notes to staves (properly is very subjective).
The counterpoint made is that this input method so far is requested not enough to warrant spending implementation time on it; knowing that there are other excellent free tools out there that perform the audio-to-MIDI functionality and that MuseScore can interface with MIDI data.

MuseScore being open source does not mean MuseScore should lose focus of the product it aims to be. As development time is not limitless, this also means not including every useful feature for every use case. Cfr with notation input methods; abc-notation for example is not natively supported by MuseScore; but there is a plugin for it.
A similar approach was suggested for your idea.

I'm sorry that you get frustrated by not seeing your idea take shape within the core of MuseScore. Whether it'd require rocket science or not is largely irrelevant. There are loads of outstanding feature requests (tempo lines for example) which don't require rocket science, will cater to a far larger audience and are closer in line with the core function of MuseScore.. but they've also not taken form (sometimes even after years).

To me, there is no arrogance in declining to include a new feature if the effort vs gain evaluation for those contributing to MuseScore currently turns out negative. It comes across more as trying to focus your limited time on providing the best possible quality of the things you can handle.

You explicitly mention not having the time for learning the MuseScore environment and code to try and connect your functionality request; but are somehow frustrated that other developers may make a similar statement about having to learn "all other pieces [which] are available in open source or free code"?

Now, with all that out of the way, I do agree that it should be possible to improve MIDI-out configuration handling of the metronome. I didn't find an open request for that in the issue tracker so a next step would be open an new item there and mark it as "S5 - Suggestion".

Reply

Do you still have an unanswered question? Please log in first to post your question.