"sing your melody" - Is there MIDI and/or audio input for plugins?

• Dec 18, 2019 - 02:13

(Apologies, if this has been suggested before - please direct me there then.)

Is a MIDI input or audio input available for plugins?

I am sporadically thinking about starting tinkering with a "sing/hum your melody" type plugin. It would record audio, analyze (find pitched notes), then interactively transcribe it in cooperation with the user.

First the plugin draws the notes on a continuous time (no tempo, no measures, no beats) vs. MIDI-note-number-graph, and allow the user to pull the notes into place in pitch and in loudness/velocity and in time - with the possibility of playback of the current state - until they are satisfied with the notes sounding as they want them.

Second comes the transcription into a score. Here the user overlays tempo/beats/measures/clicktrack/whathaveyou grid on the time axis, and flexibly massages the played notes and/or the tempo/measures into the musically correct and meaningful timings and lengths; if there is a tempo change, the user can "cut" the tempo grid into sections, and drag the grid wider or narrower if needed, with some musically meaningful snapping of the tempo grid to the existing notes (in an advanced version, even gradual tempo changes could be drawn like this). Ideally, a mini-score/staff is continuously shown below to show (at least partial) feedback about how the transcribed score would look, so that the user can be satisfied with the note lengths and positioning, and that the score concisely but precisely captures their actual musical idea.

Finally they can commit the thusly transcribed piece into the underlying actual score.

There could be an option to keep the audio recording (or perhaps just the extracted pitch contour) and other settings attached to the score, so that the user can return to it to amend transcription parameters when later needed.

Useful as a practical tool for getting musical ideas more directly from instrument to score - but even more importantly as an educational tool, to very graphically help beginner musicians/composers learn how notes/pitches they have in their head and can sing or play, actually convert into rhythms and notes in a score.

For MIDI input, the pitch estimation step is omitted, but beyond that the experience would be pretty much the same.

Not a small feature for sure. But we could work through it in multiple iterations, start with an MVP (the smallest set of features that are already useful), and gradually enhance.


Comments

No., and not likely ever. There are other applications from other vendors that listen to audio. MuseScore is a notation system.

In reply to by Marr11317

Sure, I'd use it, as well as the machine that correctly predicts the outcome of horse races, and the machine that copy-pastes funds. It's imaginable in a sci-fi movie, just like HAL-9000. Clearly, it is doable in principle, just like any other AI vision, but it has nothing it needs in the MuseScore code base; MuseScore is not an imaginary AI projection (and nothing wrong with those), but a product that already has specializations, and audio input and understanding is nowhere in its repertoire or capabilities. In 100 years, maybe such programs can transcribe a recording of an orchestra to a score. Not today.

In reply to by BSG

We are not talking about Musescore's code base, it's a plugin...
But yes, I agree, that some tools for that already exist, and that we should rather use them. Maybe a bridge to make it work directly within MuseScore wouldn't be so irrealistic though...

In reply to by Marr11317

Given the current architecture, there is no reason that a plugin could not launch an AI application that does this and writes music XML (which it then reads), i.e., the "bridge" you propose. But I wouldn't really call that a capability. But it wouldn't be very fast or convenient.

In reply to by BSG

not fast? Why not?
not convenient? Why not?
Hitting a shortcut, selecting an audio file and hitting enter seems pretty convenient to me if I get a rendered score.
I mean... Never having solfeggio homeworks would kinda feel like a dream... :)

In reply to by Marr11317

No, I don't think so. As I said, there are experiments and academic apps that can do some aural transcription, but that is a ball of functionality that is really conceptually divorced from what MuseScore does. Just my opinion. I don't think there's much more to discuss about it. Surprise me and implement a gnignis (singing spelled backwards) plugin.

Plugins are currently capable to communicate with external processes. If you implement recording (and maybe analyzing) audio samples in some external program it should be possible to get it working together with MuseScore plugin to enable such kind of note input.

Do you still have an unanswered question? Please log in first to post your question.