Video Player Plugin

• Jul 31, 2011 - 01:18

Today I decided I'm going to try to develop a video player plugin for MuseScore. I'm moving the old thread here from feature request, so far what I have, is I'm going to try to use the cursor object to get the video in sync with with the score, and I'm thinking of looking into Java video playback for the video player, if I can't find a simple open source video player in Java, I will try to write one.


I've been looking at the Java Media Framework (JMF) and I am going to work on making a simple video player, no controls accept a button to turn sound on and off. I'm going to probably be working on this for a while, never handled video in Java before, so it's new for me. I am also going to try to integrate Jffmpeg (Java ffmpeg) so that it can handle more formats other than AVI and MOV.

I'll keep posting my updates.

In reply to by KoRnKloWn

The time function will tell tell you the time position of the note the cursor is on, but again, that has nothing to do with what's being *displayed*. The score would just sit there not moving in any way while the video plays. There is no way to get to the score to play from the plugin architecture.

In reply to by Marc Sabatella

What I believe KoRnKloWn is talking about, is that the plugin itself wouldn't have a play button, but when the SCORE play button is pushed, if the video player plugin is open, then it would play along with it. Am I right?

There has to be something you can link it to. Maybe not the cursor since you really can't control that as it is now. But at least the score play button.

And what cursor are we talking about? Do you mean the blue bar that you follow across the score or the slider in the play panel?

In reply to by Gabriel H

That's pretty much exactly what I'm talking about. And I'm talking about the Cursor object:

Basically what I have in mind is it would use time() from the cursor object to return the location on the score in milliseconds, and relay that to the video player to set the video at the same time, and yes, it would be the main play button on the score that would start it, and I'm not sure how that would work yet, I need to find out how to get the player to watch it, some kind of action listener or something. But the video player part doesn't seem like it will be too hard, the Java Media Framework makes video playback pretty simple. It would be way too hard (if not impossible) to make the video player straight out of Javascript.

In reply to by KoRnKloWn

Actually now that I look at it, I'm not 100% sure the cursor object would work, the cursor is only created within the plugin, it's not referring to the blue cursor. But I'm still just going to worry about making the simple video player for now. If the cursor object doesn't work, there's got to be something that will.

In reply to by KoRnKloWn

That's what I mean when I say that the cursor object has nothing to do with what's displayed. There is no way to get any information about what is being played.

If you do create your own video player, you could certainly synchronize it from the beginning of the score. Or, if you've selected something, you can get at the selection. But I can't think of any way to start MuseScore's playback and your video player simultaneously. Synchronization would basically be limited to how fast you can manually press both "start" buttons.

In reply to by Marc Sabatella

Yeah, I just realized that, I was getting ahead of myself a little bit. But I still think there may be possibilities, maybe something more on the Java side, if I can at least find a way to make some kind of external action listener that can get the signal to start the video when the button is pressed on MuseScore, do you think that may be possible?

In reply to by KoRnKloWn

I suspect that even that might not be currently possible, but you might be able to implement a short patch to hook something like that up for a future release. Have you looked around much at the musescore source? Adding a method to an existing object type appears to be relatively straightforward, if it's something already supported within the application.

In reply to by Marc Sabatella

I'm really not sure, I haven't looked through the source, it's C or C++ right? I can try to look through it and see if I can make sence of it, but I'm very much a novice programmer, and mostly know Java, of course C and C++ are very similar to it. Besides, this will be a good learning experience for my self taught programming. I'll do whatever I can.

In reply to by KoRnKloWn

OK, I managed to get the source code into eclipse correctly, however when looking through it, I'm afraid this is just too much over my head at the moment. As I learn more programming I'll continue to check in to see what I can do, and if someone else hasn't already done or started it. In the mean time let's try to convince the main developers to implement, if not a full video player, then at least the required objects for us to make a plugin. All we need is a way to track the BLUE cursor so we can get it's current play position, and input from the controls (play, pause, etc.). And I'll continue to make a simple Java video player that can eventually be used with this, unless of course someone else writes a better one sooner.

Sorry I don't have more formal training, I've never taken a class on programming (other than HTML/CSS, but that hardly counts), everything I've learned myself from a few For Dummies books and the Internet. I definitely have a long way to go.

I don't think a plugin is the right approach to this problem. There have been discussion about implementing Jack transport ( in MuseScore. It seems a more viable road. Xjadeo or VLC could be used as external video player and be synced with MuseScore.

Okay so maybe not a plugin, but what about implementing the video player in the play panel (shortcut key -F1, and having it sync to the time display or slider bar?
I think the play panel would be the most obvious place to have a video window anyway. It saves on space.
What I am talking about is different then the blue cursor right?

In reply to by Gabriel H

I think it does make sense to have the video player in the play panel, but that's just a location for where the video player can be displayed, unfortunately it doean't help us much. And about the blue cursor, all it is is a visual representation of what's being played, like the little tracker on a video or music player.

Also, I'm not sure Jack would work either, because as far as I know it doesn't necessarily have support for syncronizing play time, it just transports media from one app to another. Really, I don't think it would be too hard for the developers to just implement it as a feature, a full video player integrated into the program like what Sibelius has. I think it's just a matter of priorities, what they think is most important.

In reply to by KoRnKloWn

Getting the time from the score is not enough. Since you will want to move in the video and the score should move along too.
The first feature of Jack is indeed to pass audio or MIDI for one software to another but Jack Transport is exactly what you are looking for. A way to synchronize the time between several applications.
Jack transport is used for example between Ardour (a free digital audio workstation) and Blender (a free 3D animation software) to add music to the video. See for example
On the left Ardour, on the right Blender, and Jack doing the communication. If you move the cursor in Blender, it moves it in Ardour and the other way around. More info in this discussion

Enabling Jack transport features in MuseScore would make MuseScore compatible with ardour, Blender, XJadeo for the purpose of film scoring but for other purpose too !

Currently, Jack support in MuseScore should enable you to start the playback of a video and of the score at the same time (but no to stay synchronized)

Of course enabling jack transport in MuseScore doesn't mean a simple audio/video player embedded in MuseScore doesn't make sense.

In reply to by Nicolas

OK, I actually didn't know Jack had that function, thanks for the info, I'm sure that will come in handy! But about moving the score and video along, that is what I've been talking about, if they created plugin objects that could get a call from the main MuseScore play button being pressed, we could use that to start the video at the same time. But now that I see what you mean about Jack, I do think they could work pretty well with the transport feature. In fact we need several new plugin objects, you are actually extremely limited to what kind of plugins can even be made, all you can really do is alter the notes themselves, and text, but when it comes to lines, or dynamic markings like pianizzimo and forte, we are out of luck. I was thinking of trying to write a plugin to realize the dynamics of crescendos and diminuendos, but there's currently no way of having a plugin find the markings for them. It kinda sucks, they need to give us more freedom for plugin development.

In reply to by KoRnKloWn

I wrote half of the plugin framework :) and it's not easy at all to give more access to score element in the plugin framework. So it's done little by little and with precaution. In particular we don't want the plugins to slow MuseScore. As you may know, Musescore is open source software, you can also contribute to the plugin interface development if you want.

In reply to by Nicolas

I see, I really wish I could contribute, I've already looked through the source and it's pretty much over my head, I'm a really novice programmer, I mostly know Java, and I've never taken classes (except for HTML/CSS). Hopefully I will learn enough to contribute, because I would love to contribute to several projects (especially MuseScore and LMMS).

In reply to by KoRnKloWn

Hi, just to mention I've been using jack transport to synchronize Ms , xjadeo and Ardour all together for some time, it works quite well. My issue is that when I open certain scores, Ms disconnects from jack (using it just for audio, no transport). So I have to go a lot to I/O tab to click ok and restore the connection. It's annoying. Anyone had that problem?

In reply to by Nicolas

I made a small video-tutorial about it, using Musescore, two synthesizers, xjadeo and Ardour. Must say Jack-Transport only does start-stop with Musescore, but I think the idea is there. It's also useful to record audio from MS with sounds not available inside MS (synthesizers). It will be ready tomorrow, more or less.
I forgot, the setup is on the video. Basically, TangoStudio (awesome music-distro) and MS thru jack.

Hi, I did a small video to show how to connect MS thru Jack to Xjadeo-video window, also to synthesizers and Ardour. As lasconic commented, Musescore only supports start-stop jack transport for the moment. It would be awesome to have full capabilities, like Ardour, Rosegarden, and so. The video is in spanish, but easy to follow. Feel free to comment,


In reply to by pdro74

Totally awesome ! Thanks for this video! For those who can't speak spanish. The setup is as follow. Jack is the central part, it's a router for Audio, Midi and also a synchronizer for the transport (play/pause at the same time for every client for example).

MuseScore acts as a client for MIDI and Audio. The score is rendered to MIDI and sent via Jack to 2 instances of Yoshimi (a software synthesizer for linux only) to play with organ and strings. MuseScore's internal synthesizer generates also audio as usual but sends it to Jack. Yoshimi output is also sent to Jack again. The 3 audio streams (Organ & Strings from yoshimi, Piano from MuseScore) are redirected by Jack to Ardour (a DAW for Linux and MacOSX). Ardour is recording all these streams.
At the same time, a video is playing in Xjadeo. The playback of the video is started at the same time than the playback of MuseScore. As stated in a comment, pausing MuseScore would pause the video but seeking in the score would not seek in the video since the support of Jack Transport in MuseScore is incomplete. Any volunteer :) ?

(* btw, all the software is free as in free speech)

Do you still have an unanswered question? Please log in first to post your question.