Better Musical User Interfaces

• Apr 15, 2017 - 18:35

I stumbled upon this in link from news.ycombinator.com: http://arthurcarabott.com/mui/ which asks two important questions regarding Musical User Interfaces:

1. If I weren't interacting with a computer, how would I express this musical idea?
2. As I am interacting with a computer, what can it do for me that traditional equipment can't?

It is amazing how so many DAWs and plugins nowadays still try to mimic the "look" of hardware boards in order to appeal to people already familiar with those devices. In particular, they insist on forcing the user to try to adjust levels with a virtual potentiometer knob, which although makes sense in the context of a physical board, is very impractical to adjust with a mouse, labtop touchpad, and especially touchscreen (hurts your fingers after a while). Plus there is so much more opportunity for visual feedback that a computer can provide. As an example, he demos below a very simple limiter slider that show how the entire song's waveform will be adjusted, providing instant holistic feedback when adjusting: http://arthurcarabott.com/mui-dynamic-range/

This got me thinking about how musescore still uses knobs in so many places. Anyway just some food for thought about how there might be better ways to do things in MuseScore, like adjust things like relative volume of instruments or the synthesizer effects rig. In fact most MuseScore users aren't already accustomed to those hardware boards, so there is a less of a need to try to mimic them.

Comments

• Apr 15, 2017 - 22:02

Agreed, ericfontainejazz. That arthurcarabott.com site was good, thanks. Reading that I understood for the first time how to use those damned knobs in the Mixer in MS. In my defence I have not used them much, but I was confounded by the fact that they went backwards compared to how I moved the mouse cursor. That site you linked explained that you input just like a slider, up and down, instead of cirlcing around the knob. Sheesh. How embarrassing...

Reply

• Apr 16, 2017 - 00:58

I was just brainstorming an example of a way of avoiding knobs for mixing relative volumes of each instrument might be something as simple as having an integer count representing​ the number of musicians playing that part. That would be more in line with how a composer/band director would think as opposed to how a DAW/pro audio user would think. (Note: I'm not saying this is the best idea or that it should replace the mixer, but I'm putting this down as maybe a way to think about these things outside of the knob box).

Reply

• Apr 23, 2017 - 13:25

Congratulations: It was more confusing than the knobs. :D

Wouldn't one vertical volume and one horizontal pan be better?
volume-pan-201704231503.png

Sometimes, while we seek to be better, our minds lead us to complex areas. These areas are good for ideas but very bad for practice. That's why it's more complicated in the end.

Make it simple, make it simpler, make it simplest...

Otherwise: we may have to wear 3D glasses for a simple volume / pan setting. >B^)

Reply

• Apr 23, 2017 - 13:54

I don't think that one vertical + one horizontal pan would be better than my combined one, because there is not a lot of space in the mixer to fit both sliders in to begin with, and even if added more vertical space to fit both sliders, some of the vertical range of the volume slider is still lost due to the presence of the panning slider.

Also I find it generally impracticable to put both horizontal sliders and vertical sliders on the same window...I think it will look ugly if try to fit two small sliders on every single instrument of the mixer...would be a little overwhelming. But my combined sliders fit nicely inside of the same amount of space of a knob:

5ae925c0-27de-11e7-97f0-ad9b0fbafa2e.png

Also, I don't think my design is complex...it is fairly simple: just X & Y.

Reply

• Apr 23, 2017 - 15:30

@ericfontainjazz,

You idea is so radically different than I have seen anywhere that it would take a lot of getting used to. It is not at all intuitive. I understand very well how it functions and it make sense. Most people are used to dials or sliders. I would suggest 2 sets of parallel sliders like this rough draft

Vol |--------| Rev |-------|
Pan |----|----| Cho |-------|
L 0 R

It would also be very nice if spinning the mouse wheel did not automatically change the Sound so the drop down acts more like a normal drop down.

Reply

• Apr 23, 2017 - 15:54

I sortof like your idea of two sets of horizontal sliders...It would be significantly better than the knobs, and I like the similar knobs of (vol + pan) and (reverb + chorus) are grouped together.

And good point about reserving mouse wheel for scrolling the mixer UI.

Regarding my idea, it is radically different, and I don't expect musescore to adopt it. But it was something I came up after a bit of brainstorming about a way to do things outside of the box of sliders and knobs, which I think makes sense visually for a 2-D interface. So many people are used to dials and sliders, so I accept the reluctance to try anything radically different.

Reply

• Apr 23, 2017 - 16:37

mike320, it would be very easy to make the changes you desribe. If I recall correctly, you are not a programmer right? In that case I could easily make the changes and submit a PR with a screenshot for review.

Reply

• Apr 23, 2017 - 16:52

I'm not a programmer. I think this would be an improvement over the current system of dials. It would be interesting to see what others say about the new layout. Perhaps is will be adopted in 3.0. I don't know how difficult it would be to turn the volume slider in the synthesizer sideways or something similar to that so it looks more like a real slide bar on a mixer than a graphically drawn text tool.

Reply

• Apr 23, 2017 - 16:51

I saw and used this kind of interface.
I still have it on my computer.
Using it seems like a nightmare :)

I'm tired of pulling the Mouse right and left and up and down while using it.

xy-201704231841.png

It was a well-intentioned proposal for my experience.

If the sliders are separate; this is enough for me: D

Reply

• Apr 25, 2017 - 00:14

My only concern with that design is how intuitive it is to use it. I stated that the triangle was a radically different idea, not that it was a bad idea, so don't get me wrong. I have the same concerns about the learning curve for this design that I did for the triangle design.

The idea behind the design is extremely logical. There is an obvious relationship between volume and pan. The only problem is that people are used to different controls for the two functions. Even in my truck I turn the same dial, but I press it once and it tells me I'm controlling the volume, I press it again and it tells me I'm controlling the balance (pan). Your design is one control with a not so obvious manner of manipulating it.

Perhaps a better interaction with the control would be to click on the volume line to activate it. Click it again and the volume will move to that spot with the option of moving the mouse where to adjust the volume. The pan would work similarly. I thank that would be a more intuitive interface. I know it's similar to your plan, but I think mine is simpler.

The one thing you will have to figure out is what to do when the volume is active and they click on the volume at the pan line, does this make the volume 0 or activate the pan line...The joys of programming.

Reply

• Apr 25, 2017 - 00:53

Sliders were necessary in the past because they make sense for physical hardware, but you are seeing them less and less nowadays. I think if someone who never say a slider before, they would find design #11 more intuitive and more obvious. Now there is one way I can think of to make my design appeal more to people who are only familiar with sliders, and that would be to make it look like sliders by adding the little handles, like:

merged-volume-pan-mockup-2D-slider-single-column-with-handle.png

What you are saying as a better interaction sounds more complicated to me. But as I said mine moves vertically if the vertical drag is greater than the horizontal drag, and moves horizontally if the horizontal drag is greater than the vertical drag. That is fewer steps than what you are saying by first clicking to activate vertical movement and then moving and clicking again. Now what I could do if have these little handles added is that if user had initiated the drag from either of the handles, then it would only drag in the direction for that handle.

Now if used the two sliders and had zero volume, then would have to figure out best way to look like...might just have the vertical handle be drawn ontop horizontal handle, like:

merged-volume-pan-mockup-2D-slider-single-column-with-handle-no-volume.png

In which case dragging from that point would only move vertically since the vertical handle was on top. Actually, you're last point is another reason why I wouldn't want to use and activation/mode mechanism.

Reply

• Apr 25, 2017 - 01:06

Your point about sliders being necessary in the past because of physical hardware does not justify the slider mechanism you are trying to make here. I have actually seen something similar that on a physical device. If your talking about necessity, the only thing necessary is a number that can be changed within certain parameters. One of the boxes like they use for the vertical and horizontal offset would be sufficient in that case.

Reply

• Apr 25, 2017 - 01:24

One disadvantage of two horizontal sliders is that it is going to be hardest to quickly glance at mixer ui to instantly get a feel for the current state of the volume and gain settings for everything at once. This is because the volume and panning slider will look similar. But design #11 on the other hand allows user to quickly glance and human brain can easily understand the state of all instruments gain and volume at once without inspecting each slider and manually differentiating between which is gain and which is panning. Maybe I'll do a mockup to make this clear...

Reply

• Apr 25, 2017 - 01:35

If the horizontal sliders were colored similar to the volume in the synthesizer, the pan would only be lit to the left or right to show how much is directed to the left or right. This would give the visual necessary to show what is going on with the volume and pan.

I don't want to discourage you from thinking. I would very much like to see an innovative control that would let you do this, but I think your idea is too complicated for such a simple result.

Reply

• Apr 25, 2017 - 08:38

Ziya's last comment:

The only benefit of the X, Y jerks is:
Change the setting of the Volume as you wish to adjust the Pan.
Change the setting of the Pan while you want to adjust the Volume.

ignores the fact that I said if the magnitude of the vertical drag is greater than the horizontal drag, then only the gain would change, while if the magnitude of the horizontal drag amount is greater than the vertical drag, then only the panning would change. So Ziya's quip against design #11 is invalid.

Reply

• Nov 7, 2017 - 04:10

I really like this idea instead of having them as completely separate things to adjust, since this design does not give a false sense that volume and pan are completely independent of one another. Technically, it is true that volume and pan have nothing to do with one another, but that's not really how our ears work (since they use relative volumes to as a cue to help judge distances).

Reply

• Apr 25, 2017 - 05:38

I like efj's mockup. I think it's compact and intuitively clear. Others have pointed out above that it's two sliders combined. But the result is more than that when you think about it. The way it positions those sliders produces a result that the other pairs of sliders don't. If you scan your eye vertically down across all the instruments, you can easily see where the concentrations of sound are, in the left-right space. You can't get that from the other slider layouts. That's because the *position* of the volume slider is used -- a factor that is not utilized in, e.g., separate horizontal sliders.

Reply

• Apr 25, 2017 - 08:35

Two horizontal sliders for Vol & Pan is not going to work. It looks like "slider hell", and to me is a devolution. Here is what it looks like using Qt's builtin slider:

merged-volume-pan-mockup-horizontal-sliders-only.png

Now inorder to incorporate mike320 last comment, that would mean someone (not me) would have to implement a custom AWL subclass for horizontal slider to implement custom painting...which is going to be wasted work which I think is better used to implement design #11. But here is a mockup for what I think it would look like:

merged-volume-pan-mockup-horizontal-sliders-only-colored.png

It it still confusing. I don't think changing those colors or changing the handle is going to make that any less confusing. This is actually the type of design stuff I would want to avoid. It communicates worse than the design #11 even though it takes up more horizontal space. The original knobs are actually better than this.

It is at this point that I'm convinced of the superiority of design #11, and am going to go ahead and implement it now that I'm thinking about it. MikeN brings up excellent points...especially when scanning eye vertically down it communicates very well and my brain is able to virtually imagine all of the individual bars as if they were on one combined graph. I think user can "see" exactly where the instrument is in the L-R spectrum and "see" how loud each instrument is. And back to the car control analogy, I think two separate horizontal sliders (one for fan speed and one for fan direction) is worse than a single control you could move left & right to control direction and move up and down to control speed, for example.

Reply

• Apr 25, 2017 - 09:42

1. Why are the reverb and chorus sliders that never work, still there, if they are dysfunctional?

Yes, I know: maybe it works if it connects to another midi device via Jack etc. But there is no function here. It does not work with Soundfonts. Just a few dumb buttons. If we need these nonfunctional things, we should add "delay slider" to them. (Currently: When we import something from the ".mid" file, the settings are not automatically entered into the corresponding buttons (except volume).)

2. Please don't choose "red" if you absolutely want to use color. "Red" color means error or something that needs to be set up.

info: Only information, nothing you can do: The drumset instruments (in the kit) have their own pan settings. Normally we don't need this kind of pan setting for the drumset. But it is necessary for single percussion instruments.

--------

Here is the reason for suggesting a vertical volume and a horizontal pan before (intuitive):
Volume runs up and down.
Pan works right and left.
However, if at least separate controls are available, they may be below or above each other.

PS: My writing style may not be very polite. I can say a little straight.
But all my reaction is just to make things better (or right).

Reply

• Apr 25, 2017 - 10:32

1. Regarding reverb and chorus, they should probably be removed if they don't work. I know personally I have never ever used reverb or chorus. I personally think anything beyond volume & pan is unnecessary for the %99+ of musescore users and even the small percentage who do use them would rather export midi from musescore and then perform advanced audio processing in their favorite DAW.

2. I am not suggesting red. I just picked a common color different from blue. If someone suggests a better color, than please suggest, and maybe make a mockup. Anyway, even if I don't use a colored bar for panning at all (but only for volume), then it still looks busy & confusing:
merged-volume-pan-mockup-horizontal-sliders-only-colored.png

Regarding the drumset, even if the drumset kit has their own pan settings, the user should still be allowed to perform a master pan in addition to each individual drum's pan, for instance by putting all the drums on the left channel. I would presume that is what changing pan knob already does for drumsets, but if this is not the case then I would consider that something that needs to be fixed.

Regarding suggesting a vertical volume and horizontal pan, that is something that makes intuitive sense, but unfortunately that is not something that will work within the context of the mixer .ui, since if they are placed ontop of each other then there would not be enough vertical range control for volume, while if they were place side by side then the mixer ui would take up too much horinzontal space:

merged-volume-pan-mockup-horizontal-and-vertical.png

And if the pan was instead place above the reverb & chorus sliders, then it would still be too compact. And either way as horizontal sliders or vertical sliders, it will nevertheless still appear as what I'm calling "slider hell" once you start dealing with large numbers of instruments.

At this point I'm getting tired of making mockups and discussing this, cause it is taking time away from me implementing what is clearly the superior solution clearly to me.

Reply

• Apr 25, 2017 - 19:21

FWIW, reverb and chrous *do* work for MIDI export and/or Jack output, something like that. That's my understanding anyhow.

Also, for Pan at least, the one nice thing about the "knob" is the fact it makes it more visually clear if you're offcenter. That's harder to judge with the slider, unless we incorporate a center detent.

My only real request: make sure whatever we implement is totally keyboard-accessible.

Reply

• Apr 25, 2017 - 20:31

Thanks for the clarification, Marc.

Is there some way mixer.ui could clearly communicate that fact? Maybe could reverb and chorus be hidden under something that says something like "for midi export only"? Or maybe can musescore synth implement the most rudimentary reverb & chorus? I find any of those much preferable to having extra knobs that don't seem to work.

Considering that reverb & chorus will not be used %99+ of the time, I say their control should be hidden under the [+] tab so they don't clutter mixer.ui.

Reply

• Apr 27, 2017 - 08:04

I've been prototyping a "Minimalist hierarchical mixer with tabs" design which displays only mixer info for *Parts* at the top level, each Part in a single line:

mixer-expanded-out-tab.png

But then the expansion button will bring up a QTabLayout for switching between different channels of that part (with tabs named like: Violin-Arco, Violin-Pizz, Violin-Trem). I group the confusing "Rev" and "Chorus" inside of a QGroupBox labeled Midi Output so that it is clearer that that "Rev" and "Cho" affect the midi output. Each channel has offset volume to inc or dec the channel's volume relative to that Part's baseline volume (which I think is much more useful than having separate independent volume controls for each channel of a part, since most of the time you want to adjust volume for all channels of a part together). To the right of hte tab, I have "Mute Voices" buttons inside a group box for clarity...they operate on all channels a part simultaneously (which is what I believe is the desired operation). And last you see the panning control. Note I might implement that knob differently, but I'm somewhat content with the panning knob now that I've kept it hidden away underneath the expansion button (but I might figure out a way to incorporate my combined panning & volume idea somehow). What I like about the horizontal volume sliders here is that it makes this .UI very compact if none of the expanded buttons are pressed.

Anyway, I double posted this idea at #100081: Hierarchy for instrument and channel change in Mixer, so feel free to comment...I don't know where is the best place to comment though...

Reply

• Apr 27, 2017 - 14:29

The nice thing about adding the "L" and "R" is that it makes it very clear that the rightmost slider is for pan. (Of course would have to implement custom slider and not reuse Qt's builtin to get the L and R like that, and to get the slider to start from the middle).

I also had considered having both sliders, but I was worrying it would get too wide to be dockable...but it might make sense to not have the pan slider when docked.

Note: the Rev & Cho knobs would have to start from bottom left and grow clockwise. It is the "offset volume" which needs to represent +/- displacement from 12-o'clock.

Reply

• Apr 27, 2017 - 16:19

It is a good idea to limit the names of the instruments on the left to a certain number of characters.

This is the some of longest preset names in Fluid SF :
Bright Yamaha Grand -20
Coupled Harpsichord - 20
Fifth Sawtooth Wave -20
Mellow Yamaha Grand -20
Yamaha Grand Piano -19
 

The maximum number of characters allowed by SF2 is 20:

http://freepats.zenvoid.org/sf2/sfspec24.pdf page:21

struct sfPresetHeader 
{ 
              CHAR              achPresetName[20];              
              WORD              wPreset;              
              WORD              wBank;              
              WORD              wPresetBagNdx;              
              DWORD              dwLibrary;              
              DWORD              dwGenre;              
              DWORD              dwMorphology;              
};

Reply

• Apr 28, 2017 - 01:35

If indeed there are only 20 chars in soundfont name, then I guess instrument combo doesn't need to be any wider than 20 max-width chars.

Regarding sounfont instrument names, my memory is triggered by #63666: Improve mixer patch selection which is something I might consider, I'm linking here to remind myself or anyone else who might be interested in that.

I'm trying to evaluate whether to use implement custom slider or QSlider ...built in Qt slider has advantages because for one thing it already accepts keyboard focus and other things I wouldn't need to worry about, so I'll probably start with QSlider.

Still regarding panning, I don't really like having panning slider on the top level...it takes away horizontal space from the volume and makes it harder to fit as a dockable widget in left or right dock. My idea for the top level is to only have for things that would be used more commonly, while rest can be hidden. Panning is one of those things that if you are going to change it, you are only probably going to do it maximum of once per score. But muting, soloing, and adjusting relative strength of one part (to hear the part your are working on better) is the type of thing that I believe users would be doing much more often than panning while composing (I can only speak for my own usage with certainty though).

Also I can't figure out how to center the QSlider about 0 such that the blue fill extends to the left or right of the center of the slider without doing a custom paint implementation... Anyway, as marc pointed out, one of the nice things about the panning knob is that it is centered about 0 at 12 o'clock, with the blue coming out from that 0 point. So I'm going to leave panning knob as is for now...

I'm also considering merging the Part name and the Expansion button into one flat button, labeled with the part name, that when you press will bring up the expansion. That saves a little bit of horizontal space, again making it easier to dock.

I'm not sure I want reverb and chorus control so big...since it isn't actually used for musescore's synth, and probably isn't used except for those pro users who send musescore's midi out to some advance synth that has reverb.

Anyway, a lot of little tweaks I don't necessarily want to talk about on the forum because they take too long to discuss and I'd much rather just try to get something out instead of putting up every little thing to a vote. But I think everyone is ok with the big picture view of keeping top level for controlling entire part while moving the less-frequency accessed controls inside the expansion with channel-specific controls inside the QTabWidget.

Reply

• Apr 28, 2017 - 06:35

I've had an idea of hiding both a panning slider and the voice mute buttons to the right of the volume slider, but they are hidden away if the width of the window is shorter than some threshold, and then the panning slider appears once the windows width is greater than some larger threshold, and then finally the voice mute buttons only appear once the width is greater than some even larger threshold. This is nice, because I still get to hide them away while in a compact view so they don't take up limited horizontal space from the volume slider, to help out with dockability. I'll put the part-specific controls all on a new file "mixerStripPart.ui", (it could even possibly be reused by the "Timeline" to provide the basic mixer controls, which was an idea I remember lasconic was considering.)

Then "mixer.ui" will just container a header for the topmost-level to save precious horizontal space for "mixerStripPart.ui", and will contain the labels spelled-out completely for clarity:

Part Name     Solo/Mute    Volume    Panning    Mute Voices

This top-level header will be responsible for the logic to show or hide panning & mute voices based on its window width.

Then the expansion button will only be for channel-specific info only.

Reply

• Apr 28, 2017 - 09:55

lasonic has a touchscreen windows machine, and said via IRC that the PlayPanel's QSlider as well as the awl::Slider both work with touch. I'll probably use the Awl::VolSlider, maybe with some modifications, in that case. He found the nice thing about the PlayPanel's tempo slider is that there is feedback outside the area being touched. So in that case, I'm going to add a QDoubleSpinBox containing the current value of the volume ("0" to "100" or maybe should use dB since is volume) as well as for the Panning ("-100" (L) to "+100" (R) that way user gets feedback if touching, and also so that keyboard-only and blind users can type values if desired. Althogh to allow for dockability and compactness, if the window size is too small, then I'm going to hide these extra spin boxes if window size is less than a certain threshold.

Reply

• Dec 18, 2017 - 22:28

Before my anger is over ;-) - for me it would be very useful (respectively more than useful) to have a "text field" inside the mixer to indicate exactly the value of a volume (similar to indicate the same color for different elements or to indicate the position of elements with vertical or horizontal offset).

Reply

• May 28, 2019 - 01:43

The OP made an important point about not copying hardware designs in software because the interface may not be optimised for use with a keyboard and mouse. A wider point is that it is not good to start with a visual design and then work backwards to make it usable: you should design for usability first and only try to make it look pretty afterwards! If your design process starts with a picture then you will never see the alternatives...

Anyway, see https://musescore.org/en/node/289790 for a proposed new Mixer design based primarily on accessibility requirements.

Reply

Do you still have an unanswered question? Please log in first to post your question.