Mixer: Designing a more compact and accessible UI

• May 28, 2019 - 01:24

MuseScore 3's Mixer looks much better than it did in MuseScore 2, and the side-by-side volume sliders are particularly useful when you have a second monitor on which to display them. However, if you only have one monitor then the Mixer's relatively large dialog gets in the way of the score and other parts of MuseScore's interface.

Size is not the only problem. There are various usability issues, and the UI is almost completely inaccessible to the keyboard users (e.g. people who are blind or motor impaired, or those who simply prefer using the keyboard over the mouse).

Current Mixer

Mixer_expanded_1x.png

The relationship between the top and bottom half of the dialog is not immediately obvious. Once it is understood, you realise that volume and pan controls for the selected instrumental part (Oboe II) are provided twice, which is inefficient. (It's actually three times if you include the number spinners, but those are admittedly quite useful.)

Proposed new Mixer

Mixer_compact_1x.png

Users are encouraged to first select an instrument from the list, and then to adjust controls for that instrument alone. This saves space and avoids duplication. Master Gain control has been removed; overall volume can be set in the Play Panel.

How I came up with the proposed design

I seem to remember one of @ericfont's many proposed Mixer redesigns included a list (possibly as a dropdown combo box) to select the instrument, but I can't find the post in question. Anyway, it is actually possible to arrive at the new design simply by considering accessibility requirements. If you are not blind (I am not) then try to imagine that you are, and that you rely on a screen reader program to tell you what is happening on the screen...

As a blind user, imagine that you open the Mixer and the screen reader tells you that the “mute button” currently has focus. This is no good! Which instrument is it for? The first thing a blind user must be presented with is an instrument, and a quick way to choose a different one if necessary. This means a list of instruments. Once the blind user has selected an instrument they should be presented with controls for that instrument and no others.

Aside: If you think about it, an instrument list is the only way to arrange the Mixer that makes sense for keyboard users. If controls for all instruments were visible at once then keyboard users would have to press the Tab key many times to navigate between them. However, if controls are only displayed for one instrument at a time then keyboard users don't have to press the Tab key nearly as often.

Visual benefits

The proposed design obviously takes up a lot less screen space, but there are other benefits:

  • The instrument list scrolls vertically instead of horizontally (easier to scroll with a mouse)
  • We can see more instruments than we could previously (less scrolling is needed)
  • Instrument names are vertically aligned (easier to spot groupings like “Corno” and “Fagotto”)
  • Names can be longer before they get cut off (e.g. wouldn’t confuse Tenor Trombones 1 and 2)

These visual improvements all happened because of the instrument list, but the list was introduced to help blind users. In other words, the visual problems with the current design are simply byproducts of the fact that it is inaccessible to blind users! If you fix accessibility problems for blind users then the design becomes more usable for everyone.

Further improvements to the design

We can take the accessibility analysis a step further and consider the fact that blind users might want quick access to frequently used controls without having to leave the list. We could introduce new list columns with checkboxes for mute and solo, and of course this benefits sighted users too by making those options visible regardless of which instrument is selected.

Navigating the list with the keyboard becomes tiring if there are lots of instruments, and blind users need to remember the order of every instrument in the list (otherwise they might try to go up the list to get to the instrument they want when it is quicker to go down). We can help blind users by grouping related instruments together in a tree structure, with categories like woodwind, brass, strings, etc. Naturally, this grouping is also beneficial to sighted users.

Aside: A tree structure also enables settings to be adjusted for categories as well as for the instruments. Now users can adjust settings for all brass instruments at once, or expand the tree to adjust only the horns, expand the horns to adjust Horn II, expand again to adjust Horn II in C, and so on to whatever level of detail is required!

So far we have only looked at the list. We could apply the accessibility analysis to the other controls in the Mixer, and eventually everything would be presented in the way that makes most sense to blind users as they Tab over all the controls in the dialog. The optimum arrangement for blind users puts frequently used controls closest to the list, and this naturally results in the shortest cursor travel distance for mouse users. It also means controls that are needed less often would appear on the right of the dialog, so this part of the dialog could be hidden until an Advanced button is pressed, thereby saving even more space. As it stands, the current design is not far off optimum anyway, so the remaining adjustments won't alter the layout significantly.

Restoring the sliders

Having made all possible accommodations for blind users (all of which benefited sighted users too), it is finally time to consider whether there are any enhancements that could be made specifically for sighted users. The only people who will be unhappy with the proposed design are users who own a second monitor. These users will miss the side-by-side volume sliders, but we can simply reintroduce the sliders as an optional feature via a button to expand the Mixer downwards.

Viewing the designs in context

The attachments below show the two Mixer designs with MuseScore's interface in the background to provide context.

The screenshot of the current Mixer was taken in MuseScore 3.0.5 running on a 2015 MacBook Pro (background has been lightened and blurred). The image of the proposed Mixer is a mock-up at the same scale.

Attachment Size
Current_Mixer_in_context.png 244.05 KB
Proposed_Mixer_in_context.png 289.67 KB

Comments

I basically like the design you are proposing, there is one thing that is not addressed at all. There is no apparent way to view and adjust the different channels (like mute and open on a trumpet).

In reply to by mike320

Actually there is. I talk about the list becoming a tree structure where instruments are grouped by family (woodwind, brass, strings, etc.). It is equally possible to expand instruments into sub-instruments for different channels.

If you look closely at the list you will see a little triangle next to each instrument name. If you clicked that triangle then it would expand to show the sub-instruments, just like the template categories in the New Score Wizard.

In reply to by shoogle

I see, that wasn't clear in your description. Like I said, I really like the design better than the bulky redundant mixer we currently have.

A couple more things I would like to have clarified. Where does the category list come from, the instruments.xml file? That would be fine, but there are issues to consider. Currently the instruments are listed in the same order as in the score. There are people who do not follow standard orders for instruments and may mix woodwinds and brass for example, how will this correlation be handled, especially when it comes to categories of instruments. Also in the case of percussion, it's standard to have all unpitched percussion listed after the timani (a pitched percussion) but before all other pitched percussion (like marimbas and tubular bells).

In reply to by mike320

You're right, it wasn't clear. I guess I got caught up in all the new benefits of the tree structure and forgot that its most important job would be to replicate the channel control that that current design already provides. :)

The instrument categories would indeed be defined in instruments.xml according to the standard orchestral ordering, but I have plans to overhaul the Edit Instruments dialog to enable users to pick other groupings too (e.g. wind band) or even to define custom groupings of their own. This would take some time to implement, but there is no need to enable groupings in the Mixer until it is finished.

In reply to by shoogle

Since I have no insider knowledge, I can contribute little. As for the grouping of instruments ('or even to define custom groupings'), I would like to point out (for the purpose of brainstorming) that existing brackets already represent a grouping.

Perhaps this becomes clearer with the example of polyphonic music. See e.g. Wikipedia 'Venetian polychoral style'. A work here consists of several choruses, each consisting of voices and/or instruments, which are grouped in the score with brackets. An example can be found in the appendix. The mixer would serve here to determine the position of the choruses in the room (e.g., chorus 1 to the left, chorus 2 to the right, etc.).
If you can do this per group, it is certainly easier than per single voice.

Attachment Size
Schütz_Nun_lob41.pdf 71.09 KB

In reply to by Nikolaus Hold

You are quite right. Brackets do indeed represent a grouping, and one of the objectives of my planned overhaul of the Edit Instruments dialog is to enable groups to be defined there, and enable brackets to be defined for those groups. However, that is a discussion for another thread. As far as the Mixer is concerned, all that matters is that these groups are defined somewhere.

In reply to by BSG

The order would be the same either way. I don't really want to discuss the Edit Instruments dialog in this thread, but basically groups created in that dialog would be bracketed together in the score, but equally, brackets added in the score would also create groups in the Edit Instrument dialog. The two would be linked, and instruments would be displayed in the same order (and with the same groupings) in the Mixer. This is, as you say, the only correct order.

In reply to by Nikolaus Hold

@Nikolaus, let's say you were able to set the positions as you wanted (chorus 1 to the left, chorus 2 to the right). We could represent this as positions of the Pan sliders between -1 (100% Left) and +1 (100% Right):

  • Chorus 1: -0.5 (50% Left)
  • Chorus 2: +0.5 (50% Right)

Having made that adjustment, where would you expect the Pan sliders for voice parts within each chorus to end up? Should they stay at zero (the default) or should they all have moved to match the parent group?

To put it a different way, how would you then set Chorus 1 Sopranos to be slightly more to the left than Chorus 1 Altos?

Would you want to set relative values?

  • Chorus 1: -0.5 (50% Left)
    • Sopranos: -0.05
    • Altos: +0.05

Or would you prefer to set absolute values?

  • Chorus 1: -0.5 (50% Left)
    • Sopranos: -0.55
    • Altos: -0.45

What would you expect to happen if you later changed Chorus 1 Pan back to zero (centered), or -1 (100% Left)?

In reply to by shoogle

Personally, I would be content to imagine myself standing in a large church with one choir to the left and the other to the right. (The reverb would make a more accurate location difficult for me.)

But I understand your questions and would probably ask them myself if I had to implement the thing. So let's hope someone with finer ears and more mixer experience can answer.

In reply to by Startled Bee

Docking to the bottom of the screen makes more sense in my opinion as other areas are already used by the Palettes and Inspector. However, permit me to draw your attention to this sentence in the proposal:

controls that are needed less often would appear on the right of the dialog, so this part of the dialog could be hidden until an Advanced button is pressed.

In reply to by Marc Sabatella

Thank you! What I think is most interesting is not the design itself but the fact that:

  1. It inevitably arises as a result of analysing the Mixer from an accessibility perspective.
  2. The accessibility analysis resulted in the best design for all users, not just blind users.

In other words, the element of subjectivity in good UI design is practically zero.

In reply to by shoogle

Shoogle, you are definitely on the right track. Everyone has known for years that the mixer needs fixed, and the latest "fix" made somethings easier, its size is atrocious. As you are well aware, I am a big fan of making MuseScore as accessible as possible and this will definitely improve that area.

Very nice and efficient design indeed :)
I second the effort!

One suggestion though...

Sometimes it is useful to have as many "console sliders" as possible in front of you, to get an overall picture of the instrument adjustments (plus a master volume slider).

So how about instead of disappearing the sliders all together, just moving them inside the left pane, horizontally aligned. Just put them where the instrument names are? No extra space needed. The names of the instruments could be shown in each slider.

Unfortunately I can't draw a picture of this but you get the idea, right?

In reply to by vasmakk

I get what you mean. I did consider putting actual sliders in the list itself in a new column to the right of the instrument names, but it's not clear how keyboard users could operate them (the arrow keys would be needed to navigate between rows and columns in the list, so they couldn't also be used for the sliders).

If we can't put actual sliders in the list then there might be a way to give a visual indication of volume, perhaps by coloring the region behind each instrument name. For example, if the Flute is on zero volume then the background region behind its name would be entirely white, but as you start to increase the volume a green bar would start to appear on the left hand side, and once you reach 100% volume the entire background region behind the Flute's name would be fully green. You wouldn't be able to adjust volumes in the list itself, but you would be able to see all the relative volumes for comparison purposes, and then quickly click on an Instrument to adjust its volume using the separate volume slider elsewhere in the dialog.

I don't know how customisation Qt allows over the appearence of a listview/treeview, so it may turn out that partial coloring is not possible (or not worth the effort), but even without it the existing proposal would still allow you to compare instruments very quickly by clicking on the first instrument, then pressing the Down arrow repeatedly while keeping an eye on the position of the volume slider. You can then go back and click on instruments in any order to compare their slider positions, and if you want an exact numeric value then you can use the number spinner.

In reply to by shoogle

How about the use of the TAB key to move from column to column.
And inside each column, we could use the arrow keys, to move from slider to slider and adjust the volume of each one of them.

I'm not sure though, if the TAB key is reserved by some OS, for other purposes...

In reply to by vasmakk

> "How about the use of the TAB key to move from column to column."

Part of the benefit of having the list structure is that it counts as one big widget, so you can navigate it with the arrow keys instead of Tab. This means that Tab is still available to navigate to other widgets, and the total number of keypresses required to reach any widget is reduced.

In a "flat" design, where every control is a separate widget, you only have one degree of freedom:

  1. Press Tab or Shift+Tab to cycle through widgets

This means you have to cycle through all Flute controls (mute, solo, volume, etc.) before you get to the controls for the next instrument.

A list adds a second "dimension", giving two degrees of freedom:

  1. Press Up or Down to change instrument in the list
  2. Press Tab or Shift+Tab to change widget (mute, solo, volume, etc.) for that instrument

Now you can get from the Flute to the next instrument without tabbing through all of the Flute controls.

A table or grid adds a third degree of freedom:

  1. Press Up or Down to change instrument (i.e. change row in the grid)
  2. Press Left or Right to change column and access other controls in the grid (e.g. mute and solo)
  3. Press Tab or Shift+Tab to change widget and access controls not in the grid (e.g. volume sliders)

Now you can access mute and solo without leaving the grid, and you can access volume sliders without tabbing through mute and solo, so the number of keypresses required to access any control is reduced.

If we used Tab to navigate within the grid then we lose a degree of freedom, and more keypresses are required to reach any given control. Equally, if we put all controls in the grid then you can no longer access some controls with Tab and Shift+Tab, so you lose a degree of freedom that way too. The optimum arrangment puts some controls in the grid and some outside the grid.

In reply to by shoogle

QTreeWidget has a method setItemWidget(). This can be used to add a widget to the the row so you don't just get text. So you could definitely create a custom widget that drew a bar indicating the volume level using this approach.

If the additional vertical space were tolerable, it's also possible to add a slider to each item in the tree view. I did a proof of concept on this by modifying the existing mixer code which allowed me to generate the following screenshot.

Screenshot 2019-06-01 at 12.28.15.png

In reply to by Startled Bee

Looks good! I would definitely use a separate column for Volume. However two questions remain for keyboard users:

  1. How to expand and contract branches of the tree?
  2. How to adjust the volume sliders?

By default, Qt assigns the Up and Down arrow keys to change rows in the grid (i.e. select an instrument) and Right and Left to expand or contract branches of the tree. Qt does not provide a default way to change column via the keyboard alone.

I recommend keeping Up and Down for changing row, and implementing Left and Right to change column. As for the two questions:

  1. When in the first column, use the Spacebar (or Enter/Return) to toggle a branch as expanded or contracted.
  2. When in the Volume column, use the Spacebar (or Enter/Return) to enable editing, then use the arrow keys to adjust the sliders, then use the Spacebar again (or Enter/Return) to confirm the edit and return focus to the grid.

Alternatively (or in addition) you could use Shift+Right and Shift+Left to expand and contract branches and to adjust sliders.

There is a current limitation that I would like to see addressed in a new mixer. Currently, if you change something in the iteration of the instrument that's visible before you click the arrow button (I'll call this the "master" instrument), then only the first sub-instrument after it is changed when you click the arrow and expand the rest of the instruments.

In the case of instruments like clarinet and horn, it is common that the only change is the pitch of the instrument and the sound from the font does not change. If you change the volume or even the sound in the font, you must go through each instrument change rather than being able to change the "master" instrument and it applying to all of it's sub-instruments. You would still have the freedom to adjust the pizz or tremolo channels on strings and any other sub-instruments you need to adjust. I'm currently working on a score with 6 clarinet changes on one staff and it was a long process to change the soundfont for that clarinet.

In reply to by mike320

@mike320, that is a very good point. It can be partially addressed through time-saving features:

  1. Presumably some of your 6 clarinet changes were actually changes back to a previous clarinet (e.g. Clarinet in A --> Clarinet in C --> Clarinet in A again). You don't really need independent control for all 6 changes, just one setting for each unique instrument. This is addressed in #42301: Instrument Changes: add ability to change back to a previous instrument.

  2. Back in the Mixer, it could be made possible to select multiple instruments at once (e.g. by holding the Ctrl or Shift key). If you selected both the Clarinet in A and the Clarinet in C then you could adjust volume and patch sounds for both instruments at the same time.

With these time-saving features in place, the issue of how sub-instruments should "inherit" settings from their parent becomes less important. In fact, we might decide it is best to disable settings for parent instruments and force users to expand the tree and adjust sub-instrument settings instead. This makes parents (and groups) a purely organisational feature, which is a bit of a shame but at least it avoids confusion.

In reply to by shoogle

If you do want some form of inheritance, then there are a few ways I could see it working. Let's say you have a parent Violin that expands to show sub-instruments for Arco and Pizzicato.

  • The parent Violin's volume is at some arbitrary value
    • The Arco sub-instrument is at exactly 60% volume
    • The Pizzicato sub-instrument is at exactly 70% volume

Now you try to adjust the parent Violin's volume to 50%. What should happen to the sub-instruments?

Case 0: Nothing

It is not possible to adjust the volume for parent instruments (i.e. the Violin has no volume slider, or it is disabled).

Case 1: Overwrite

All sub-instruments are now at 50% volume. Relative volume adjustments for sub-instruments are lost.

Case 2: Scale on playback

Arco and Pizzicato are still displayed at 60% and 70% volume in the Mixer, but playback is at half of those values because the parent instrument is set to 50% volume.

Case 3: Scale in Mixer (upper limit)

Assuming the Violin was initally at 100% volume, Arco is now explicitly displayed at 30% volume in the Mixer, and Pizzicato at 35% volume. Playback is at the displayed volumes.

The sub-instrument sliders cannot now be moved above the 50% limit set by the parent instrument.

Case 4: Scale in Mixer (enforced mean)

The parent volume is always the mean average of all of its children. The Violin must have initially been at a volume of 65% (mean of 60 and 70). This has now been reduced to 50%, so the volumes of the Arco and Pizzicato sub-instruments are reduced in proportion to give 46% and 54% respectively.

Note that the 60:70 ratio cannot be maintained if the Violin volume is allowed to go above 93%.

In reply to by BSG

It says under the image:

"Master Gain control has been removed; overall volume can be set in the Play Panel."

But the image is only a "compact view". The proposal later says:

"users will miss the side-by-side volume sliders, but we can simply reintroduce the sliders as an optional feature via a button to expand the Mixer downwards."

Naturally this could include the Master Gain control.

@shoogle I recently opened an issue regarding some improvements to the layout of the mixer and another user has brought to my attention your post here; see #290458: Improvements to the mixer GUI.

If your design would be implemented, I think that some of the suggestions I have made are still very relevant, particularly using QT push buttons instead of those svg files, displaying the voice mute buttons only as either depressed or not using a default grey colour (instead of using voice colour), etc.

But I must say I am not too convinced about this proposed new mixer here. I do get that there are some good things about the design you propose, but there are several advantages of having all faders and knobs form all tracks displayed side by side; it makes it very intuitive to change panning and volume and mute/solo in a sensible way, comparing the overall settings. For instance, say you have 10 instruments, it's very easy to see if their pan is distributed in the way you wish just by glancing a single screen as opposed to having to memorise 10 pan values since you can't see them side by side. Same for muting or soloing, you can see at a quick glance which of your instruments are currently playing or not. My point is that there is a good reason for the mixer layout that virtually every DAW uses; and while I see that your solution is elegant and intuitive in a certain way, I believe that a lot of the advantages are lost.

In reply to by gsagostinho

Getting an overview of volume or pan settings is not actually as difficult as you might think with the proposed design: simply select the first instrument in the list, then press and hold the Down arrow key while looking at the position of the volume or pan slider. (Notice how you would never think to use the keyboard with the current Mixer design - unless, of course, you are blind and have no choice - but with the proposed design the keyboard because the obvious way to interact with it.) Having gotten a rough idea of slider positions, you can then go back and click on individual instruments to look at their settings in more detail.

Of course this doesn't fully replace side-by-side controls, but if you read the section of the proposal titled "restoring the sliders", you will see that my plan was not to remove the DAW layout at all, but simply to hide it until the user clicks an "expand" button. Perhaps it's my fault for not including this button in the image of the proposed design, but that image was only really there to illustrate the improvement made in the very first step of the accessibility analysis. The analysis didn't stop there though, so I encourage you to read the rest of the proposal to find out about the other improvements in the design.

In reality, my design is actually very similar to the existing Mixer. The only real differences are the addition of an instrument list, and swapping the behaviour of the current Mixer's "collapse" button so that it hides the sliders rather than the controls at the top of the dialog. There are many other improvements that I would like to make, but they are relatively minor in comparison to the list and the expand/collapse button.

Other users have suggested putting volume sliders in the instrument list itself to provide side-by-side control in a more compact form. This is something that I had considered but rejected due to the difficulty in controlling grid sliders with the keyboard. However, further experimentation has led me to believe that this is not an insurmountable problem. Naturally, if it can be done with volume then it can also be done with pan.

I address the points about button styles in the other thread. I am more concerned with functionality than style, so I would probably just do whatever is easiest and leave the styling to somebody who cares about that sort of thing. I don't think the existing styles are too bad, but I agree yours are better.

In reply to by shoogle

@shoogle Apologies, I've missed the part about restoring the sliders.

> Naturally, if it can be done with volume then it can also be done with pan.

I think this would be a great idea, see the mock up I created:

mockup.png

Adding panning and solo/muting add just a bit to the size of the panel, but the reverb slider can also shrink a lot. I personally think this looks really good, would keep the current functionality and allow for a ocmplete overview of all tracks, at the same time as have a much improved design. What do you think?

As for the styling, I completely understand your point. But perhaps if it would not be too much trouble, would you consider to at least replace the solo/mute svg with QT buttons? They look so much more natural and should be fairly trivial to add.

There seems to be some enthusiasm for new design. @shoogle are you working on a PR for this? If not, I'd be interested in making a start on implementing the proposed design, i.e. making the quick hack-work I did to generate my screenshots into (a) tidy code and (b) actually functional. I suspect that implementing the whole re-design might involve a few staged steps to get right 😀. Of course, I don't want to duplicate effort if you're already on it.

In reply to by Startled Bee

@StartledBee. Sure, feel free!

In terms of making the code tidy, the best thing you can do is to use a QTreeView instead of a QTreeWidget. Views are bit more tricky to understand at first, but they have huge benefits down the line in terms of code maintenance and reusability. (In fact, it appears that you may not have a choice in this matter as setItemWidget is can't be used for editing, so you need to use QTreeView and QItemDelegate instead.)

The GSoC student I am currently mentoring for the Palette Accessibility project has created a small example Qt project that demonstrates how to create a simple tree structure, firstly as a QTreeWidget, and then as a QTreeView. If you've not used a QTreeView before then playing around with this example will really help you to understand the differences between the two.

> I suspect that implementing the whole re-design might involve a few staged steps to get right

I agree. i'd just try to create a single-column list of instruments in the first stage (like in my original image). I'd leave the columns of checkboxes and sliders until later, after you have got the hang of creating a model for a TreeView.

Ping me in MuseScore's developer channel on Telegram if you need help getting started.

Do you still have an unanswered question? Please log in first to post your question.