Musescore 3.0.4 Accessibility drawback

Hmm. I take it you are referring to Windows? Is it Windows 10? Other users have reported success, so I'm not sure what is going on here. Anything unusual about your configuration?

I can reproduce this with NVDA 2018.1 on Windows 7 64-Bit and MuseScore 3.0.4 64-Bit too. You have to press the ALT key twice to get the information about the current selected note via speech and braille. But moving around the score with ArrowLeft and ArrowRight has no effects on NVDA's side. Maybe the 32-Bit version of MuseScore works with NVDA, but I haven't tested this.

For me, it does read, but a bit sporadically. Some things read right away, but others read only if I touch the mouse (don't have to actually click anything, just drag my finger along the touchpad after doing whatever). Can you verify this? If that's the issue, I should be able to investigate this myself.

Marc, most blind people find it difficult to use a mouse or mouse pad. Try it with the keyboard like they are describing.

I realize that. I just wanted to make sure we are experiencing the same problem. If they report no sound whatsoever, no matter what they do, even if they do what I said (use cursor key to navigate to next note then just drag finger across touchpad), then they must be experiencing a more fundamental compatibility problem than I am, so I can't help. But if they report they do in fact hear some thing but not others, and in particular they do hear the feedback after touching the mouse, then there is a possibility I can investigate and fix that much.

PR: https://github.com/musescore/MuseScore/pull/4780

This was caused by the upgrade to Qt 5.12. I was aware of some regressions but it appear to be more widespread than I thought. I have submitted a PR reverting to 5.9 until we work out whether this is a Qt problem or if there is something we need to do differently in Qt 5.12.

NVDA users should continue using MuseScore 3.0.3 (available here) until the fix is released.

Are the Qt folks aware of that bug? Is it https://bugreports.qt.io/browse/QTBUG-69472? And https://github.com/nvaccess/nvda/issues/8604?

@shoogle: The 32- and 64-Bit version of MuseScore 3.0.3 are already using QT 5.12.1. MuseScore 3.0.2 (64-Bit), where this issue doesn't exist, uses QT 5.9.7. (Is it correct that there isn't a 32-Bit version of MuseScore 3.0.2? I couldn't find it.)
See also: https://github.com/nvaccess/nvda/issues/8604

MuseScore 3.0.2 is the last one using Qt 5.9:

• Win 64-bit: https://github.com/musescore/MuseScore/releases/download/v3.0.2/MuseSco…

MuseScore 3.0.1 (unofficial):

• Win 32-bit: https://ftp.osuosl.org/pub/musescore/releases/MuseScore-3.0.1/unsupport…

@hhpmusic: PR #4780 wasn't merged into MuseScore 3.0.5. Therefore QT 5.12.1 (or so) is still using. (See "About QT" in the help menu for more details.) Please use MuseScore 3.0.2 until this issue has solved.
See: https://github.com/musescore/MuseScore/pull/4780

What is the workaround?

@frfancha: Downgrade to MuseScore 3.0.2. See: https://github.com/musescore/MuseScore/releases/tag/v3.0.2

or use the "artifacts" (basically development builds) from https://ci.appveyor.com/project/shoogle/musescore/builds/22918675/job/e… (32bit) or https://ci.appveyor.com/project/shoogle/musescore/builds/22918675/job/p… (64bit)

@Sooom sorry, but using a previous version can't be qualified as workaround.
Otherwise ALL regressions would automatically have a workaround !

indeed, all regressions do have a workaround

@frfancha: Depending of the environment in which MuseScore is used, installing a test build isn't recommended. In that case, the only "workaround" is the downgrade option, if this is possible based on all the other fixed bugs in newer versions. Sadly, the downgrade option isn't always possible for all users. TL;DR: Install a test build on your own risk – and don't forget to fully backup your system first, if MuseScore is a critical application for you.

There are potentially multiple issues here.

The first issue is that Qt dropped support for a legacy accessibility framework called MSAA in favour of a newer one called UI Automation. This shouldn't matter because UI Automation can do everything that MSAA can (in fact it can do a lot more!), and NVDA supports both frameworks. As long as Qt implements UI Automation correctly their job is done and it is up to NVDA to use that information properly. The Qt developers reckon that that they implement UI Automation correctly for the most part, so the rest is up to NVDA.

The second issue follows on from the first. Unfortunately, NVDA does not use the information from UI Automation properly. According to the NVDA developers, this is because various other programs (i.e. non-Qt programs) are not implementing UI Automation correctly, and are actually spamming certain mouse-based text selection events so frequently that it causes NVDA to hang. The only way to prevent NVDA from hanging is to ignore these events entirely (apparently it's not possible to let them through for some programs and not others). However, this shouldn't matter because these are mouse events and people generally do not use NVDA with a mouse. Furthermore, the NVDA developers don't think this can be the cause of the problems with MuseScore because the issue reported here is not mouse-related. Still, the only way to find out for sure is to try building NVDA from source with the events no longer ignored. Even if that does solve the issue it doesn't prove it is NVDA's fault because Qt probably shouldn't be firing mouse events anyway... or maybe they should be and it is MuseScore that is using the wrong Qt function to fire events... or maybe its both! This brings us to the final issue.

Finally, it is possible that MuseScore is using Qt's accessibility features incorrectly. This could be true regardless of whether Qt or NVDA themselves have issues with UI Accessibility. Of course you could argue that, since it worked under Qt 5.9 and MSAA, therefore it can't possibly be MuseScore's fault. However, this ignores the fact that it only ever worked in NVDA,: it never worked under any other screen reader on Windows, nor under any screen reader on Linux or macOS. This makes me think that MuseScore is probably doing something wrong here as well. Unfortunately it is almost impossible work out what we should be doing in the Score View because it is a custom control, and there is a complete lack of developer tutorials for implementing accessibility in custom controls, not to mention the fact that NVDA and Qt could have issues that mask any improvements we make in MuseScore.

If Qt and NVDA can't fix this at their end and/or there are problems at our end that we can't fix properly due to lack of developer accessibility tutorials, there are some hacks that we could try without too much difficulty:

1. Create a plug-in module for NVDA that gives it special handling for MuseScore. For example, the module could tell NVDA to watch for changes to the text written in the status bar and have it read that rather than waiting for UI Automation events.

2. Use Qt's text-to-speech functionality to implement our own screen reader in the score view. This would give the greatest flexibility, enabling us to tell the text-to-speech engine exactly what to say and when to say it, rather than being at the mercy of a screen reader. It would allow us ensure that information like the measure number is not repeated unnecessarily when navigating between notes in the same measure, and to create shortcuts like "Repeat last announcement" or a "Where am I?" button that would cause all information about the current position to be read out. The downsides are that the voice could be very different to NVDA's, and the shortcuts would have to be different. However, It would ensure that shortcuts, and the information that is actually read out, is the same on all platforms. Reading the status bar text would be trivial, and the other aspects would make a good project for GSoC.

@shoogle: As far as I know, it isn't possible to access TTS Engines, which are installed as a NVDA add-on, by other applications than NVDA itself. And furthermore we shouldn't forget the braille output. For controlling the notes in a score, reading them on a braille display without TTS is much faster than listening to the TTS – even with a high speed level. So a speech output directly in MuseScore wouldn't be a good solution.

In the end creating a NVDA-appModule for MuseScore, which is part of the NVDA core, should be a better solution here. NVDA is able to recognize that "MuseScore3.exe" is running in the foreground and then loads its appModule from the "library.zip/appModules" folder.

Using Qt's TTS will enable screen reading for all platforms out of the box, right?

@Anatoly-os, that's right, but we would probably only use it in the score view. We would rely on traditional screen readers for menus and dialogs because they handle those things very well, it's just the score view which is a custom interface that they can't handle.

Screen readers understand what a button or a checkbox is, but they have no idea what a treble clef is. In theory you can "teach" the screen reader to understand the score object model:

• a chord is a list of notes
• a measure is a list of chords
• a staff is a list of measures
• etc.

In this way it is possible to treat the score as a tree structure that the screen reader would understand without changing how it is rendered on screen for sighted users. Unfortunately this is very difficult to do, and with a lack of examples or tutorials most applications (including MuseScore) cheat and just try to "trick" the screen reader into reading whatever is displayed in the status bar.bHowever, screen readers are not designed to be used this way: they are supposed to give an overview of what is going on with a configurable verbosity level, which means they actually need to understand what is going on, not just be given fixed text to say to the user. If we want full control of what gets spoken then the only way is via TTS.

If anyone want's to try Qt's TTS then there is a Hello Speak Example application included with Qt. Simply open Qt Creator, go to the Welcome screen, click "Examples" and search for "speech". It only takes a couple of seconds to build.

@Dr. Sooom, Qt TTS does not use the NVDA speech engine; it uses the operating system's built in speech engine (so it would probably use the Narrator voice on Windows and the VoiceOver voice on macOS, but it wouldn't actually use those screen readers; just the voices). It provides full control over the speed and pitch of the voice, which we could make adjustable in MuseScore's Preferences. Overall the quality and control would be much better than relying on a screen reader.

However, you do make one very good point, which is that TTS will not provide Braille output. We might be able add that later though if we can find a C++ library that allows access to Braille terminals, though it is not clear how well that would work if the user also has a screen reader accessing the Braille terminal at the same time.

@shoogle: Well, and you also have to handle the different braille displays with native braille drivers (USB and Bluetooth, only some braille displays have also or only HID-support) as well. And I'm not talking about the length alone. What about the different buttons on a braille display? And what shall happen on pressing a routing key on a braille display? Furthermore I suppose that the access to the braille display is only possible via one single application at the same time. And in the end you also have to implement Liblouis too, because there are hundreds of braille tables using around the globe.
TL;DR: Extremely quite a lot of additional work, which wouldn't be necessary if you use the screenreader for speech and braille output.

As you were talking about the status bar; I guess it would be easier to fire a command to the screenreader on pressing different keys. So after pressing ArrowLeft or CTRL+ArrowRight, as a simple example, the command to read the current status bar content will be fired to the screenreader and this content would then be represented via speech and braille at the same time, but only if the focus/cursor is located anywhere on the score. If the focus is located at a menu bar entry, at an icon in the symbol lists or anywhere in the mixer, this firing command shouldn't be enforced, because that wouldn't make any sense.

Well, and how many information are displayed in the status bar in MuseScore should be configurable via the MuseScore settings. That should be easy to implement. But the fire mechanism is different between IAccessible2 and UIA. It might be better to ask NV Access directly about this, because I'm not a developer.

Just for clarification: The QT-TTS method is similar to the TTS reading functionality in the Adobe Reader. Both are useful for visual impaired users who needn't a braille display and only need speech output for a few things. But NVDA provides much more, also in PDF files, because when you open a PDF file in Adobe Reader, NVDA will render this in a virtual document, similar to a website in Firefox or Chrome. In this situation you can navigation between headlines, lists, tables, links etc. The TTS reading function in Adobe Reader itself doesn't provide these additional (quick) navigation methods.

TL;DR: The QT-TTS is just an additional solution, but absolutely no replacement for the MSAA/UIA screenreader accessibility. Both interfaces are needed for different user groups.

And to the mouse movements: The mouse tracking can be disabled in NVDA. Furthermore it shouldn't be quite difficult to implement a little delay after the mouse cursor has stopped. So in the end the element below the mouse cursor is only spoken after the mouse movements has stopped, and no longer during the mouse movements. This optional function would be new in NVDA. Shall I open a new NVDA issue for it? Would this help here a little bit, especially regarding the performance?

Came up again: https://musescore.org/en/node/287733

If can be a "read only mode" suitable for screenreaders, that can read the score in user language... But i think that Musescore cant do that without heavy model reworking or a specific funtionality scripted ad-hoc.
I'm wondering if a simple plugin can do that.
Is a matter of: if the cursor is over a beat, the screenreader can read the entire beat without all those actual verbosity, sending to braille display the score in their standard.

This has been fixed in Qt 5.12.4. I've tested it with my own build and now NVDA and Narrator can read the status text again. Unfortunately AppVeyor is still using Qt 5.12.2 so we cannot release the fix just yet.

@Dr. Sooom

> when you open a PDF file in Adobe Reader, NVDA will render this in a virtual document, similar to a website in Firefox or Chrome. In this situation you can navigation between headlines, lists, tables, links etc.

Unfortunately these items don't exist in music so I doubt that we can expose a standard document structure to the screen reader. If we provide navigation between things like tempos and key signatures then it will probably have to be via an internal MuseScore shortcut available to all users rather than something the screen reader provides.

We are planning a hot fix release very soon. We can update Qt to 5.12.4 if it is required to make accessibility features work again.

But see also #291079: "Save online" fails in Windows, when building with Qt 5.12.4
Also AppVeyor is not yet on Qt 5.12.4

AppVeyor is now having Qt 5.12.4 and ,despite be default using 5.12.3, is now forced to use 5.12.4.
Via https://github.com/musescore/MuseScore/pull/5150

https://github.com/musescore/MuseScore/pull/4780 has been closed a while ago.

I guess we can mark this issue fixed then

I can confirm that NVDA at least works with the current nightly build. Current builds also include a number of fixes / improvement I made to the navigation and screenreading I made over the past weeks, although a few others are still pending.

However, I do share the concern about the way we are making the scoreview read the score. My understanding from the code is that we do it by creating an "accessible value change event". I gather these are normally used for things like spin boxes or other UI element that control values. We then ask Qt to update the accessibility info for the ScoreView object based on that event. This triggers NVDA at least to read the updated value, but I gather not all readers do this. Maybe it's because we aren't correctly defining the QAccessibleInterface correctly, I don't know. My limited understanding suggests this approach should work, but apparently it doesn't outside NVDA (I can only verify it doesn't with Narrator).

It's a relatively small number of places we use this particular technique, so replacing them with explicit TTS calls seems viable to me. For simplicity we could leave the current code in place but never actually do the updateAccessibility() calls, just in case we decide to return to the current approach if that someday seems to make sense.

I have agreed with RNIB to spend a few days investigating whether it would be feasible to reimplement the Score View as a subclass of QAbstractItemView so that it would give us accessibility information "for free". If this is not feasible then I will investigate the TTS solution (i.e. a built-in screen reader specific to MuseScore).

That sounds like a good plan. Should we open an issue for further discussion of this as needed? Or use Telegram, forum, etc?

Let me know how I can help.

Thanks! Let's use Telegram for now. If I do end up creating a new post or issue then I will post a link here.

After (and before next days) a short time abroad: Not sure, where's the right place to discuss it (see also: https://musescore.org/en/node/292383), if I'll find time, I would like also to check it with "Orca" (Aware, beside NVDA it should be also available for screen readers for MAC).
At time it seems only work with the menu entries (not more, not less) - maybe I miss something (?).
Is it worth to open a new forum topic/issue entry? Or should be the focus at time to get run accessibility features for NVDA?

The score reading for NVDA works with Qt 5.12.4 and later, that's what we've been discussing, but current nightly builds have been rolled back to 5.12.3 because of an unrelated bug in 5.12.4, which is why accessibility is temporarily broken again in the nightly builds, while we await the official release of 5.12.5 any day now.

Meanwhile, once you get a version of MuseScore built with 5.12.4 or later, then feel free to post your feedback on your experience with it. Feel free to respond to that forum topic you linked, or start a new one. Things should actually be in a pretty good state, just a couple of remaining glitches we are still sorting out.

Automatically closed -- issue fixed for 2 weeks with no activity.