Using Musescore with JAWS screen reader
I am blind and use the JAWS screen reader for Windows to access my PC. I recently came across Musescore and was impressed by its feature set and useability.
I know that the documentation says that Musescore works best with the NVDA screen reder, but since I and many other users are JAWS users I thought it would be worth seeing how well Musescore worked with JAWS.
When using NVDA and navigating through a score, NVDA speaks note names, durations, chords, etc. Very nice. Using JAWS, however, these items are not spoken.
My question is how NVDA is accessing this info in order to speak it. I have actually written and contributed JAWS scripts for several other music programs including Band-in-a-Box, Samplitude, and Sonar, and was trying to see if it was possible to put together some JAWS scripts to provide speech and braille feedback on these values as one navigates through a score. Unfortunately, I didn't see that any of this info was visible on the screen, so there was no easy way to capture this info with JAWS.
Is there perhaps a setting in JAWS or Musescore that would enable these two to work together a bit better like NVDA?
If anyone has any suggestions as to what is different between how NVDA and JAWS is capturing info from Musescore or has any suggestions let me know. I would like to not have to keep switching between screen readers to effectively Musescore.
BTW, thanks for the great job the development team did on making this great program accessible!
--Pete
Comments
Welcome Pete. You've probably already found https://musescore.org/en/handbook/3/accessibility where you will find the link to https://www.nvaccess.org/
Wait with a little patience some more comprehensive answers.
I do not know how NVDA speaks note names, durations, chords, etc., but perhaps it gathers info. from the Status Bar. The Status Bar is visible at the very bottom of the score window. (It is toggled via the View menu.)
For example, I currently have a note highlighted and the Status bar shows:
Note; Pitch: D5; Duration: Triplet Eighth; Voice: 1; Measure: 9; Beat: 1.66667; Staff 2 (Alto Saxophone)
So, that means measure 9 of the second staff (played by an alto sax) plays a D5 which is the third note of a triplet since it occurs on the last third of the first beat - 1.66667.
(Intuitively, it also means that the triplet starts on beat 1.)
Navigating further using Alt+arrow keys, I drop down to the staff below and arrive at a slur - which the Status bar interprets:
Slur; Start Measure: 9; Start Beat: 1; End Measure: 10; End Beat: 1; Staff 3 (Tenor Saxophone)
Things like barlines, time signatures, clefs, etc. are also textually described in the Status Bar.
Perhaps Jaws can be made to capture such information.
In reply to I do not know how NVDA… by Jm6stringer
thanks for your reply about the Status bar. That helped locate this info with JAWS.
For some reason, however, JAWS is not seeing this at the bottom of the display. I used a JAWS utility to check the Window Class of the area where this info is displayed, but it, unfortunately, appears not to be unique. In addition, when I set JAWS to speak all changes on the screen (as opposed to just changes in highlighted text which is the default), JAWS is not speaking this note/chord/duration info as it changes on the screen. Very strange. I have never seen this with JAWS.
I'll have to look into this more.
Thanks again.
--Pete
Hello, and thanks for the comments!
We'd love to get MsueScore working with JAWS (and Narrator, and VoiceOver, and Orca) but so far no luck. If you can help that would be fantastic!
It is sort of true that we use the status bar for thew accessibility text, but not totally. The same function generates both, but they are customized differently. At the end of each command that might change the selection, we generate the status text and the accessibility text. We use the Qt framework to handle this and most other UI elements. Specifically, we define the score view to be a QAccessibleWidget, and override the QAccessibleWidget::text with a value of QAccessible::Name to return the accessibility text. Everything seems to be working right but somehow NVDA is the only screenreader that actually reads anything by default. Apparently JAWS will if transfer keyboard focus elsewhere then back? It's frustrating that we haven't been able to figure out the problem, and we've been looking at totally different ways of solving it but I suspect it will turn out to be some simple one-line change that is what we're missing.
In reply to Hello, and thanks for the… by Marc Sabatella
mark,
Thanks for your reply.
Unfortunately I am not familiar with the QT framework and how JAWS can be made to interact with this. Knowing what is going on under the hood in Musescore is a help and at least a starting point.
I'll see what I can find out from other JAWS scripters to see if they can shed any light on this issue and perhaps give us a hint on what can be done to make JAWS work better with Musescore.
--Pete
In reply to mark, Thanks for your reply… by ptorpey
Can you explain a bit more about what kinds of things you normally can access with JAWS scripts?
In reply to Can you explain a bit more… by Marc Sabatella
Mark,
There are a number of ways that JAWS can grab information to report to the user in speech/braille.
Many of the scripts I write depend on JAWS knowing the Window class or name of a programmatic element in the UI such as a window, dialog, button, etc. JAWS has functions for finding these elements based on the class or name or type of the element. After finding the element, JAWS then has functions for describing the element or text associated with the element and providing the user with that info or acting on the info accordingly.
Sometimes the information one is looking for is contained within a single window and interspersed with lots of other text. Then a JAWS script can search for certain text by an attribute such as the foreground/background color of the text. JAWS can also look for a particular string of text.
In my JAWS scripts for Band-in-a-box, for example, I have the scripts:
1. Read through the title bar to find the bar and beat that has focus. This info is displayed in the UI as "bar= n, beat = m".
2. I then have JAWS look for a window with a specific class name that contains the b ar numbers and chords of the score.
3. Within that window I look for the bar number and text with a particular background color indicating the chord that has focus in the particular bar.
4. I then present that info in speech and braille so that the user knows the bar/beat and highlighted chord.
There are also more sophisticated ways in which JAWS scripts can use objects and hook directly into the API of some programs through associated DLL's. For example JAWS hooks directly into MS Word objects to interact with content in Word documents.
JAWS also has some built in tools for scouring the screen to see the class and window names that contain the text on the screen. When I looked to see what programmatic windows were showing in the MuseScore UI JAWS seemed a bit confused and didn't see many unique window class names. Also, the text you described as being in the Status bar (which I assumed was at the bottom of the screen?) appeared in the middle of the display using the JAWS browse mode. Maybe the QT tools create aplications in a different way than I am used to seeing them with JAWS or in which JAWS sees the elements of these aplications. I am trying to learn more a about how MuseScore is put together before I contact some of the developers I know at FS.
Anyway, I hope that is helpful. I will be poking around to see if I can learn more.
Thanks for your interest.
--Pete
In reply to Mark, There are a number of… by ptorpey
This is great information, thanks! Seems like it should be possible to get this working, then. For instance, the info in the status bar is, while not ideal, better than nothing, so if you knew how to find the status bar and read when it changed, that's something right there. Does the JAWS tool you have see the status bar? Should show as a standard status bar type of widget. We don't deal with Windows directly, everything goes through Qt, which is what enables us to be cross platform. So I can tell you Qt object names etc, maybe they relate in some way you can use? Or maybe unstead of reading the status bar, you can access the scoreview widget directly and grab that accessible text we are placing there. One of the names you could look for, if you have a way of finding this, is "Ms::ScoreView".
In reply to Can you explain a bit more… by Marc Sabatella
marc,
From one of the other JAWS scripters, he indicates that a fairly recent update to QT changed from using MSAA under the hood to using UIA. The app he is scripting (I believe it is TeamTalk) uses:
QT 5.12.3
Checking the Properties of the QT5core.dll for MuseScore gives:
5.9.8.0
I believe that MS is moving away from MSAA for accessibility and that most newer programs are using UIA.
Don't know if that helps or what the QT version MuseScore is developed with is using for accessibility.
--Pete
In reply to marc, From one of the other… by ptorpey
We do currently use 5.9.8, although we briefly flirted with using 5.12.5, and I recall that the MSAA to UIA change was one of the reasons we wanted to. 5.12.5 seems to work better for accessibility in some ways but led to problems elsewhere, so we rolled back. If it helps, I'm also happy to try to make sure we provide a 5.12.5-based build for accessibility purposes.
In reply to We do currently use 5.9.8,… by Marc Sabatella
Marc,
One of the JAWS scripters whom I had contacted on another list actually wrote up some quick JAWS scripts for MuseScore. I haven't had a chance to test them yet, but below are some comments he has about trying the new beta version as well as the QT framework. Maybe some of this might be helpful to you.
"Ok, got the non-beta, and a lot more info and even some scripts!
The beta and the non-beta both use Qt versions that support MSAA better than UIA. The beta's MSAA structure is nearly empty though, whereas the stable release has decent accessibility info by comparison.
The reason my beta version would not launch is that closing it with Alt+F4 left the app running as a background process with no windows. Killing this process from the Task Manager did let me relaunch, but the windows were again completely unreadable with JAWS.
All of what follows concerns only the stable (non-beta) MuseScore release, which works well here:
The structure is typical MSAA for this Qt version: An Application object at the top, with all windows and elements descending from it. With this Qt version, UIA should not be needed.
My NVDA, latest stable release, does not say anything on Left and Right, or even Up and Down arrows in a score. Notes do play, however. I therefore don't know for certain what is currently expected by JAWS users that have used NVDA.
Having scripted for a few of these apps, I had some thoughts and code to bear offhand; so, there are now some experimental scripts for the stable MuseScore version at https://www.dlee.org/musescore/
What you'll find at that address is a user guide for the scripts. The guide contains a download link for a professional installer for the scripts themselves. At this writing, I allow the scripts to install into any JAWS version as far back as 2015, though I only tested them under JAWS 2020.
Script features are listed in the guide; but in summary, at present they are
Better announcement of combo box and radio button controls.
Instructions for the user to use Left and Right arrows for radio buttons, because Up and Down arrows do not work in this application for those.
SayLine (Insert+UpArrow) reads the current note in a score. Note though that it only reads the last note added if the current position contains more than one.
The JAWSKey+PgDn command for reading the status line works in this application, though it natively would not.
Key sequence [ 1 reads part 1 of the status line, [ 2 the second, etc. (A key sequence is a set of keys typed one at a time, not together at once.)
"
I still need to understand why JAWS does not natively provide the same feedback on these MSAA objects as NVDA does. Anyway, I'll try to find some time this weekend to test Doug's scripts.
--Pete
In reply to Marc, One of the JAWS… by ptorpey
Thanks for the info, but I find it confusing. Can you ask your contact which versions specifically (Help / About) he is testing? There should be no appreciable difference between the latest stable version (3.3.4) and the current beta (3.4). Both use Qt 5.9.8. So both should work equally well with NVDA, which is to say, extremely well - we have many users using 3.3.4 successfully, and I see no reason it should be any different with 3.4 beta. But I will try to check that for myself soon.
In reply to Thanks for the info, but I… by Marc Sabatella
Marc,
Below is Doug's response. He is definitely interested in following up with scripts for JAWS:
"Unfortunately I deleted the beta before installing the stable version and don't remember the version number, but it's what came down from the website as Beta the day I posted here about it.
Do you know how to handle the forums via RSS? I have an RSS reader but don't know the URLs to use.
"
--Pete
In reply to Marc, Below is Doug's… by ptorpey
I'm not aware of an RSS interface for this forum, but actually a better place to discuss these issues might be the MuseScore Developers Chat on Telegram: https://t.me/musescoreeditorchat. Feel free also to pass along my email: marc@musescore.com. But the chat really is the best place I think, it's faster, more people ca participate, etc.
Anyhow, I assume then the beta was in 3.4, and the last stable version refers to 3.3.4. In which case I remain confused, because these should be virtually identical as far as any of this is concerned. Same version of Qt, same object structure, etc. And I tested it and found it works as expected with NVDA, just as 3.3.4 does.
One thing to be aware of is that occasionally NVDA does lose track of where the focus is and will stop reading the score. Easy fix for that is to press Alt to move focus to the menu bar, then again to return it to the score view.
Also, be sure to read the accessibility documentation: https://musescore.org/en/handbook/3/accessibility. You'll see that left/right aren't the only or main navigation commands. Instead use Alt+Left/Right to make sure you hear note just the note but also the other markings: clefs, dynamics, tempo markings, articulations, etc. Also, check out this demo video I made: https://youtu.be/Ocl9gBxmeKg.
I don't have JAWS so I can't test the scripts myself, but I can check with my student who does and get feedback so she can compare to her experience with NVDA, which she has been using successfully the last several months. We're still on break for another week, but I should be getting together with her soon.
I would love to see the source for scripts themselves, could a link be added to that page? I see there is a command for reading the status bar, if it works to simply switch that out for reading Ms::ScoreView, that would be ideal, as the info in this object is tailored specifically for screenreaders. For instance, instead of reading all information about every note (pitch, duration, measure, beat, voice, staff) it only reads what has changes as you navigate, so if you're reading a bunch of notes in the same measure on the same staff, it skips everything but pitch and duration. Lots of other optimizations, too, plus some places where it reads additional useful information. So if possible, it would be better to read the ScoreView object, which is what NVDA is reading natively. Ideally, we could figure out why JAWS isn't reading it, though, because as far as we know we are doing the right things.
My student tells me that for her, JAWS does read the accessibility info if you switch windows (eg, Alt+Tab to another window then back).
I do hope we can continue this conversation in real time on Telegram!
In reply to Marc, Below is Doug's… by ptorpey
I'm continuing to investigate and explore using Microsoft's "Accessibility Insights" tool, and am learning some useful things. The scoreview widget that contains the accessibility text does exist and I can see the text, but it seems not hooked up to the object tree correctly so you might not be able to find it (it's a few levels deep under the topmost "Split Button", but the parent of the topmost split button doesn't seem to be set back to the main window). Also the name of the object is the name of the score, which is great for the screenreader but probably less great for scripting. I'm looking into giving this control better identifying info, which if nothing else should make it easier to script. But hopefully along the way we'll finally figure out why screenreaders other than NVDA are not reading the changed value for this object even though we are triggered a "value changed" event that should do the trick.
Some initial success!
Armed with clues and new determination as a result of this discussion, I did some more investigation and found a number of small tweaks we can make to improve things for JAWS (which I have now installed, what the heck, a one-year subscription isn't that expensive). I also installed the scripts above, but now I don't know how to uninstall them to see how it works out of the box? I'd also like to be able to edit them if possible but can't figure out where to find them.
Here is a link to where you can download a trial build:
https://ci.appveyor.com/api/buildjobs/w8i9v8hwvs5c29do/artifacts/MuseSc…
Still wrinkles to iron out, palettes tend to crash when screenreader is turned on :-(. But I'm curious to hear if this works better for you in terms of the score being read as you navigate (with left/right or with alt+left/right).
In reply to Some initial success! Armed… by Marc Sabatella
I'm a jaws user as well. I went to the page where the jaws scripts were which you created, but couldn't install them, due to having a newer jaws version than what was supported by that version of the scripts. Are you planning to update the scripts so they work with newer versions of jaws?