Introducing Telemetry in the Notation Software

• Dec 8, 2019 - 00:25

We're all aware that a lot of improvements need to be made to Musescore 3. In the past, our focus has been directed by conversations held on the forum or feedback sent directly by users. In addition to this, we have now started to regularly user-test the app on a wide variety of musicians (thanks to Tantacrul). This is helping us get a sense of who our users are and where they are succeeding or struggling with Musescore 3.

Feedback and forum conversations help us learn about bugs, missing features and underperforming functionality; testing allows us to see what parts of the app are working well and what parts are invisible to the user. However, there are a lot of other things we want to understand. For example, we currently populate a significant portion of our top bar with large buttons for the following: New, Open, Save, Save to Cloud, Print, Undo, Redo, Zoom Controls & 'Concert Pitch'. What we don't know is the percentage of our audience who use these buttons instead of the file menu or shortcuts. Tantacrul: "From my previous experience designing Paint 3D at Microsoft, I can tell you that our most used control was the Undo button and I would expect that to be the case here." However, we have a suspicion that print, save to cloud, zoom controls and 'concert pitch' are far less used and could safely be moved elsewhere to make space for functions that get a lot more day-to-day use.

The ability to look at telemetry data to understand what aspects of the interface are being used is a vital tool in helping us prioritise effectively. Take the inspector, for example. We all know that it needs to be rearranged to provide better contextual options but how should we go about doing that? Our first problem is trying to determine the 'value' of a certain functionalities over others. If we determine that a feature is niche and demote it to a secondary or 'advanced' menu, there are inevitably going to be complaints. The problem then is determining whether those complaints represent 0.1% of users or 10%? Telemetry can help us answer these types of questions.

Telemetry would also help us determine whether new designs are performing well. At the moment, there's an open question about how many people have used the new customisation features in the Palettes panel and whether it has made much a difference to the experience of first-time users. If we suspected that aspects of the new design are suffering due to poor communication, we could track small experiments to validate that suspicion. In some cases, a big impact can result from very simple changes like rewording a phrase or making an icon less ambiguous.

High level telemetry

Another goal of telemetry is to track why Musescore is being used. For example: how many of our users open Musescore to complete quick musical tasks, like converting one file format to another? How many use it for large-scale orchestration and how many export parts? We'd like to know how much time is spent in the app too: how many sessions are under 30 seconds and how many are over two hours? How many sessions include an export to a midi file? High level telemetry should help us measure retention and stability more reliably too. It would be very useful to know how many people who have downloaded the app in the last 6 months are still using it? What percentage of sessions end in a crash?

The dangers of relying on telemetry

Despite everything mentioned above, I think it's important to state here that we do not intend to rely on telemetry to make decisions. Telemetry can tell you that something isn't being clicked on but it can't tell you whether the problem is down to bad design or just general disinterest. All telemetry can do is provide the smoke. Feedback and user-testing will show us the fire.

Respecting privacy & being transparent

We are now looking into systems that will allow us to track how people interact with Musescore without collecting any personal information and we think the most open way to do this is to ask permission from users (probably via a dialog box) when they install a new version.

This is our thinking so far and we'd like to open it up to the community to get your thoughts. There have been plenty of open source projects that have collected telemetry in the past. The ones that have been relatively successful (Citra, TimeScale DB, etc.) were open and transparent about the process. However, we're aware that there can be a resistance to information collection of any kind and we'd also like to hear the arguments for not doing it, too.

Снимок экрана от 2019-12-05 08-54-21.png

Comments

• Mar 17, 2020 - 10:19

I for one don't use plain dragging of elements but instead the Inspector's x and y positions mostly because those grid settings are not set by default.
So this i IMHO clearly is a case where lack of usage of a certain feature can lead to completely wrong conclusions.

Note though that I in do see why telemetry is useful. Just the conclusions drawn from them need to get thought through thoroughly.
Numbers can never ever get taken by their face value only, they can only give indications and food for further thought.

Reply

• Mar 17, 2020 - 13:51

It’s true that telemetry alone isn’t enough - and also true that this is a straw man since no one has ever claimed otherwise. But it’s not true that relying on telemetry data equates to saying, seldom used features will be removed. Only a naive designer would think that. Seldom used doesn’t necessarily mean unimportant, it might mean not sufficiently well exposed or placed or a dozen other things. And while the forum is one way to learn more, it’s hardly the only way - really not a great way, because responses will skew heavily to experienced rather than new users. But don’t worry - people who design software are not naive, we realize these things too. So please don’t assume telemetry data would lead to bad decisions, that’s both naive and disrespectful of software designers.

Reply

• Mar 17, 2020 - 14:36

It’s OK to be skeptical. To me the right response is then, to provide helpful encouraging suggestions on how to best interpret data, not to argue that the data shouldn’t be collected. Because for every bad decision coming from incorrect interpretation of statistics, there are ten more coming from not having the statistics to begin with.

Reply

• Mar 17, 2020 - 15:07

I would also agree that certain things seem obvious. And I would agree that in many cases, what the experienced users on this forum think of as obvious may turn out to have nothing to do with reality. Also, while to one users space bar might seem a commonly used key, another may literally never use it at all - either because they don't playback much, or because they tend to use the toolbar. And the list of "deeper things" that are far from obvious but can be detected through data analysis is enormous, far more than the trivially short example list provided here. See my recent common about the Style item on the context menu as but one of literally thousands of possible such examples.

Which is why both expert insight and raw data are useful.

Reply

• Mar 17, 2020 - 11:17

I've been using MuseScore 6+ years, and I didn't know until a couple of days ago that you could get the Style menu by right-clicking on open space. If you had "data-collected" (value-neutral term) me, you might infer that I never used it. It is not uncommon that Marc tells me "why didn't you just ...." and I learn another such thing. This skews data.

Reply

• Mar 17, 2020 - 15:02

For the record, I think the Style dialog has only been available in the context menu since maybe 3.3. It's pretty clever actually - it's not just for empty spaces, but actually for most elements, there is a Style item on the context menu that takes you directly to the corresponding page of the dialog. I believe 2.x also had style entries for some element types (text in particular) but it was more haphazard, and in particular, no Style item when clicking an empty area.

FWIW, it's pretty obvious this feature won't get much use yet because people haven't discovered it, and people already familiar with other ways of getting to the Style dialog will continue to do what they are used to. I almost never remember this myself, maybe this conversation will change that :-). Maybe over time it will prove itself more.

But it seems to me that the addition of features like this is exactly the sort of thing that can come from careful analysis of telemetry data. Prior to this feature being present, new users may well have tried right-click as a way to get to style settings, then become frustrated there was nothing there. Maybe they kept trying, and eventually found it in the menu. That's the sort of the thing telemetry data could in principle show was common. In other words, looking at what do people try to do in an effort to find some particular piece of functionality is data that helps one understand what the new user sees as "intuitive". I'm not saying it's the only way to get that sort of information, and maybe some implementation detail of how the data is collected actually makes this particular pattern something that couldn't have been spotted. But it's a good example of the type of thing I see it being useful for.

Reply

• Mar 17, 2020 - 15:13

Phew, so at least I didn't miss to notice it all those years, but my constant watching and even participation in the development still didn't result in me noticing.
I don't see though how telemetry could have helped here, yes, users might have right-clicked and yes, that might be recorded, but the reason for that right-click won't! So I don't see how anyone could from a right-click into an empty spot of the score come to think "Hey, that user is looking for the style dialog!", unless consulting a crystal ball, in which case the telemetry would not have been needed at all ;-)

Still I'm not against collecting those telemetry data (as long as the users have a choice, which they have). Meanwhile quite a bunch od such data should have accumulated, and I would like to see what that data shows now.

Reply

• Mar 17, 2020 - 15:32

Well, as I said, it's possible implementation detail would prevent this from working, but let's say we are trying to answer the question, "which items make most sense to include in the context menu". Anything already there and used a lot is an obvious candidate. Anything already there that Isn't used a lot is an obvious candidate for removal, if we get the sense the menu is becoming too cluttered. But the question of what new things to add we might try to answer by looking for patterns like this:

1) user right-clicks an element
2) user closes context menu without executing any command
3) user possibly opens another series of menus or windows and doesn't take any action
4) the next actual successful action is ... "X" (e.g., pressing OK in the Style dialog)

That would be pretty good evidence the user was looking for "X" all along, and they tried at first to find it in the context menu. One or two such cases proves nothing, could be the user only just happened to run into the Style dialog and got distracted away from whatever they were originally looking for. But collect enough data and maybe the trend develops. Look for patterns like this, find the most common values for "X", now those become good candidates to consider adding.

Reply

• Mar 17, 2020 - 20:42

I'm afraid the main point of the objections is being missed. It is the ethical issue. As far as I know, we, users, don't have any way to know what information is being sent home. If I'm wrong, please ignore what follows, and advise how to enable recording or saving the reports.
It is considered a general principle to allow anybody to know what information about them is or has been collected by an organization. I don't mean just personal information as it is defined by the law, but all information that is somehow related to the user's habits. I don't want at this moment to theorize about what could the seamingly innocent information of keystrokes or mouse clicks tell about myself, or about the individual using this computer, and what it could be used for: that's another discussion. And I'm not necessarily casting suspicion on the MuseScore team but, for instance, on a skillful hacker who could unnoticeably take control of some of the team's computers. No one is free from that possibility. Even high security governmental organizations have been hacked from time to time.
Note that I don't feel at ease just not opting in, since I could be helping and i'm not, so the phrase "just don't opt in" is not a solution fo me (and I believe many users would agree with me). Note also that I'm not asking for the reciprocity of knowing your keystrokes or even what you really do with my information, but just what information you collect, not as a general concept but exactly what you get, even if I probably cannot understand or make sense of it.

Reply

• Mar 17, 2020 - 22:23

As I said, not only conceptually, but exactly what package of information is sent each time. It's not a question of distrust, but of knowledge. Is that information available?
I've seen many times, when an application crashes, a message inviting to send some information that will help to understand why it crashed and, hopefully, to improve the application, along with a button to allow seeing the details of what will be sent. I'm not sure, because MuseScore is quite stable :) , but I think it also offered to see what is being sent. If not, it should...
Is it so difficult to offer to see the infomation being uploaded?
Are the keystrokes, clicks, etc. from people like me, who just want to know what of their information is collected, so little important for the project to just dismiss them with an "opt out" prompt?

Reply

• Mar 17, 2020 - 23:59

It is not that obvious. Even if I were able to read and understand the code, the information catched from my last session would be completely lost for me, since I cannot replicate each keystroke.
What I mean is if an end-user with little or no programming knowledge (one who doesn't have the know-how to hack the program and add some code to capture and show the information that is about to be uploaded) can see the specific intormation that is being sent home.
The answer is, clearly, no. Otherwise, somebody would have hurried to say here how to do so...
If this obvious feature had been implemented from Day 1, most of this discussion would have been avoided and much more users would have opted in.

Reply

• Mar 17, 2020 - 23:29

As I see it, those against telemetry have two points to make:
1. Invasion of privacy, therefore unethical.
2. Might opt in if they knew exactly what was collected. Otherwise see number 1.
(3. Not needed, therefore see number 1.)

I have looked at crash reports. they tend to be a few hundred lines of data that means less than nothing to me. I suspect there are some that would know something about what they would be looking at. Great. How would you know that what they said they collected was actually from your computer. Easy enough to switch ID data. Or that what they said was collected was actually everything they collected to begin with.

If you believe this to be an unethical invasion of privacy, great. So for you it is. But beware that not everyone thinks so. For them it is not an unethical invasion of privacy. Strange, I know. Just because you believe it is does not make it so for everyone. Likewise, just because I believe it isn't doesn't make it so for those who have a problem with it. We are over 220 posts into this thread and there hasn't been much new. Round and round we go.

Reply

• Mar 18, 2020 - 00:40

I'm not against telemetry. I reset to factory presets just the other day and again I opted in. What is confusing things is that there is a dialog box, yet we have not engaged in an actual dialog to find out what people are agreeing to. The question in the dialog box is "Would you like to help us improve Musescore?" Of course I want to improve Musescore. Then, it implies that I should either act by sending my data or refuse to act. OK. I want to send my data. I want my data included in the collection of data. I am sending the data to the other participants of the telemetry program. I am agreeing for the whole collective (of users who are opted in, like I am) to participate in reviewing and discussing the collection of data. I expect that whenever someone turns on telemetry in their Preferences, that they will have immediate access to this anonymous data. I am not agreeing that my data should be contributed to a bank that becomes the private property of a small class of worthy participants in this project.

Marc - Many of the arguments you presented recently are irrelevant - whether I am a stranger here, how much I have contributed, whether I can decipher the data. If people want to learn how to read this data, they can ask. How did anyone else here learn to read the data? You access it first.

Marc - Everyone here is a contributor. My questions and answers are just as valid as any other person here. Our feedback is invaluable. People who come in here asking for help are also giving help through their participation, their questions, their follow ups. If millions of people use Musescore, millions of those users do not post in these forums and give the kind of feedback that you are getting even now. The OP asked for objections and I am making my contribution. You who are not objecting, you who are defending some imagined agreement behind the option in the dialog box are not making a contribution to our purpose in this discussion. Our purpose here is to vent our objections. If anyone bothered to read the fine print in the OP.

The OP was basically a propaganda campaign after the fact. the campaign was started after telemetry was installed. the developers admit that they could have implemented this without ever notifying us. that is the really terrify aspect and the crux of the ethical arguments. The developers should have no power to do something like this. the question here is how to prohibit such activities taking place without our express consent, a clear understanding as to what is occurring and what we agree to. Simple dialog boxes cannot do this for us.

I cannot accept that there is a class of people in this project who are trustworthy with anonymous data and that there is some other class that is not. It's anonymous data. Data has already been forcefully taken from us against our will by this forum. And I strongly suspect that it has been misused at times. I don't like to suspect. I like to know with certainty. I don't believe of some class of technocrats that are always on the up and up. I don't believe that transparency should mean that I should be transparent to some secret class that refuses to be transparent with me. I asked who are the developers who are accessing this data, and got no response. Why? Is it a violation of their privacy? I asked for screenshots of all of the data. I haven't seen it yet. If it is anonymous data, why can I not see it? My eyeballs are the same as yours. One or some of the telemetry participants who donate their data to you might learn to understand the data and give a completely different interpretation to it that could propel you into a whole new phase of development.

You asked me in the dialog box if I would help you and I gladly said yes. now I'm asking you to help me. As I said when I entered this conversation - in order for me to object, as the OP has asked from us, I need complete information. How can I evaluate this without complete and accurate information? I need to see what I am giving to you and what it means. I need to see what this could mean if it were placed in conjunction with the forum data that you are and have been forcefully taking from me. I think that the hidden argument here is motivated by the fact that you have already taken data without real approvals, regardless of what it said in the Registration Agreement on this forum. No dialog box is valid without discussion of what we are agreeing to among ourselves.

I am beginning to conclude that you don't want me to see the data because I might object. I am helping you. I think you ought to reciprocate. You need full and complete data to evaluate the program. I need full and complete data to monitor your activities as administrators. We need to install telemetry upon the developers in order to stop them from ever taking something like this from us by force.

Reply

• Mar 18, 2020 - 02:55

" I expect that whenever someone turns on telemetry in their Preferences, that they will have immediate access to this anonymous data." Why? They already have the data.
" Data has already been forcefully taken from us against our will by this forum. And I strongly suspect that it has been misused at times." No one forced you to post on the forum. Oops,I took your quote out of context. Is that misuse?
Yes, there is a "class" of people here that are different from you. They are the ones that are responsible for this place being here.
I suspect that even if you saw what data was collected and how it was to be used, that you would still not trust those who would be using it. Even if you knew who "they" were. Now you are just being over-dramatic.

Reply

• Mar 18, 2020 - 03:39

No one forced me to post on the forum. I agree to post on the forum. I do not agree to anyone working with the administration of the forum to collect data on me. The data is being taken by force. That includes my location at this moment. Whatever the class of participants, we are all responsible for this place being here. The developers rely on information that comes from the users. That information allows them to evaluate the project. How can the users evaluate telemetry without the information? The purpose of this discussion is predetermined. It's a propaganda piece in support of telemetry with a request to hear objections listed in fine print at the end of the OP. This is another reason why concerns are raised. The act of installing this without notifying the users first, and the fact that it is possible to install these kinds of programs without the users ever knowing about it, is the basis for a discussion in ethics.

Reply

• Mar 18, 2020 - 03:38

It’s possible you are confusing me with someone else, as I never said one word about whether anyone was a stranger or asked how much anyone else has contributed.

Also, to be clear, these discussions were started before the first release that included telemetry. So it is factually incorrect to claim it is a “propaganda campaign” after the fact.

So pretty much the entire premise of your post is invalid, meaning there really isn’t that much else to respond to.

I’ll just reiterate what I already said: there is a public statement about what is being collected. If you personally are so distrusting of the people who provide this software that you continue to question it, the source is available. If you lack the skills yourself to read and understand the source, maybe start another thread where you politely ask others who do have the skill - and the necessary incentive, and free time - to do the research for you.

But I am completely baffled by your unfounded and patently absurd claim that someone has taken something from you here on this forum without your consent. it’s pretty difficult to continue rational discussion with someone who makes such utterly ridiculous statements.

Reply

• Mar 18, 2020 - 01:53

bobjp,
1) I'm not against telemetry. People included in your item 2) are not against telemetry. However they may have objections to some aspects of its implementation.
2) I don't believe telemetry per se is an unethical invasion of privacy, especially if I'm given the possibility to opt out.
3) What is unethical is that if I'm opting in (thus helping the project--and the future me, granted) I'm not allowed to know exactly what specific information a third party will possess from me. This makes me unable to decide if it is or not an invasion of privacy. The mere description of the kind of information to collect provided at https://musescore.org/en/faq#faq-300080 is not enough, much the same as the written law cannot substitute a Judge to interpret it in the light of a specifc situation.
4) The fact that the software is open source is no excuse, since 99.99 % of the users or more are unable to change the code to make the information available prior to uploading it.
5) I'm the exclusive owner of any information generated by my activities. Opting in is sort of a non-exclusive license for using my information for a specific objective, but I retain the right to have a copy of that information.
6) The part where you argue that the information they would provide could be fake, it is here where my confidence in the developers (which I myself have never challenged) comes into play.

Reply

• Mar 18, 2020 - 03:40

As I have said before, if you personally lack the skills or the time to peruse the source to learn the answers to the questions you have, feel free to politely try to encourage others who do have the skills, incentive, and time to do your research for you. Until then, if the public statements aren’t enough, it is your right to decline of course.

Reply

• Mar 18, 2020 - 08:14

Hmmm... I have really no questions and no request either. It is not me who needs telemetry. But I'm ready to sign in if I'm able to keep a copy of my information without having to hack the code myself or politely ask others to do so for me. If such condition is not fulfilled, I just opt out--be assured I know my rights.

Reply

• Jan 22, 2022 - 06:47

Hello

Will MuseScore be as "aggressive" as the newly acquired Audacity software in terms of Telemetry ?
Will the collected data be send to US and Russian law enforcement agencies upon request ?

Thanks

Reply

Do you still have an unanswered question? Please log in first to post your question.