GSoC 2017 - Timeline Navigation - Project Description
Hello! My name is Joshua Bonn and I am starting work on my project as a part of Google Summer of Code 2017. My project is called Timeline Navigator.
Bio
First, let me tell you a bit about myself.
I am going into my senior year at the University of Idaho studying in Computer Science. My current focus at school is Artificial Intelligence and Evolutionary Computation.
At the end of my sophomore year, I was also a Music Theory major until time conflicts arose. I have played piano since around 3rd grade and cello since 7th grade after switching from viola.
Thanks for reading. Now back to the project.
What exactly is a timeline in MuseScore?
The timeline is a simplification of the current score into a grid format as seen below. It contains rows for tempo, key signature, and time signature changes, as well as instruments. The columns indicate different measures. By clicking the different boxes or changes, the program will focus on those measures in the score.
Why is this helpful?
By creating this timeline, a user can simply navigate through large works quickly while keeping track of important features like rehearsal numbers/letters. The simplification helps place all the important information in a small area to allow for easy traversal.
Features:
I will have three sections. The first section is the main items of the timeline. These will be the ones I am working on first. The second section is for the additional items, such as the tabs at the top, the coloring of the rows, and any other small features. The last section will include all the interactions. This ranges from clicking on a box to focus on it, to CTRL+LeftClick on a rehearsal number to jump to the next one.
Main Items:
The Grid:
The rows are split into the following: Measure numbers, tempo, key signature, time signature, a blank separating row, and all the instruments. The measure number row has a number at every fifth measure as shown in the picture above. The tempo, key signature, and time signature rows hold the changes for those specific rows, also shown in the picture above. After the solid separating row, all the instruments are listed. Each box after an instrument name corresponds to the measure for that instrument. If an instrument is playing in a measure, the corresponding box will be filled with a color. As shown above, there is a slight change in hues between each row to make it easier to track each row.
Additional Items:
The Tabs:
Located above the grid (as shown in the picture), there are tabs relating to the different movement numbers. When a movement is in focus, the different tempo markings are displayed next to the number.
Rehearsal Numbers/Letters:
The timeline will also include rehearsal numbers/letters above the measures that have them. They will be located directly over the grid.
Changing Color Intensity:
If you look closely at the instrument rows, you'll see that the color changes within that row. This is displaying a change in intensity of that measure. Intensity is calculated as the number of notes in a measure. A higher intensity is a darker color whereas a lower intensity is a lighter color.
Bar lines:
At the end of measure 37 in the picture, there is a bar line depicted. This is just a simple example of the result. In the end, each possible bar line would be displayed on their corresponding column. For example, a repeat would be a double bar line with dots on the correct side.
Selected Boxes:
On measure 26 of the picture, there is a slightly darker gray box in the measure row. This signifies that there are selected measures somewhere in that column. As the picture shows, multiple measures can be selected and will be displayed with a black box around them.
Interactions:
Selecting a Box:
To select a box, the user clicks on the box they'd like to go to. To select multiple boxes, the user can click and drag over multiple boxes, click and then Shift+LeftClick to select a large box, or Ctrl+LeftClick multiple individual boxes.
Selecting a Tempo/Key Signature/Time Signature Change:
Clicking on one of the boxes found above the divider in the grid will automatically move you to that location. CTRL+LeftClick on any of these changes will move to the next change of its type. CTRL+Shift+LeftClick will move it to the previous change of its type.
Other Shortcuts:
Pressing CTRL+LeftArrow or CTRL+RightArrow will automatically shift the view on the timeline a specific number of measures to the left or right respectively. The number of measures shifted is decided by the number of measures currently displayed on the screen. Pressing CTRL+Shift+LeftArrow or CTRL+Shift+RightArrow will shift the view to the beginning or end of the movement respectively.
General Navigation:
To navigate the timeline, there will be a horizontal and vertical scroll bar. For the horizontal scroll bar, it will scroll through the measures, but all the row names at the left (in the picture) will stay visible. For the vertical scroll bar, it will scroll through all the instruments but will leave the information above the separator.
Hiding Rows:
The user will have the option to hide specific rows to make navigation easier. The options include: all the instruments or different sections of instruments as the user decides. To select a section to hide, the user will select multiple rows using CTRL+LeftClick. After they press Enter, it will collapse into a row with a small arrow on the left side to depict hidden rows. If the arrow is clicked, the rows are placed back. This same functionality is used if all instruments are hidden.
Displaying Vocals or Chords:
If the user hovers over a box, a tool-tip will be displayed with the chord in that measure, if there is one. If the row is a vocalist, the tool-tip will display the lyrics in that specific measure.
Phew! Thanks for reading all that!
Please take a moment to comment on the project. Currently, I am working on finalizing the user interface (UI) of the timeline before beginning coding on the 30th of this month. I would love to hear all feedback you have on both the UI and the project itself.
*NOTE: I have finals until the 12th so I might be a bit slow at responding. *
Thank you!
-Joshua Bonn
GitHub: https://github.com/JoshuaBonn1
Comments
I like this a lot! Just a few points where the concept could maybe be improved:
I'm not sure the changing color intensity is a good idea. For one thing, "calculated as the number of notes in a measure" is not the only metric for "intensity"; it would make a measure filled with one long held note the same color as a measure with one very short note and rests. (Also, how would you account for ties?) You could just as well measure "intensity" by how much of a measure's duration is note vs. rest, but I'd prefer none of the above; just two colors for "contains note(s)" and "does not contain notes" would be simpler all around, including for the user to interpret.
For hiding rows, the process you describe seems non-obvious. What about something visible in the UI, like a "Hide" button that appears over the row label on mouse-over, or even a simple checkbox? I'm also not sure I fully grasp what the UI would look like to show that rows are hidden—a single row with the word "More...", perhaps?
What you describe about selecting multiple individual boxes would probably require more work than you realize, because MuseScore currently has no way to select non-contiguous measures. See https://musescore.org/en/handbook/selection-modes, particularly the part about range selections.
No reason to slavishly copy an existing realization of the same idea, because I'm sure you can do it better (in fact your mockup already looks nicer), but you may find inspiration in Sibelius's Timeline, described here: http://www.scoringnotes.com/news/sibelius-7-5-announced/
Looking forward to using your work in the future!
In reply to I like this a lot! Just a few by Isaac Weiss
The different colors should probably be described as density rather than intensity. In that case I would suggest an 1/8 note and whole note constitute the same density and would be lighter compared to a measure filled with 16th notes.
In reply to The different colors should by mike320
The original idea was that a measure with a few notes, like a pickup measure, would have a lighter color, where as an intense measure, like 1/16 note runs, would have a darker color. So, basically the density of the measure. Note that the colors in the mockup are not quite what they would look like in the end (if this ends up happening) but it gives the basic idea.
Thank you for your comment!
-Joshua Bonn
In reply to I like this a lot! Just a few by Isaac Weiss
Thanks for the comment! I appreciate that you like the idea.
As for color changing, that was one of the reasons I wanted to have this posted early to discuss the idea. I do agree that it would be more visually simple and easier to interpret without extra coloring. I will keep listening for more input on the matter.
I realize that the description might be a bit poor for hiding rows. Let me explain it here (with a bit more obvious choices):
When hiding rows, the user could select rows using the keyboard shortcut and a "Hide..." button will appear next to the first selection. After clicking that button, the rows will collapse into one row with a button "Show..." next to it (or something along those lines). After clicking that button, it expands the rows to show all the instruments again.
In terms of the multiple boxes, I realized that I did not properly remember how MuseScore did selection. So, selecting multiple individual boxes would be out of the question, but selecting sections of the score would stay the same, using the keyboard shortcuts described in that link.
Thank you for the comment!
-Joshua Bonn
In reply to I like this a lot! Just a few by Isaac Weiss
Regarding shading density, I think that different users may prefer one way or the other. I think charts for a smaller number of instruments are generally going to have all those instruments playing for almost all of the time, in which case binary shading is going to produce a timeline that is just a contiguous block of a single intensity, which to me would not communicate any useful information, so I know I would prefer having shading intensity be a function of density of notes for such charts to give some high level overview. While charts that have more instruments are going to generally not have every instrument playing all the time, and so maybe the user would prefer to have a binary shading if is primarily concerned with instruments playing a notes or not. So to me I would have the shading as an option, probably with default being binary on/off (and I would implement the binary option first).
Looking at that sibelius pic now...
I like how the grid lines are grey and pixel thin. I like that better than heavy black because the grid is not important information.
Sibelius doesn't have a thick black diving row, and instead has the measure number row in a darker shade of grey in place of the that. The thick black row isn't terribly useful. Having the measure numbers where that thick black row is more useful since their measure row acts as divider between the meta rows and the instrument rows, keeps the measure numbers closer to the instrument rows, and also and also saves vertical space.
I'm also noticing that sibelius has added a light bluish-grey shading to every measure not currently visible in the score view. In particular it is interesting how the 2nd Violas through Contrabass in measures 303-306 are not actually visible in the score view, and the timeline shading reflects this. I think that would be a useful feature.
I'm also noticing the sibelius has the same color (purple) for the rectangle in the score view as in the timeline. (I've never used sibilieous so I don't know if that is the playback rectangle or selection rectangle). Now I'm thinking it would be nice is musescore likewise used a filled blue cell to mean that cell is being played back and an outline blue to mean that the the measure is selected, so that color is consistent between timeline and score view. This unfortunately means that couldn't use blue to represent occupation (and that would mean couldn't alternate blue & green for odd/even shading).
Looking at that sibelius pic now, I am appreciating the advantage of just using boring grey for occupation. But now I realize that it is not possible to do odd/even shading if use greyscale while having intensity representing density of notes...which is a dilemma. I'm also noticing sibelius doesn't use odd/even shading and that makes it really hard for my eyes to determine what rows correspond to what instrument when looking at the rightmost part of the timeline, and this is especially going to be a problem on wider screens. Maybe there is some other way to keep track of which rows belong to which instruments.
For rehearsal marks, I much prefer your design of putting rehearsal marks inside of black-outline rectangles (since that corresponds to how rehearsal marks appear in the score), instead of how sibelius has them as another meta row.
In reply to Looking at that sibelius pic by ericfontainejazz
I was brainstorming a possible way to avoid using odd/even shading while still being able to keep track of which rows belong to what instrument on wide screens, and it occurred to me that maybe putting the instrument abbreviation at the begging of each instrument's re-entrace might do the trick. I modified to the sibelius screenshot to get a feel for how it looks:
I sortof like this over the use of odd/even shading, because now don't have to move my eyes all the way to the left edge of screen just to figure out what instrument a rectangle belongs too. (I realize I had suggested you try odd/even shading in the first place.)
In reply to I was brainstorming a by ericfontainejazz
Do you think there could be a problem of a very dense score where everyone is playing the whole time? If say 6 parts were playing continuously for the 40 measures shown above, there wouldn't be any beginnings of each instrument's re-entrance. If that was the case, I think putting the abbreviation at regular intervals would solve that problem.
I think I like the look of that more than the odd even shading as well.
In reply to Do you think there could be a by JoshuaBonn1
good point... It might make sense to include instrument abbreviation every 20-or-so measures if for instruments that have been continuously playing that long. Or instead only have instrument label at every 20-measure mark. Never know for sure until see it for real.
I found an example how Sibelius' repeat meta row look like from https://www.youtube.com/watch?v=FyaTTHEkQf8:
In reply to I found an example how by ericfontainejazz
The "Repeats" row has the label "Repeat" I'm guessing whenever there is a repeat. (That doesn't seem to be too useful to me...I'd much rather that use a unicode or MUSFUL glyph for ":|" or "|:"). Also has the sign symbol and "To Coda" and "Ds al Coda").
Also notice that it is zoomed way out. Each instrument row is very small.
After a discussion on the IRC, I now have an updated timeline. There are two pictures to be posted here, one with repeat barlines with a graphical representation and one with repeats on their own row.
Graphical representation:
![TimelineNoRepeatMeta.png]( "TimelineNoRepeatMeta.png")
New Row Representation:
![TimelineRepeatMeta.png]( "TimelineRepeatMeta.png")
There are a couple of changes to the timeline. The orange box signifies the measures currently in focus on the screen. It will change as the focus changes. The blue box signifies the measures currently selected. Also, the tabs at the top have changed slightly. Because MuseScore has no interpretation of a movement currently, the only use for the tabs is for section breaks in the score. So, each tab would have the text entered in a section break. If no text is entered, it would say "Section Break #_" in its place. I would also round the edges of the boxes that have the tempo changes, key signature changes, and time signature changes (and maybe repeats).
Please comment with your preference and any other ideas you may have!
Thank you!
-Joshua Bonn
In reply to After a discussion on the by JoshuaBonn1
I definitely prefer the graphical representation of repeat, since it communicates the type of repeat. However, we might still need to put DS'es and Codas and Signs somewhere...I would personally prefer them to be on the same row as Rehearsal Marks rather than wasting an extra row just for them. There would only be a conflict when a DS/Coda/Sign occurs in the same measure as a sign (the only real life use case scenario would be a coda that also has a rehearsal mark at that measure), so I don't really see a problem with sharing a row. Could resolve the conflict by first rendering the DS/Coda/Sign and then render the Rehearsal Mark overlapping but slightly to the right, similar to how Sibelius resolves conflicts when text labels overlap on the timeline rows.
In reply to After a discussion on the by JoshuaBonn1
I wanted to post my mockup of using the same shade of grey used for the repeat barlines as for the vertical lines when they overlap a playing region, and to also round the corners when an instrument enters and exits, which helps to visually identify each instrument's playing region as one bigger rounded rectangle as opposed to a bunch of little cells:
Regarding vertical grids, there is one issue to consider...that is when zoomed out so far that it is impractical to display every vertical line...for instance here is screenshot of sibelius zoomed out:
where there is only a vertical line every 5 measures. The corner case of an start repeat and then an end repeat in the next measure would be hard to display because the paint jobs would overlap. But I should note that the way siblelius writes the word "Repeat" would also cause a conflict anyway...I guess sibelius just puts two "Repeat" labels on top of eachother.
In reply to After a discussion on the by JoshuaBonn1
One other thing I mentioned in IRC was to round the right edges of each label, so is obvious that they're referring to the first measure:
In reply to One other thing I mentioned by ericfontainejazz
What happened to the slightly different hues for the alternating rows? That, along with the clear dark outlines of each cell, was one of the main reasons I thought the first mockup looked nicer than Sibelius's; both seem to be gone in these latest mockups.
I would strongly advise against rounding the corners where an instrument enters and exits; as mentioned in the Sibelius 7.5 review, that's what a DAW's UI looks like, and it makes it look like you can drag things around on top of the grid instead of them being set into the grid.
I definitely prefer the graphical representation of the repeat barlines, but like the standard barlines, the contrast is too low, and they need to be darker.
In reply to What happened to the slightly by Isaac Weiss
The slightly different hue might still work. The problem with blue is that the selected measure rectangle and the currently played-back measure ought to be blue since that is how musescore colors selection & playback, and I'd like to remain consistent with that interpretation. If a slightly different hue for alternating measures were used, I would thing need to avoid the same blue. Maybe even be a greyish-blue and greyish-green...just enough that can identify the lines as different provided that blue selection works. The nice thing about the Sibelius screenshot is that its is very obvious what the purple rectangle represents.
The dark out line of the cells might look fine for a small png, but it is going to look very ugly when extended full width of screen.
Can you point me to the sibelius review you are talking about. I can imagine how somebody might think they can drag it around, but I'm don't quite see there is a problem. I think it helps identify an instrument's continuously playing time as one unit, which I think is good.
The repeats could be made a little darker, but as I mentioned below they stood out too much when fully black, and there is a problem I mentioned elsewhere with making the grid black.
In reply to The slightly different hue by ericfontainejazz
It's the same link as before: http://www.scoringnotes.com/news/sibelius-7-5-announced/
Quote: "...the Timeline so resembles a sequencer-style track editor that instinctively I found myself clicking within it and trying to copy or move items, or trying to re-order the tracks in the left-hand column by simply dragging them."
As to "it helps identify an instrument's continuously playing time as one unit," that's already clearly identified by the color difference where the instrument is playing. Rounding corners doesn't add anything more, but it does misleadingly appear exactly like a different user interface metaphor.
In reply to It's the same link as before: by Isaac Weiss
Yes you posted that link, which I had read, but sorry I didn't know that was the "review", which is why I had to ask again, but thanks for reposting. I still don't necessarily think that an average user would expect each region to be movable just because of the rounded edges, unless maybe for heavy users of DAWs that do this. I'm curious if other musescore users think the rounded edges cause them to think that they can drag.
There were quite a bit of variants and rationales we discussed on IRC last night after looking at that link you posted for inspiration and comparing mockups, but me quickly try to summarize. When looking at a full width timeline zoomed out with a lot of instruments on a widescreen, I realized that even with odd/even shading, it is very hard to keep track of which filled area corresponds to which instrument (and this problem is even slightly more difficult when considering that there may be a vertical scroll bar), since you have to scan your eyeballs along a narrow row all the way to the left of the screen just to figure out what instrument a region belongs too. So we decided would be useful to add an instrument abbreviation at the beginning of each instrument's reentrance (and maybe after every N measures if a region was very long), although that abbreviation was not included in this latest mockup. I find that rounded entrance helps to communicate that the label at the start of the region belongs to that entire region, and also not to instrument above or below. And actually these little labels to make it easier to identify the instrument than by using odd/even shading, so odd/even shading is no longer necessary (and odd/even shading might even cause some confusion when trying to grey out regions outside the area that is not currently visible in score view...so for the time being we were looking at avoiding odd/even shading)
Also, when both the vertical and horizontal gridlines were the same color, it was even harder to identify each instrument region (and especially now without odd/even shading). (Maybe there is a way to go back to odd/even...maybe we don't need to grey out the region outside the currently viewable area...maybe the orange outline is sufficient). But anyway that is why I tried a modified mockup with the vertical gridlines to the same dark grey as the repeats, underneath the the light grey horizontal gridlines...and looking at that grid is easier for me to identify each instrument, so maybe the rounded edges are not needed as much if use that type of grid.
Anyway it was taking a bit of time make mockups...and unfortunately a lot these issues only become apparent when looking at a widescreen mockup instead of just a small mockup.
In reply to Yes you posted that link, by ericfontainejazz
I believe that a compromise can be found. After looking at MuseScore's general UI, most of the rectangles tend to be rounded slightly. Also, MuseScore is more rounded in style overall (see the main logo). After messing around a bit, I've created three simplified mockups displaying: no rounded rectangles, slightly rounded rectangles, and more rounded rectangles.
No Rounded Rectangles:
![MockupNoRound.PNG]( "MockupNoRound.PNG")
More Rounded Rectangles:
![MockupLargeRound.PNG]( "MockupLargeRound.PNG")
Slightly Rounded Rectangles:
![MockupSmallRound.PNG]( "MockupSmallRound.PNG")
For the moment, the UI is extremely simple as this is just an example.
I also experimented with more instruments (simulating a zoomed out view) and had this result with the more rounded rectangles.
Many More Rounded Rectangles:
![MockupManyLargeRound.PNG]( "MockupManyLargeRound.PNG")
There is still some wiggle room on how rounded the rectangle is, but I hope this gives a good idea for how it feels.
In reply to I believe that a compromise by JoshuaBonn1
Thanks for these. I definitely prefer rounded rectangles (although I don't know how much), since makes each instrument region feel definately separate from its above and below neighbors.
Anyway, no need to focus too much on that now since not everyone will agree...can always tweak at end of summer.
In reply to After a discussion on the by JoshuaBonn1
I liked the blue hues in the measures indicating intensity. The black repeat lines would stand out on it.
One more question, does your second picture imply that you will have the option of hiding score information lines. Picture two has a Repeat line, while picture one does not.
In reply to I liked the blue hues in the by mike320
The problem with black repeat lines is that they stood out too much when the regular vertical grid lines were not black. The problem with having the regular gridlines black is that then the grid becomes too heavy, especially when zoomed out, and gets in the way of the information. Black grid is going to look very ugly when the timeline is extended the full width of the musescore window, especially if on widescreen.