Possible Design of App Store-like Plugin Manager

• Mar 19, 2019 - 16:00

Note: you can view this passage in a better format at Github page

Hi! I'm a GSoC student candidate. These days I've been investigating the idea of App Store-like Plugin Manager. Briefly speaking, it would be a new plugin manager, from which it is possible to discover, (un-)install, (auto-)update plugins in an easy manner.

Now, I've worked out a rather complete feature list, corresponding UI demos, and some discussion of implementation approach. All of them are listed below.

Any feedback from anyone is much appreciated.

Overview of Design

A new app-store like Plugin Manager is planned to be implemented. This tool can reside in Resource Manager(find it in MuseScore menu: Preferences->General->Update translations, or Help->Resource Manager) as a new tab, or just replace the old plugin manager.

In short, two main facilities are covered in this new plugin manager:

  • (Part I) Automatically install and manage plugin packages from MuseScore plugin repository.
  • (Part II) Traditional facilities for local installed plugins(some of those plugins may be not published online).

> Here I refer to the stuff you download from the plugin repository as a "plugin package".
>
> I want to stress the difference between a plugin package and a qml file, since one plugin package(such as this one) may contain multiple qml files.
>
> In light of this, one qml plugin can have two meaningful names: the first is base name of the qml file(corresponding to one particular item in MuseScore Plugin's drop-down menu), and the second is the title from the web plugin page, i.e., the name of corresponding plugin package.

Part I

Conceptual UI demo I made by merely modifying the UI file:

ui_plugin_store.png

All available plugins from the online plugin repository will be displayed in this table.

Possible features:

  • you can search for new plugin packages and filter them by category labels.

  • you can download new plugin packages and install them simply by clicking the "Install" button.

    > For plugins that are incompatible for your MuseScore version, the manager should disable the install button.

  • For downloaded plugin packages, you can enable, disable or delete them by clicking corresponding buttons.

    > These operation applies to all qmls that belong to the plugin package.
    >
    > In Part II, there are operations that apply to one single qml.

Part II

Conceptual UI demo I made by merely modifying the UI file:

ui_local_plugin.png

All installed plugins, at least for qmls that you manually added, will be displayed in the left QListWidget.

The text of each item in QListWidget is currently base name of the qml file.

Possible features:

  • In addition to using Part I's facility, you can still manually download the qml file yourself or write your own plugin qmls locally, and copy them to plugin directory. Then they will appear after reloading.

  • If local plugins are checked to be incompatible, necessary prompts can be added.

  • The traditional plugin manager's facilities will be reserved in this tab, including:

    • enabling/disabling each qml
    • shortcut configuration
    • displaying name, path, version and description.

    These facilities are arranged as before in the right panel.

  • For plugin packages installed from repository, there're several choices for displaying:

    • Just don't display them in this tab. Let users manage them completely in the first tab(Part I).

    Then facilities like shortcut configuration are lost.

    • One plugin package corresponds to one item in the left QListWidget.

    • Like local plugins, one plugin qml corresponds to one item in the left QListWidget.

    Personally I prefer the third choice, as it's flexible for users and relative simpler than the second choice.

    But the second choice is also considerable if we implement a tree view, displaying all qmls of one package under that package item.

Miscellaneous

  • To make things consistent, Add "Check for new version of MuseScore plugins" in MuseScore->Preferences -> Update.

    And add corresponding facility of update reminder.

Implementation

Fetching from Web

When we launch the resource manager, the crawler should fetch a list of available plugin packages from https://musescore.org/en/plugins?category=All&compatibility=some_version_id.

The map between MuseScore version andsome_version_id is:

MuseScore Version some_version_id
1.x 4311
2.x 4316
3.x 4321

> It's not hard to parse the raw HTML table using some libraries, though the parsing code would be simpler if there were a JSON file.

Each entry of the plugin list should contain:

  • Title(string)

    > The title here should use the title from the plugin page, i.e., the name of plugin package.

  • API Compatibility(a tuple of bool indicating compatibilities for 3 versions)

  • URL of the plugin page(string)

  • Categories(vector of enum object)

Download

Then when we click on Install button of any entry of plugin, the crawler will fetch the plugin page and extract structured data, including:

  • GitHub repo URL(if applicable)
  • Direct links that are recognized to be suitable plugin distributions
  • Attachment URLs that are recognized to be suitable plugin distributions

> This fetch procedure can also be considered to run automagically in advance and store in cache, as the total number of plugins is not huge.

If GitHub repo URL is available, we should use Github Release APIs to check if there's a release.

Else, look for direct links in the page or the attachments.

These steps would require sophisticated pattern recognizing algorithms to choose the correct version(2.x or 3.x, etc. ), which can be further discussed and optimized later.

Extract and Install

There are various types of downloaded plugin files. Some are zips, and others are just qmls.

> There are some code snippets used for unzipping language packages, which can be used similarly for plugin packages.

After the zips are extracted, we should remove irrelevant files such as README and others, and copy all qml files and folders that contain them to the plugin directory.

Warnings should be reported if there are conflicts when copying files, such as qml files with the same name already exist.

Maintaining the Local Plugin List

Permanent Storage

Not only the downloaded qmls are stored, necessary information about local plugins and plugin packages need to be stored.

Currently, MuseScore only stores some information for local plugins in plugins.xml(located in C:\[your username]\AppData\Local\MuseScore\MuseScore3). The xml file only stores the qml file path and flag of whether loaded for each local plugin. After MuseScore is launched, these items will be read into QList _pluginList in class PluginManager. (See here)

Qml files of plugin packages should also be added into plugins.xml. But beyond that, for each plugin package downloaded from repository, additional information should be maintained:

  • plugin package name or page URL. These are helpful to identify installed plugin package from the plugin list fetched from repository. (We cannot tell it merely from installed qml files.)

  • Paths of its qml files. This can be obtained after download/extracting the zip file. Those paths are useful when checking the integrity of local plugin packages and removing packages.

These information can be saved in a separate xml file.

Runtime Data Structure

Currently MuseScore uses QList PluginManager::_pluginList to maintain local plugins. When MuseScore launches, it fills this QList with contents from plugins.xml, and loads and registers plugins that are marked to load.

For plugin packages, there ought to be a new QList to maintain installed plugin packages. PluginPackage will be a structure that contains package identiity(name or URL), and an array of paths of its qml files.

This QList is also filled with contents from corresponding xml file. When the plugin manager is launched, integrities of each PluginPackage are checked by verifying existence of each qml file.

Compatibility Check

Each plugin has its API compatibility specified in its web page.

As far as I know, plugins of compatibility 1.x, 2.x and 3.x can only be run on MuseScore 1.x, 2.x and 3.x respectively. Though some of 2.x plugins can be ported to 3.x with some code changes, this process cannot be done automagically yet.

Compatibility check should happen in two cases:

  • When installing plugins from repository, before the download begins, compatibility list specified in the plugin web page should be verified against current MuseScore version. If current MuseScore version is not in the list, download will be disabled.

  • When importing/reloading local plugins, check whether the plugin is imported successfully.

    this can be done by analyzing the result of QQmlComponent::create(). See example code, where the errors() method contains related info.

Automatic Update

Most plugins don't have their own version numbers currently. So how to detect updates of plugins seems to be a tough problem.

A stupid way is to download the whole plugin again and look for difference.


Comments

  • comptibility check: the most simple change could be checked for automagically: changing the version of the import MuseScore and import FileIO from 1.0 to 3.0
  • automatic update: In case of lacking a version number, howabout looking at the timestamp?

In reply to by Jojo-Schmitz

Do you mean the manager should try to transform the plugin by substituting version numbers in the two import statements and then load again? Or just to use the version(1.0 or 3.0) to judge whether this plugin is for 2.x or 3.x?

I think timestamp is worth considering. I've checked the HTTP response from musescore.com, the Last Modified field of the response header is available. And for GitHub archives, we can look at the commit history.

I'll make an update in the proposal, thanks!

In reply to by songchao

Not sure, at first there's the 1.x, 2.x and 3.x field and we probably should only show plugins that are listed as compatible with 3.x.
But maybe as an advanced setting, show those for 2.x and offer the attempt to convert it by replacing just those one or two import statements? Including some feedback to that plugin page whether it worked or not, maybe via an issue raised against that plugin?
No point at all in offering 1.x plugins though, they are way too different.

In reply to by Jojo-Schmitz

OK, I'll add the converting facility to the list. Though personally, I think this is not a core feature.
As for feedback on whether the plugin worked, maybe it's the users themselves' duty to submit issues against the plugin, not the manager's. But maybe we can add an entry in UI, pointing to the issue page if the plugin loads failed?
Anyway, I think those 2 features should be considered secondary, compared to crawling, analyzing download links, etc. But of course, I'll consider them when the core part is done.

In reply to by Jojo-Schmitz

To your knowledge, do you know if there is a JSON/RSS in musescore.org that gives a list of plugins that are in the repository? Such as http://extensions.musescore.org/3.0.2/languages/details.json for language packages.
I guess there isn't one, but just to make sure :)
I'm planning to write a prototype for this project, just as a proof of concept, demonstrating basic features of fetching, displaying and downloading if I'm quick enough.

In reply to by Jojo-Schmitz

In the “Plugin Store” tab, packages installed from this tab can be easily marked as “Installed” according to the pluginpackages.xml in my current implementation.
But how should we mark those plugin packages that users have installed manually? Still mark them as “Uninstalled”? Before downloading one package, names of qml files are not known, and it would be hard for the manager to detect existing local files of that package.
I think one possible way is, to check for the existence of those qml files in plugin directory after the archive is downloaded, and give the options of replacing existing files, aborting installing or duplicating the files.

In reply to by jeetee

Good question.
Before I would assume the latest release is always for 3.x. That makes sense in most situations, since the developer usually makes updates to 3.x, not 2.x now.

But now I've come up with a smarter approach: check "target_commitish" in the json returned by GitHub API.
This field shows which branch the release comes from.
See the json at https://api.github.com/repos/jeetee/MuseScore_TempoChanges/releases.
Assuming the publishing rules like putting 2.x qml files in "2.x" branch and 3.x qml files in "master", we can easily check which release corresponds to 3.x.

Here are my thoughts on using QWebEngineView vs. hard-coded interfaces for the plugin store. (@peterjonas asked it and I post my answer here.)

  • One advantage of QtWebEngine is that, it shows the HTML table directly, which is more robust than the hard-coded Qt table, and won't be easily affected by potential HTML style changes, like changing the sequence of table columns...

    But the store is displaying not just the original table, but also each plugin's status, and needs to expose interfaces to let users manipulate them. Then the HTML table would be customized and be added with HTML buttons, and we need to find a way to detect users' actions in HTML(like clicking buttons) and trigger native C++ codes. It seems to involve more work to do in general...

    Meanwhile, as mentioned in "Implementation Overview", the crawler currently used is not hard to write and at least for now, I find it works fine enough, too.

    So as a conclusion, I prefer the Qt table interfaces for now, but different ideas are always welcomed:)

  • Another potential usage of QtWebEngine is that, it may be possible for users to preview the plugin's description directly in MuseScore by displaying the detail page.
    Generally, I don't think it's a good idea to let this project work like a mini browser in MuseScore...
    But it seems really plausible to only fetch the description body in the plugin detail page(a div element in HTML, you can find it by searching field field--name-body in that page, after pressing F12), and display this partial HTML via QWebEngine somewhere in the store tab.

    Seems like a good design? 😉

Here I'd like to put forward my draft specifications of the uploaded plugin format, including GitHub ones and musescore.org attachments. By design, supported repos and attachments should be correctly recognized and downloaded by our auto-updater. Unsupported ones, however, may still be downloaded but are likely to encounter problems.

If you find something not looking good, please feel free to tell me:)

GitHub repos that satisfy the following specifications are supported:

  • One repo only holds one plugin(which may contain multiple qml files though).

  • The 3.x version is in “master” branch.

  • Other versions, if available, should be in other separated branches.

GitHub repos with one or more following issues are NOT supported.

  • One repo holds multiple plugins.

  • Mix both 3.x and 2.x versions in the same branch.

Attachments within plugin pages that satisfy one of the following specifications are supported:

  • There's only one archive file as the attachment.

  • There's only one qml file as the attachment.

  • There're multiple archives or/and qml files, and each file is a complete plugin corresponding to a different version. Each file's name must explicitly contain info about plugin version like "3.x", "2.x" or others. This info could appear not only in the hyperlink, but also in other text in the same line I suppose.

Attachments with the following format are NOT supported:

  • There're multiple qml files and each of them is partial to the plugin.
    This means you should upload an archive if your plugin contains multiple qml files, rather than upload each file one by one.

Do you still have an unanswered question? Please log in first to post your question.