implement OpenSL ES audio driver backend for android

• Apr 3, 2016 - 21:17
Type
Functional
Severity
S5 - Suggestion
Status
active
Project

Audio output from MuseScore desktop app in Android incurs quite a bit of overhead, at least for the two cases I've tried:

  • if using a VNC, the audio first gets rendered inside musescore to ALSA, but then has to go through vnc network packets through loopback interface, possibly compressed/decompressed, and then in the VNC app is sent to android audio system.
  • if using XSDL X-sever, which comes with a PulseAudio server, then before executing musescore you first set an environment variable to specify the PulseAudio server as the localhost ip address pluse some port number. But similarly, the audio still has to be sent over network packets through loopback and the XSDL android pulse audio server still has to unpack those packets and send to android audio system. Seems to produce very choopy audio, which can be compensated by lowering bitrate.

So it seems would be much more direct to add another audio backend for OpenSL ES in musescore, just like mscore/alsa.h, mscore/pulseaudio.h, or mscore/jackaudio.h, which would extend the Driver class, so that musescore can speak directly to the android audio system. Documentaiton on how to use the C++ bindings via native development kit are here: http://developer.android.com/ndk/guides/audio/index.html

Also probably in order to get this working, might have to not use a chroot, and rather put every single library dependency in the AppImage. That's a separate issue, which might require compiling with android's NDK toolkit, and possibly maybe even have to use libbionic instead of standard linux libc. (I might have to test this out with a simpler program before trying musescore.)


Comments

FWIW, Chromebooks operate on an at least slightly similar principle (a system that allows you to have a full Linux environment within a chroot). Chrome uses a utility called "cras" to allow applications to interface with the audio system more easily, and manages to work somewhow within the chroot. Not sure if something similar exits or could be created for Android. Seems like each application shouldn't have to do this for itself. But I fully admit I don't understand the issues involved.

>> "Seems like each application shouldn't have to do this for itself."

I looked to see if portaudio supported OpenSL ES, but all I could find was some discussion in 2010 mailing lists as well as this ticket for portaudio from 2011 with priority low and milestone V20, but which has no activity: https://www.assembla.com/spaces/portaudio/tickets/154-portaudio-impleme…

It might be more worthwhile to implement OpenSL ES as backend for PortAudio. Then other apps that use PortAudio, like Audacity, could benefit. That ticket seems to be set to "Private"...I tried loging into assembla with my google account but when I visit that ticket page, there doesn't seem to be anyway for me to post a comment.