This blog is by Bryan, Software Engineer here at Junction Networks. This is a good read about adapting a product to the iPad; it contains some tips for developers and is the first of a series Brian will write about updating my.OnSIP for the iPad.
In the previous installment of this series, I mentioned that audio in my.OnSIP wasn't working on the iPad. In this installment, I'll go over the issues and workarounds.
my.OnSIP uses HTML5 audio whenever possible. If, for some reason, it can't use HTML5, we fall back to a Flash player; and if that doesn't work, we use the truly old-school
<embed> tag to load audio.
Because Mobile WebKit supports HTML5 and doesn't support Flash, I'll elaborate on the quirks and gotchyas of HTML5 audio in the Mobile WebKit environment.
Mobile WebKit Audio
The first thing to be aware of when trying to load audio or video on iOS (née iPhone OS) devices is that media is not loaded until a user specifically requests it. From the Safari HTML5 Audio and Video Guide:
This, obviously, has ramifications when trying to automatically beep on incoming messages and, not obviously, when trying to play a voicemail message.
Because the beep is played whenever a message is received, instead of when a user requests it, I can't just stick an audio element in and ask it to play. Somehow, the media of that element has to be buffered before it can be used. I tried a couple approaches to solve this problem, but they each had issues.
First, I thought that I could hijack the tap of the login button, which must be pressed every time my.OnSIP is loaded and before any messages can be received (and, therefore, before any sounds related to incoming messages can be played). Unfortunately, there is one important case where Safari will stop, but my.OnSIP will keep working: switching briefly (for less than one minute) to another app and then switching back to Safari running my.OnSIP.
Safari closes and forgets that the media has already been buffered, but my.OnSIP will continue running. In fact, when returning to my.OnSIP it will only appear as though it had been running. To deliver real-time events to the browser, my.OnSIP uses XMPP over BOSH. BOSH allows up to a minute for a session to re-establish before closing the connection down. When Safari is returned to within a minute, the BOSH session re-establishes and it looks as though everything had been running all along even though Safari had actually closed down for a while.
This had effectively shut me down, but I had one more thought after talking with Nicole about the issue: what about HTML5 manifests?
Manifests allow an application to request that data be stored locally on a given device so they won't need to be loaded from the network after a cache flush. This should circumvent the media loading policy because no data is being requested over potentially expensive cellular networks.
I played around with this for a while; and I could, in some tests, actually get sounds to play automatically by adding them to a manifest file even across restarts of Safari. However, the manifest file introduced more trouble when trying to get it to play nicely with app updates and cross-browser compatibility. In the end, I decided to call it a wash and just forgo incoming message sounds, but this avenue still warrants further research.
Voicemail playback is slightly more complicated. To play a voicemail you have to make a request to our XMPP API which, if you're authorized, will return an HTTP URL pointing to the voicemail requested.
Because the URL of the voicemail isn't known until it's requested I can't simply stick an <audio> tag in the HTML ahead of time and let the browser fetch and play it when the play button is tapped. Remember that audio and video requires a user-generated event in order to load and play. On top of this we only receive the proper URL as an asynchronous response, which means that we no longer have a user-generated event by the time the voicemail URL is returned to us. The ramification of this is a two-stage playback. First, you select the voicemail to be played, which makes the XMPP API request, when the request is returned we create an <audio> tag with the appropriate URL as a
src attribute. This will display the native iOS audio control, which must be tapped in order to actually play the content.
In theory, the DOM's
Audio object is equivalent to an <audio> tag. In practice, many browsers have flaky support for either the object or HTML tag form of the audio element. The iOS devices I tested could not cope well with the
Audio object; and, some of the Firefox browsers I tested could not cope well with the <audio> tag. Because we already had a Flash-based player that worked well with our desktop browsers, I chose to use the existing player everywhere possible and only fall back to the native controls via <audio> on iOS devices.
My.OnSIP on My.iPAD