my.OnSIP for iPad: Part the Second

This blog is by Bryan, Software Engineer here at Junction Networks. This is a good read about adapting a product to the iPad; it contains some tips for developers and is the first of a series Brian will write about updating my.OnSIP for the iPad.


In the previous installment of this series, I mentioned that audio in my.OnSIP wasn't working on the iPad. In this installment, I'll go over the issues and workarounds.

my.OnSIP uses HTML5 audio whenever possible. If, for some reason, it can't use HTML5, we fall back to a Flash player; and if that doesn't work, we use the truly old-school <embed> tag to load audio.

Because Mobile WebKit supports HTML5 and doesn't support Flash, I'll elaborate on the quirks and gotchyas of HTML5 audio in the Mobile WebKit environment.

Mobile WebKit Audio

The first thing to be aware of when trying to load audio or video on iOS (née iPhone OS) devices is that media is not loaded until a user specifically requests it. From the Safari HTML5 Audio and Video Guide:

In Safari on iPhone OS (for all devices, including iPad), where the user may be on a cellular network and be charged per data unit, autobuffering and autoplay are disabled. No data is loaded until the user initiates it. This means the JavaScript play() and load() methods are also inactive until the user initiates playback, unless the play() method is triggered by user action. In other words, a user-initiated Play button works, but an onLoad play event does not.

This, obviously, has ramifications when trying to automatically beep on incoming messages and, not obviously, when trying to play a voicemail message.

Whither beep?

Because the beep is played whenever a message is received, instead of when a user requests it, I can't just stick an audio element in and ask it to play. Somehow, the media of that element has to be buffered before it can be used. I tried a couple approaches to solve this problem, but they each had issues.

First, I thought that I could hijack the tap of the login button, which must be pressed every time my.OnSIP is loaded and before any messages can be received (and, therefore, before any sounds related to incoming messages can be played). Unfortunately, there is one important case where Safari will stop, but my.OnSIP will keep working: switching briefly (for less than one minute) to another app and then switching back to Safari running my.OnSIP.

Safari closes and forgets that the media has already been buffered, but my.OnSIP will continue running. In fact, when returning to my.OnSIP it will only appear as though it had been running. To deliver real-time events to the browser, my.OnSIP uses XMPP over BOSH. BOSH allows up to a minute for a session to re-establish before closing the connection down. When Safari is returned to within a minute, the BOSH session re-establishes and it looks as though everything had been running all along even though Safari had actually closed down for a while.

This had effectively shut me down, but I had one more thought after talking with Nicole about the issue: what about HTML5 manifests?

Manifests allow an application to request that data be stored locally on a given device so they won't need to be loaded from the network after a cache flush. This should circumvent the media loading policy because no data is being requested over potentially expensive cellular networks.

I played around with this for a while; and I could, in some tests, actually get sounds to play automatically by adding them to a manifest file even across restarts of Safari. However, the manifest file introduced more trouble when trying to get it to play nicely with app updates and cross-browser compatibility. In the end, I decided to call it a wash and just forgo incoming message sounds, but this avenue still warrants further research.

Voicemail Playback

Voicemail playback is slightly more complicated. To play a voicemail you have to make a request to our XMPP API which, if you're authorized, will return an HTTP URL pointing to the voicemail requested.

Because the URL of the voicemail isn't known until it's requested I can't simply stick an <audio> tag in the HTML ahead of time and let the browser fetch and play it when the play button is tapped. Remember that audio and video requires a user-generated event in order to load and play. On top of this we only receive the proper URL as an asynchronous response, which means that we no longer have a user-generated event by the time the voicemail URL is returned to us. The ramification of this is a two-stage playback. First, you select the voicemail to be played, which makes the XMPP API request, when the request is returned we create an <audio> tag with the appropriate URL as a src attribute. This will display the native iOS audio control, which must be tapped in order to actually play the content.

Audio HTML vs. JavaScript

In theory, the DOM's Audio object is equivalent to an <audio> tag. In practice, many browsers have flaky support for either the object or HTML tag form of the audio element. The iOS devices I tested could not cope well with the Audio object; and, some of the Firefox browsers I tested could not cope well with the <audio> tag. Because we already had a Flash-based player that worked well with our desktop browsers, I chose to use the existing player everywhere possible and only fall back to the native controls via <audio> on iOS devices.

My.OnSIP on My.iPAD

These were all the issues I had getting my.OnSIP to work on iOS. Overall, I found it to be both easier and more frustrating than I had first thought. Basic HTML and JavaScript support Just Worked™ like it was supposed to, but media handling turned out to be more problematic than I'd first thought. However, while I was frustrated at times trying to retrofit our media capabilities to iOS, I found the issue made sense after some research. (Even if they did require a bit of hoop-jumping.) Mostly, I was upset with the derth of documentation available and litte or no description of some of the gotchyas I hit. I hope that these articles will ease someone else's journey into iOS land. It's worth it!