techno.blog("Dion")

When Ben and I joined Palm to run Developer Relations, we knew that we wanted to eat dogfood pretty quickly. We have had some mobile related projects in the past, but they were either mobile Web sites, or Java based. Being able to take our web skillz to rich mobile devices was much more new and exciting.

Building sample code and apps is useful for developers, and the team will be doing plenty of that (e.g. this is how you use the awesome List widget in every which way, and why you would do X, Y, or Z) but building a production app is a whole different level.

We wanted to learn what it is like for our developers to design, build, and distribute webOS applications. We could have started with a smallish app, but no :) Instead we took the great work of Justin Newitter (who built the original app) and went running with an updated Facebook app. Along with other great developers in the dev relations world, we have gotten our first early release out there today.

Here is a walk through of the app by PreCentral:

We have definitely learned a lot in our short time on the app, and this is the first release of many. I really want us to have a regular cadence to our releases. If there is a feature you are excited about let us know.

We added a broad feature set to the application. The first Facebook application was very much about synergy. Why put something in an application silo when you can bake it into the platform? I still contend that webOS is the best platform for Facebook users as it integrates throughout. Other mobile platforms do some of this, but I think we continue to do the best job here.

There are a bunch of Facebook features that our users still wanted though, so we worked with Facebook themselves to prioritize this list. The top features were access to Facebook Mail, full profile access (info, wall, and photos), people search, events and birthdays. We also wanted to do interesting things to the UI as we bring in these features. Oh, and a couple of mini-easter eggs.

Designing the application was challenging and a lot of fun. At a high level, working out a design that is true to webOS *and* a strong brand such as Facebook was a balancing act. You will run into this same issue even without a brand like FB. How does it have your DNA and still fit in with the core platform. This has been a design consideration since the dawn of time. Do you make a clean Mac app that looks like Mail.app? Do you do your own UI that looks like a funky Flash UI? Most of the time you are in the middle.

One feature that we spent time on was the base navigation. We wanted to make it incredibly easy to get to features you use often, and also quick to get to all features.

We ended up with a solution that made the following decisions:

News as Root

The root of the application is the news feed. Some Facebook apps have an icon window as the root, but we decided to behave more like the Facebook website itself. The news stream is the blood line, so start there. No matter where you go in the app, if you back gesture away…. you will always end up at the news stream.

Also, just like the iPhone app, we share the shake gesture as a way to refresh the data here. Shake away.

Status Matters

webOS devices have hardware keyboards and are great for creating content. In the Facebook context this means updating your status and uploading photos are prime ways to get your content into the system.

For this reason the top left area is your way to always get to your current status and update on it. In fact, if you are on the new stream, just start typing and the update area will pop down and capture your new status (while showing your last one below).

Also, if you click on the camera icon, you are sent into the core photo experience on the device which natively supports uploading to Facebook.

The Navigation Grid

For all other features, we wanted to give you a quick way to access them. Click on the top right grid and a pop down will immediately appear, giving you one tap access to any feature. No need to switch to a navigation screen first.

We are also playing with the ability to use that hardware keyboard by giving you quick key access to any feature (e.g. SYM + E == sends you to events). Is that a good idea?

Facebook Logo Power

The Facebook logo itself has some hidden love. In an homage to to websites, a tap on the logo takes you home…. which means back to the news stream. If you are on the news stream already, and have flicked down…. that same logo will bring you to the top.

Where to go next?

We are excited to offer access to data that Facebook users haven’t been able to get in an app before, but where do we go from here?

We definitely want to do a lot of polish on various sections that we have out there. One idea that I have been playing with from the get go is doing something immersive when you rotate the screen when in a news feed. Instead of just having the news feed work in that format, what if the content took over. I mentioned this in my last post about touch UI. I flick through the stream and if on photos, the album takes on the entire screen for example.

There are some other really fun features that revolve around webOS notifications and giving you a great way to “never miss a thing” (life moves fast you know ;) and choosing what content matters to you.

And, finally, we have to work out what makes sense in an app, and what could be baked into the platform. Giving access to birthdays is great, but would you like to have them as a calendar on the device? Maybe, but you would definitely want to be able to turn that view off…. and in fact you may already have birthday info in your profiles, so we should have one unified birthday view. Life gets more complicated when you go to the generic doesn’t it.

Again, we have just started here, but would love to hear from you on our feedback area in the app page.

Thanks to the team that played a part in this release! We haven’t reached Joe Hewitt foo yet, but we are having fun!

As you move to a new platform, it is interesting to watch your brain morph over time. I remember switching from Windows to Mac. At first the fonts looked blurry and weird. The mouse pointer didn’t weight right. The constant app menu was strange. There were things I liked about it right away, but they were mostly the fact that I had a command line, and the fact that apps were minimal, pretty, and useful.

Over time though, I grew to like the Mac more and more. A few months in and it was the Windows fonts that looked too sharp and weird.

I am going through the same experience with webOS and the iPhone. It took me awhile to get used to the back gesture on webOS and not look for a back button. Now I know how to organize my multiple windows, and use universal search as my quicksilver, and so much more. When I open up my iPhone now I am at the point where I try to do a back gesture by mistake. webOS is a touchier, needier device, and as I develop apps and play with the platform, I start to grok that more and more.

Embrace the touch

I discussed touching and horizontal scrolling awhile back, but the more I play with touch devices, the more I find myself wanting to build features for the touch. I have a ton of learning, but here are some of the lessons so far:

Native UI or Immersive UI

One decision that you have to make when you start building your application is the style of UI. Do you want a native looking UI for the given platform?

Everyone jumped on native off the bat. We quickly saw libraries such as iUI come out that let you mimic the iPhone UI. Having a native UI can be important. You want to fit in. However, we have also seen the growth of immersive UIs. Convertbot is the example above on the right. It is task based and you feel like you are really interacting with the app. It is almost tactile.

It is interesting to compare the Gmail and native Mail client UI in webOS. The Gmail version is deployed via a website, whereas the native version is of course an application, but they are very similar. Both use HTML/JS/CSS. Both have their look and feels. Do you try to look like your website (e.g. Google look and feel), or do you try to go fully native. The blending of the two gets interesting. Your brand has to live in another native world.

Haptics and touch feedback

It is usability 101 to make sure that you are always giving users feedback on their actions.

First, how do we signal to users that there are particular touch areas? This one is a bit of an art. We don’t really have :hover and the like. I actually like the idea of having a long press show helpful information, but users aren’t used to using that ability yet (see: need more gestures!).

Where we do have touch areas, we need to make sure to have various depress states for the touch.

Users will touch all over the app, so think about what you can do where.

We are going to see haptics in the future. For now it feels like haptics are used like this:

But the science is coming along. Sony Ericcson has a device (Satio) with haptic support for example:

Using the Keyboard: Software or hardware

This brings us to keyboards. What is the optimal input for your application use cases.

Feathers by Aral Balkan is a good example of both the task based nature of mobile apps, and custom software input. I like how Aral thought to create an app that solely creates Twitter messages. He didn’t create a full Twitter client that would do it all.

And if you have the pleasure of a hardware keyboard, how can you use that beyond the obvious inputting of text fields. The beauty of a keyboard that come out is that it doesn’t take away space from the screen. Could you offer short cut keys in the app? Different navigation? There is a lot more to explore here.

Gestures. Time to catalog and create new standards

We are seeing more and more gestures in applications. It feels like we are building out the standards right now. What will be the Ctrl-C’s of mobile? We get to build out invisible ways to navigate.

Tweetie 2 did something interesting when it threw away the refresh button and replaced it with the pull down. Isn’t it more work? Some people don’t like it (Jimmy Fallon for one!) but a lot of people find it more gratifying because it is more natural. We have buttons in the real world, but the apes in us are more used to touching the world around us in very different ways? This is one example of going back to our roots.

Speaking of reload, we are seeing another common gesture here too. Using “shake” to reload, or relayout.

Orientation: Try to accept them, and be more creative

Have you ever turned your device on its side and not seen anything happen? That frustrates me. At the least, we need to rotate the UI and let it continue. But, can we go beyond that? I have been playing with this. What if landscape brings a more immersive experience?

Take an app that loads a stream (e.g. Facebook, FriendFeed, Twitter, whatever). In landscape, you can view one entry at a time. If the type of entry contains a photo album say, take over the full screen to show the photos and let you flick through.

It really is fun to play with touch apps these days, and I get the feeling that we are still in the dark ages wrt our interaction models.

What patterns have you enjoyed in using and building mobile apps?

Remember the first dot-com boom when people started to think that creating a company was just about creating a website? They would register a domain before they had a business plan? Crazy days of sock puppets and and people renaming themselves as dot com versions. Behind the craziness was the reality that to a large extent you did have to have a website. A website was table stakes, and this continues to be the case.

With the iPhone another gold rush occurred. The initial rush is over and you can’t just throw an application into the appstore and hope to make a fortune. You actually have to create something compelling and differentiated to make hour mark. Imagine that!

The “quick buck” allure isn’t there and we are onto the next level of maturity. Having an experience for iPhone users has become table stakes for some too.

So, now you have created your website, and iPhone app, do you go to an Android app? Facebook? One for a new 3D TV? :) It is starting to get daunting.

Walking around at CES the consumer in me gets very excited to see the proliferation of devices. Everything is connected, everything has an API.

Then the developer in me wonders how this is going to scale and how we are going to deliver great experiences to all.

My hope is that the Web can become a great unifying platform. It isn’t easy to build Web apps that will just work anywhere. I just talked about the issues and fun of developing for devices and we are going to be getting a lot more form factors out there.

I have been having a great time hacking on code that let’s me do interesting things on my device. With webOS you get to use your old favorites (HTML/JS/CSS and Web APIs that you know) and now you are ready to take Web applications that you may have even already written, and make them work for the mobile form factor.

The two biggest challenges I have found so far on the design side have been dealing with the real estate available (screen size) and the touchy feely-ness of the device.

We have been trained on the desktop, thanks to the mouse as our pointer interface, to click lots of buttons. The tactile feedback that we get is the button looking depressed, but that is about it. The mouse has a lot going for it. The fact that I have a set of states due to the fact that there is a difference between having the cursor located somewhere, and having a click on that location is useful. It can be especially useful for discoverability. As a user mouses around you can unveil information “hey, if you click this button X will happen mate!” It also has the nice side effect that your hand can rest on it.

The infamous Minority Report interface looked great, and spawned many “look how we can do this now” prototypes, but try holding your hands in front of you for 10 minutes, let alone 8 hours :)

The mouse has a lot of drawbacks though. It is a level of abstraction. You aren’t touching the objects, you are touching a device that takes your input and does the real touching. This of course leads to touch screens and bypassing that abstraction. Watching my son tap and drag and manipulate a touch screen has been great to see. It matches the physical world a lot more. He tried to touch my laptop screen at first too.

Since you are mimicking the real world more though, you get into the uncanny valley problem. If you drag something…. it SHOULD drag! One of the interesting design choices that iPhone made was to allow you to drag things even though some folks would say “don’t let them do that, there is nothing back there”. You then saw studies where people were taking their browser screens and dragging them around just for fun. Crazy :)

Applications started to take advantage of this. iPhone started with many buttons. The back button is the classic one that is all over the shop. On webOS there is a gesture area that trains you quickly to swipe back whenever you want, like going back a page. It is fun to watch a webOS user try to swipe when using an iPhone. You get used to it, you like it, it feels more natural.

Applications like Tweetie 2 groked this too. They took away the refresh button and added a draggable notion. You drag the list down and let go to spin it into action. In many ways this is more work than a simple tap and is a touch less discoverable, but it has the benefit of “feeling nice” and once you know about it, there isn’t a darn button in front of you all the time. The UI can get out of the way.

The Newsroom feedreader app on webOS does a really nice job here too. You flip and drag and move throughout most of the interface. After awhile it feels like you have a large virtual space and you are just moving the view port around to see what you want to see. It feels nice. I strangely find that it makes me want to use the application much more than an app with buttons and simple taps. Maybe I am just strange.

I find that I really enjoy the horizontal scroll. I wanted to build a simple HorizontalScroller component for webOS, but first I will detail the low level side of using a Scroller webOS component to get this effect.

The code for all of this is in my HorizontalScroller GitHub repo.

A simple way to build a horizontal scroller is to have a div that contains all of the pieces of content (pre-loaded, or you can lazily load by adding nodes to this div). It extends wider than the 320px viewport size of the Pre/Pixi and contains other elements that are each 320px. We have a view that contains this, and a controller that sets it all up:

The View

Here is how we setup the component with 3 items:

<div x-mojo-element="Scroller" id="scrollerId">
    <div class="scrollerContainer">
        <div class="scrollerItem" id="scrollerItem:1">first</div>
        <div class="scrollerItem" id="scrollerItem:2">second</div>
        <div class="scrollerItem" id="scrollerItem:3">third</div>
    </div>
</div>
</div>

and we style them:

/* A scroller needs a container element */
.scrollerContainer {
    width:  960px;
    height: 400px;
    border: yellow solid 1px;
}
 
/* An item that is viewable in the scroll pane window */
.scrollerItem {
    float:     left;
    width:     320px;
    min-width: 320px;
}

The Controller

The code that turns the divs into life lives in the controller.

this.controller.setupWidget("scrollerId", {
                   mode: 'horizontal-snap'
               }, this.model = {
                   snapElements: { x: $$('.scrollerItem') },
                   snapIndex: 0
               }
        );

The key is the “horizontal-snap” mode that make the drag physics feel right and ties to the snapElements which are the DOM elements for the inner divs. The widget takes care of snapping to those elements for you!

Of course, you can use this base to do a lot. The afore mentioned Newsroom styles the divs so it looks like switching cards.

Next up: I will get the HorizontalScroller component done and dusted to make life even easier.

Conclusion

It is enjoyable to use the constraints of screen size and the natural feeling of touch and put them together to make great apps that live with you in your pocket. I thought screen size would be a huge issue, but in some ways it has been nice to have that constraint. It means you focus on the core functionality. Dealing with the touch vs. mouse issue is an interesting one too and will have you ending up with a very different interface if you embrace the feel.

What have you found enjoyable about using mobile apps vs. desktop ones?

As we watch and wait for the update to the Google Mobile iPhone application we are once again aware of the gatekeeper situation on that platform.

Google doesn’t know when it will launch, Apple does. Google had great PR into the launch (which they were told would be Friday… at least at some point). I had the pleasure to see the voice feature and was actually kinda gobsmacked with it when I saw it at work. It isn’t like we haven’t seen voice recognition apps for years. However, this one seemed to actually work, and not just for simple words but for complex queries. Random place names were picked up. Wow. Maybe the work behind GOOG-411 is paying off :)

But, the iPhone app isn’t out there, yet. If this was a Web application, the Google engineers could cut a release and be on their way. But, how would you read in the audio? You could go for Flash or a custom plugin, but it reminded me of the audio API support and Gears. When you think Audio API you think of the HTML 5 audio tag and the API that goes with it… specifically the “play” support. What interested me from the first design doc in Gears was the other side of things and the support for “record”:

// an object of this class can be got from 
// google.gears.factory.create('beta.audiorecorder')
AudioRecorder class
{
  // ---- error state ----
  readonly attribute AudioRecorderError error;
 
  // ---- recording state ----
  // says whether recorder is currently recording or not
  readonly attribute boolean recording;
  // says whether recorder is paused or not
  readonly attribute boolean paused;
  // the amount of sound detected by the microphone
  // 0 - no sound detected to 100 - maximum sound detected
  readonly attribute int activityLevel;
  // specifies the length (in milli seconds) of the audio recorded
  readonly attribute float duration;
 
  // number of channels, currently can be 1 (mono) or 2 (stereo)
           attribute int numberOfChannels;
  // sample rate for the recording
           attribute float sampleRate;
  // sample type for the recording, possible values need to be defined
  // signed 16 bit little endian linear PCM
  const unsigned short S16_LE = 0;
           attribute short sampleFormat;
  // audio file type (container and codec), possible values need to be defined
           attribute string type; 
 
  void record();
  void pause();
  void unpause();
  void stop();
 
  // ---- controls ----
  // 0.0 - silent to 1.0 - loudest
           attribute float volume;
           attribute boolean muted;
  // the amount of sound required to activate the microphone
  // 0 - capture even minutest sound to 100 - capture only loudest sound
           attribute int silenceLevel;
 
  // ---- cue ranges ----
  // provides ability to set callbacks at specific points in playback time.
  // similar to API in Audio class. Look at HTML5 spec for explanation.
  void addCueRange(in DOMString className, in float start, in float end, 
                  in boolean pauseOnExit,
                  in VoidCallback enterCallback, in VoidCallback exitCallback);
  void removeCueRanges(in DOMString className);
 
  // ---- access blob ----
  // returns handle to the blob object containing the audio data
  Blob getBlob();
};

I can’t wait to get full audio support available in the Open Web itself. Yet another barrier knocked down. Of course, the publicity around the Google Mobile app is nothing but good in many ways :)

We just announced two new ways to get location info from a browser client.

The Gears GeoLocation API is very detailed. It is able to use GPS, cell towers, WiFi, and ip addresses to work out the location, and you get an “accuracy” parameter to see what was available. As well as getting a position, you can watch a position so you are updated when a change happens. This is perfect for mobile devices that have Gears installed, and since the community is working on the W3C Geolocation spec it should be in many more places soon.

To go with the Gears API, we also have an API that goes along with the AJAX APIs, called ClientLocation.

This is an ip based geocoder that we have made available, and is very simple.

I put together a trivial example called Where Are You? that ties together this API with the Maps API:

You get access to the data from google.loader.ClientLocation, which is null if it can’t be calculated.

Here is a bit of JavaScript that ties it together:

google.load("maps", "2.x");
 
google.setOnLoadCallback(function() {
    if (google.loader.ClientLocation) {
        var cl = google.loader.ClientLocation;
        var location = [cl.address.city, cl.address.region, cl.address.country].join(', ');
 
        createMap(cl.latitude, cl.longitude, location);
    } else {
        document.getElementById('cantfindyou').innerHTML = "Crap, I don't know. Good hiding!";
    }
});
 
function createMap(lat, lng, location) {
    var mapElement = document.getElementById("map");
    mapElement.style.display = 'block';
    var map = new google.maps.Map2(mapElement);
    map.addControl(new GLargeMapControl());
    map.addControl(new GMapTypeControl());
    map.setCenter(new google.maps.LatLng(lat, lng), 13);
    map.openInfoWindow(map.getCenter(), document.createTextNode(location));
}

Russ Beattie has closed up shop for Mowser and people are rushing to declare the death of the mobile Web.

I like Russ, and was glad to see him back on the scene and blogging a storm, even if he can be a touch offensive from time to time ;)

But, just because he couldn’t find the right niche for Mowser, doesn’t mean the “mobile Web” is dead before born.

Take a look at what he really said:

In other words, I think anyone currently developing sites using XHTML-MP markup, no Javascript, geared towards cellular connections and two inch screens are simply wasting their time, and I’m tired of wasting my time.

I agree. Where are the mobile apps today? They are the iPhone specific ones, and a few stripped-down versions. The mobile Web is growing strong from where I sit. I just have to look around at how my own wife uses her laptop less and less, and her mobile browser more and more.

I am so bullish about the Web on the phone that I believe it will be THE platform for building mobile applications in the future.

If you are a hardcore mobile app builder you may snortle a little. Really? Cheesy Web technology can compete with rich application frameworks? Never.

They can, and they will. I was listening to someone talking about the battle of IPX versus NetBEUI. The viscous battle between Microsoft and Novell. This person said: “If you had told me that this TCP/IP thing would beat both of us I would have laughed in your face”. Some crappy thing uses in academia that doesn’t have all of the features that we do? In the battle of IPX and NetBEUI, TCP/IP won.

It will keep winning, and it will come to win in the mobile world. This is why I am excited about Gears for Mobile, and any other work that will come through in HTML 5 and browsers such as Mobile Safari.

It may take awhile, but would you really bet against it? The mobile Web will just be the Web. We will have limitations of course. 3G will take awhile, and the size of screens isn’t going anywhere until we have the dream of projection into your eyes and such.

Would you bet against it?

Paul Krill reported that Sun has looked at the iPhone SDK and thinks that it can port Java to the iPhone. It will be placed up on AppStore as an application, so I wonder what the user experience will be for apps that actually run Java, especially for the first time on a phone that doesn’t have it installed.

Don’t get me wrong, I want Java on the phone, just like I want full RubyCocoa, PyCocoa, CocoaJS, and any other language that you fancy.

Background Processes

So, the iPhone doesn’t allow background processes:

Only one iPhone application can run at a time, and third-party applications never run in the background. This means that when users switch to another application, answer the phone, or check their email, the application they were using quits.

This makes me worry about Java too. Startup has never been a good point of the JVM. If I flip to a Java app am I going to have to wait for the bugger to startup? Is there going to be a way to load one VM and keep it loaded (doesn’t seem like it).

It seems like this is just a guideline and not a firm issue:

I’m a programmer and I just tried it [using the iPhone SDK] and you can keep your app running in the background in the normal way ApolloIM and iFob do it. I.e. overriding applicationSuspend.

Even more worrying though is this part of the developer agreement:

3.3.2 An Application may not itself install or launch other executable code by any means, including without limitation through the use of a plug-in architecture, calling other frameworks, other APIs or otherwise. No interpreted code may be downloaded and used in an Application except for code that is interpreted and run by Apple’s Published APIs and built-in interpreter(s).

Although it has been suggested that this is to stop non AppStore code, it seems to go a lot further than that.

Open source and the iPhone

Also, Mark Pilgrim has written up thoughts on whether iPhone apps can be GPL:

Since all iPhone apps must be distributed through a third-party (Apple’s “App Store”), that would make Apple the “distributor.” Which would mean that Apple — acting as the distributor of GPL-licensed object code — must provide source code or a written offer to provide source code. It’s analogous to a Linux distribution — they distribute binaries of upstream GPL programs, so they need to host the source code as well.

Darren Waters sat down with Andy Rubin to take a look at an early version of Android:

The software stack, I was told, was Alpha, so not even Beta; but what I was shown gave a good indication that Android should be taken seriously by competitors like Windows Mobile and Symbian.

Google says they are driving the Android initiative because they want to see internet-style development on mobile platforms in the way that the openness of the web has given rise to Facebook and the Web 2.0 movement which should be able to migrate to the mobile phone.

Of course, coming in at the ground level of Android will give Google plenty of opportunity to tailor its own applications.

I got to spend some time with a few Windows Mobile devices this week. I found them incredibly hard to use. I felt like an old person using a device. I think that we forget what the iPhone has done for the mobile industry. it is just so easy to use. Going back to all of these magical keys and no touch screen is soooo painful and backwards. I can’t wait for my touchscreen-screen.

I wonder if 2008 will be the tipping point over in the US where we see more developers targeting mobile versus desktop. I am sure it is going to happen some day…. but when is that?

Kevin Lynch at Engage told us that he believes we will be developing for the mobile form factor and extracting desktop interfaces from there, rather than the other way around.

I am twittering the Engage event using hashtags.