Gestures on Android 1.6

Touch screens are a great way to interact with applications on mobile devices. With a touch screen, users can easily tap, drag, fling, or slide to quickly perform actions in their favorite applications. But it's not always that easy for developers. With Android, it's easy to recognize simple actions, like a swipe, but it's much more difficult to handle complicated gestures, which also require developers to write a lot of code. That's why we have decided to introduce a new gestures API in Android 1.6. This API, located in the new package android.gesture, lets you store, load, draw and recognize gestures. In this post I will show you how you can use the android.gesture API in your applications. Before going any further, you should download the source code of the examples.

Creating a gestures library

The Android 1.6 SDK comes with a new application pre-installed on the emulator, called Gestures Builder. You can use this application to create a set of pre-defined gestures for your own application. It also serves as an example of how to let the user define his own gestures in your applications. You can find the source code of Gestures Builders in the samples directory of Android 1.6. In our example we will use Gestures Builder to generate a set of gestures for us (make sure to create an AVD with an SD card image to use Gestures Builder.) The screenshot below shows what the application looks like after adding a few gestures:

As you can see, a gesture is always associated with a name. That name is very important because it identifies each gesture within your application. The names do not have to be unique. Actually it can be very useful to have several gestures with the same name to increase the precision of the recognition. Every time you add or edit a gesture in the Gestures Builder, a file is generated on the emulator's SD card, /sdcard/gestures. This file contains the description of all the gestures, and you will need to package it inside your application inside the resources directory, in /res/raw.

Loading the gestures library

Now that you have a set of pre-defined gestures, you must load it inside your application. This can be achieved in several ways but the easiest is to use the GestureLibraries class:

mLibrary = GestureLibraries.fromRawResource(this, R.raw.spells);
if (!mLibrary.load()) {
    finish();
}

In this example, the gesture library is loaded from the file /res/raw/spells. You can easily load libraries from other sources, like the SD card, which is very important if you want your application to be able to save the library; a library loaded from a raw resource is read-only and cannot be modified. The following diagram shows the structure of a library:

Recognizing gestures

To start recognizing gestures in your application, all you have to do is add a GestureOverlayView to your XML layout:

<android.gesture.GestureOverlayView
    android:id="@+id/gestures"
    android:layout_width="fill_parent" 
    android:layout_height="0dip"
    android:layout_weight="1.0" />

Notice that the GestureOverlayView is not part of the usual android.widget package. Therefore, you must use its fully qualified name. A gesture overlay acts as a simple drawing board on which the user can draw his gestures. You can tweak several visual properties, like the color and the width of the stroke used to draw gestures, and register various listeners to follow what the user is doing. The most commonly used listener is GestureOverlayView.OnGesturePerformedListener which fires whenever a user is done drawing a gesture:

GestureOverlayView gestures = (GestureOverlayView) findViewById(R.id.gestures);
gestures.addOnGesturePerformedListener(this);

When the listener fires, you can ask the GestureLibrary to try to recognize the gesture. In return, you will get a list of Prediction instances, each with a name - the same name you entered in the Gestures Builder - and a score. The list is sorted by descending scores; the higher the score, the more likely the associated gesture is the one the user intended to draw. The following code snippet demonstrates how to retrieve the name of the first prediction:

public void onGesturePerformed(GestureOverlayView overlay, Gesture gesture) {
    ArrayList predictions = mLibrary.recognize(gesture);

    // We want at least one prediction
    if (predictions.size() > 0) {
        Prediction prediction = predictions.get(0);
        // We want at least some confidence in the result
        if (prediction.score > 1.0) {
            // Show the spell
            Toast.makeText(this, prediction.name, Toast.LENGTH_SHORT).show();
        }
    }
}

In this example, the first prediction is taken into account only if it's score is greater than 1.0. The threshold you use is entirely up to you but know that scores lower than 1.0 are typically poor matches. And this is all the code you need to create a simple application that can recognize pre-defined gestures (see the source code of the project GesturesDemo):

Gestures overlay

In the example above, the GestureOverlayView was used as a normal view, embedded inside a LinearLayout. However, as its name suggests, it can also be used as an overlay on top of other views. This can be useful to recognize gestures in a game or just anywhere in the UI of an application. In the second example, called GesturesListDemo, we'll create an overlay on top of a list of contacts. We start again in Gestures Builder to create a new set of pre-defined gestures:

And here is what the XML layout looks like:

<android.gesture.GestureOverlayView
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/gestures"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    
    android:gestureStrokeType="multiple"
    android:eventsInterceptionEnabled="true"
    android:orientation="vertical">

    <ListView
        android:id="@android:id/list"  
        android:layout_width="fill_parent" 
        android:layout_height="fill_parent"  />

</android.gesture.GestureOverlayView>

In this application, the gestures view is an overlay on top of a regular ListView. The overlay also specifies a few properties that we did not need before:

• gestureStrokeType: indicates whether we want to recognize gestures made of a single stroke or multiple strokes. Since one of our gestures is the "+" symbol, we need multiple strokes
• eventsInterceptionEnabled: when set to true, this property tells the overlay to steal the events from its children as soon as it knows the user is really drawing a gesture. This is useful when there's a scrollable view under the overlay, to avoid scrolling the underlying child as the user draws his gesture
• orientation: indicates the scroll orientation of the views underneath. In this case the list scrolls vertically, which means that any horizontal gestures (like action_delete) can immediately be recognized as a gesture. Gestures that start with a vertical stroke must contain at least one horizontal component to be recognized. In other words, a simple vertical line cannot be recognized as a gesture since it would conflict with the list's scrolling.

The code used to load and set up the gestures library and overlay is exactly the same as before. The only difference is that we now check the name of the predictions to know what the user intended to do:

public void onGesturePerformed(GestureOverlayView overlay, Gesture gesture) {
    ArrayList<Prediction> predictions = mLibrary.recognize(gesture);
    if (predictions.size() > 0 && predictions.get(0).score > 1.0) {
        String action = predictions.get(0).name;
        if ("action_add".equals(action)) {
            Toast.makeText(this, "Adding a contact", Toast.LENGTH_SHORT).show();
        } else if ("action_delete".equals(action)) {
            Toast.makeText(this, "Removing a contact", Toast.LENGTH_SHORT).show();
        } else if ("action_refresh".equals(action)) {
            Toast.makeText(this, "Reloading contacts", Toast.LENGTH_SHORT).show();
        }
    }
}

The user is now able to draw his gestures on top of the list without interfering with the scrolling:

The overlay even gives visual clues as to whether the gesture is considered valid for recognition. In the case of a vertical overlay, for instance, a single vertical stroke cannot be recognized as a gesture and is therefore drawn with a translucent color:

It's your turn

Adding support for gestures in your application is easy and can be a valuable addition. The gestures API does not even have to be used to recognize complex shapes; it will work equally well to recognize simple swipes. We are very excited by the possibilities the gestures API offers, and we're eager to see what cool applications the community will create with it.