Adobe

Products

Acrobat

Creative Cloud

Creative Suite

Digital Marketing Suite

Digital Publishing Suite

Elements

Photoshop

Touch Apps

Student and Teacher Editions

More products

Solutions

Creative tools for business

Digital marketing

Digital media

Education

Financial services

Government

Web Experience Management

Sylvester's Band: A GPU-accelerated, interactive children's book for tablets and smartphones

by Morten Sandholt

Uncle Handsalt

Created

3 October 2011

Requirements

Prerequisite knowledge

Familiarity with basic ActionScript 3 programming concepts.

Additional required other products

Box2DFlash

User level

Beginning

Required products

Sample files

Winner of the Best Lifestyle & Community App in the Adobe AIR App Challenge, Sponsored by Sony, Sylvester's Band is an interactive children's book that I released on the Apple App Store, Android Market, and BlackBerry App World in July 2011.

My goal with Sylvester's Band was to create a beautiful storybook app with high-quality illustrations and script, original music, professional voiceover, and lots of interaction on every page that runs, sounds, and looks great on as many devices and platforms as possible (see Figure 1).

Figure 1. The opening screen for Sylvester's Band.

I chose Adobe AIR as my development platform because of the promise that I'd have to write the code only once and then deploy it to multiple platforms.

This article describes the main technical challenges in developing Sylvester's Band and how I overcame them.

Creating an interactive world

I wanted to set Sylvester's Band apart from other storybook apps by offering more and deeper interaction with the world and characters on the screen.

Box2D physics

To make the illustrations come alive, I made heavy use of the Box2DFlash physics library. Box2D lets you make physical bodies move around, collide, and react to touch in a realistic manner. It also lets you connect bodies with joints (think of hinges) to build almost any contraption.

As shown in Figure 2, using Box2D, I could give the impression that the animals are balancing on top of each other, catching the moon with a rope.

Figure 2. Box2D bodies are represented by pink circles; joints are bright blue.

Multitouch

Another key goal for me was to allow a parent and a child (or a brother and a sister) to interact with Sylvester's Band at the same time. So I decided to support multitouch for all interactions.

With regular mouse interaction, you only have to keep track of the mouseX and mouseY positions and listen for mouse click/up/down events. With multitouch, you need to keep track of multiple touch points simultaneously.

I built my own system for this that lets me access touchpoint data anywhere in the code:

        for each (touchPoint:TouchPoint in TouchHandler.points) //Looping through the touch points
{
    if (touchPoint.active) //Check if touch point is active, i.e. finger is on the screen
    {
        trace(touchPoint.id); //A unique ID, so you can keep track of each touch point
        trace(touchPoint.x); //x position
        trace(touchPoint.y); //y position
        trace(touchPoint.spd); //The speed at which the touch point is moving
        trace(touchPoint.angle); //The angle in radians at which the touch point is moving
    }
}

    

One of the challenges of multitouch is to make sure that things don't break when multiple touchpoints interact with the same object. So I established some rules, such as only being able to interact with one Box2D body per touchpoint (but allowing several touchpoints to grab the same Box2D body) and only being able to drag one fish/leaf/whatever per touchpoint.

Accelerometer

The accelerometer is used to change the gravity throughout the app. This is a fun concept that even small children understand: Turn the device on the side and everything on the screen reacts accordingly.

At every frame, I calculate the gravity vector based on the accelerometer angle and apply it to the Box2D physics simulation:

        //Getting the angle from the accelerometer and adding 90 degrees because we are in landscape mode
var angle:Number = Math.atan2(Accel.incX, Accel.incY) + Math.PI * 0.5;
//We want gravity to pull roughly like in the real world, so we multiply by 9.81 on both axes
B2D.setGravity(Math.cos(angle) * 9.81, Math.sin(angle) * 9.81);

    

Developing for multiple screen sizes

There are generally two ways to go about supporting multiple screens:

Scale the whole application, keeping the width/height ratio constant, and accept that some devices will have black bars either at the sides or at the top and bottom. This is the quick-and-dirty solution.
Develop a custom solution that makes your content fill the screen completely, regardless of resolution and width/height ratio. This involves much more work but makes your content look great on all screens.

In Sylvester's Band I went for the latter.

Global ratio variables

My goal was for content to scale intelligently to all resolutions and screen ratios, from 480 × 320 (3:2) on the original iPhone to 1280 × 800 (16:10) and beyond on the new Android tablets. To achieve this, I calculated two variables that are used to scale and lay out content throughout the app: ratioContent and ratioFullScreen . Both are 1 (100%) at our default target resolution of 1024 × 768 (iPad).

Sylvester's Band has a center vignette that by default is 768 × 512 (see Figure 3).

Figure 3. The center vignette contains the illustrations on each page.

The ratioContent variable is used for scaling the center vignette and laying out its contents. Depending on screen height, it is calculated in three different ways:

        if (Screen.hght < 512) //Screen height smaller than default center vignette height of 512
{
    Screen.ratioContent = Screen.hght / 512; //Vignette height is always 100% of screen height
}
else if (Screen.wdth == 960 && Screen.hght == 640) //For iPhone Retina Display
{
    Screen.ratioContent = 1.25; //Scaled up because of higher pixel density
}
else if (Screen.hght > 768) //Screen height bigger than target device height of 768
{
    Screen.ratioContent = Screen.hght / 768; //Same as ratioFullScreen 
}

    

Figures 4–6 show Sylvester's Band at three different resolutions.

Sylvester's Band at our target resolution of 1024 x 768

Figure 4. Sylvester's Band at our target resolution of 1024 × 768 (iPad).

Figure 5. Sylvester's Band at 854 × 480; the story text is now overlaid and can be swiped away (Android phones).

Figure 6. Sylvester's Band at 480 × 320; the center vignette fills the whole screen (original iPhone).

The variable ratioFullScreen simply changes proportionally to the height of the device screen:

Screen.ratioFullScreen = Screen.hght / 768;

I use ratioFullScreen for things like scaling and laying out the main menu, placing navigational buttons and generating the black vignette at the edge of the screen—in other words, everything that is not in the center vignette.

Scale

The two ratio variables are used to scale the size of individual objects on the screen. See more about scaling in the "Achieving great performance" section.

Position

The key to laying out your content on multiple screens is to think of all positions as relative to screen width and height. So instead of placing a menu button at an absolute x/y position, I placed it like this:

        button.x = Screen.wdth * 0.38; //38% of screen width
button.y = Screen.hght * 0.48; //48% of screen height

The same thing goes for content inside the center vignette. The position and dimensions of the center vignette are defined based on the ratioContent variable:

        Vignette.wdth = 768 * Screen.ratioContent;
Vignette.hght = 512 * Screen.ratioContent;
Vignette.x = Screen.wdth * 0.5 - Vignette.wdth * 0.5;
Vignette.y = (Screen.hght – Vignette.hght) * 0.25;

    

The following code places a lantern inside the center vignette:

        lantern.x = Vignette.x + Vignette.wdth * 0.299; //29.9% of center vignette width
lantern.y = Vignette.y + Vignette.hght * 0.265; //26.5% of center vignette height

Velocity

Velocities also have to scale so that animated objects move with the same relative speed on all screen sizes—in this case, the velocity of a bird flying inside the center vignette:

        bird.speed = 18 * Screen.ratioContent; //18 pixels per frame, scaled by ratioContent
    

Box2D physics simulation

The Box2D simulation runs underneath my own code, but internally measures everything in meters. So I needed to define how many screen pixels represent a meter in the Box2D simulation to do the conversion throughout my code:

        B2D.pixelsPerMeter = 75 * Screen.ratioContent; //75 pixels per meter, scaled by ratioContent
    

If I didn't scale by ratioContent , the Box2D bodies would have different mass properties at different resolutions and would therefore behave inconsistently on different devices.

Story text

At smaller resolutions, there isn't always room for the story text underneath the center vignette. In this case, I placed the text on a black frame that users can swipe away when they are done reading and want to interact with the illustrations (see Figures 4 and 5).

Achieving great performance

It was important for me to have smooth animations and a responsive app in general, so I set my target frame rate at 30 frames per second. One way to achieve this on mobile devices is to take advantage of the dedicated graphics processing unit in GPU rendering mode.

GPU + Bitmaps = win

The GPU is especially fast at working with Bitmap objects, so I made the decision early on that all objects on the display list should be of the Bitmap class. This was mostly done by exporting everything as PNG files in Adobe Photoshop, but as you can imagine there were a few exceptions that needed workarounds.

Everything that isn't a bitmap—such as gradient skies (Shape) and the story text that is displayed on each page (TextField)—is first created normally and then drawn into a Bitmap object and finally disposed of:

        var textField:TextField = new TextField(); //Create new TextField
textField.text = 'I hope William the wolf is home'; //Set text
var bitmapData:BitmapData = new BitmapData(textField.width, textField.height, true, 0x000000); //Create BitmapData with width and height of TextField 
bitmapData.draw(tf); //Draw TextField into the BitmapData
var bitmap:Bitmap = new Bitmap(bitmapData); //Create new Bitmap with the BitmapData we just created
textfield = null; //No need for the TextField anymore

    

Large graphics, like backgrounds or parallax layers, are chopped into smaller GPU-friendly 256 ´ 256 bitmaps and laid out so the user doesn't notice the difference (see Figure 7).

Figure 7. While the sky is chopped into 256 ´ 256 Bitmap objects (green) and so are the four parallax layers (yellow), buttons and characters are normal Bitmap objects (red).

Pre-scale everything

Scaling Bitmap objects with the scaleX and scaleY properties does the job, but it potentially uses more GPU memory and also causes the GPU to do extra calculations when transforming the Bitmap object every frame, resulting in worse performance.

The solution is to "pre-scale" the bitmaps by recreating them at their desired resolution before displaying them. This is done by taking the source Bitmap object and copying a scaled version of it into a new Bitmap object to be displayed:

        var bitmapBig:Bitmap = new Bitmap(bitmapDataBig); //Create big Bitmap with the BitmapData we want to scale 
var matrix:Matrix = new Matrix(); //We are going to need a matrix to scale the big Bitmap 
matrix.scale(Screen.ratioContent, Screen.ratioContent); //Scale the matrix by the ratio in question, in this case the ratioContent
var bitmapDataSmall:BitmapData = new BitmapData(bitmapDataBig.width * Screen.ratioContent, bitmapDataBig.height * Screen.ratioContent, true, 0x000000); //Create the new small BitmapData
bitmapDataSmall.draw(bitmapBig, matrix); //Draw the big Bitmap into the small BitmapData by applying the scaled down matrix 
var bitmapSmall:Bitmap = new Bitmap(bitmapDataSmall); //Create small Bitmap using the small BitmapData

    

Many other performance optimizations were done, but turning all display objects into pre-scaled bitmaps was the primary optimization that enabled me to achieve good performance across all devices.

Exploring the framework files

The sylvester_framework.zip file linked to at the top of this article contains a sample project that shows off the concepts I discussed:

Unpack the archive and put the SylvesterFrameWork folder on the desktop or another easily accessible location on your hard drive.
Open the FLA file in Flash Professional CS5 and compile it. Try changing the size property and recompile it to understand how the application scales to different resolutions.
Look through the individual AS files to understand the different parts of the framework. You can package the SWF with Adobe AIR to run on mobile devices.

Where to go from here

Below are links to some great resources that helped me during the development of Sylvester's Band:

Optimizing for the Adobe Flash Platform: Essential reading for anyone who wants to develop Flash content that performs well.
Box2D 2.1a Tutorial: Great Box2D tutorial with Flash examples.
Box2D video tutorials: Video tutorial walking you through the creation of a game using on Box2D.
Developing AIR applications for mobile devices: Everything you need to know about packaging applications with Adobe AIR.

Be sure to check out Sylvester's Band itself. On my website you can read more about the app, watch the trailer, and find links to the app stores.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License. Permissions beyond the scope of this license, pertaining to the examples of code included within this work are available at Adobe.