Using Pixel Bender with Flash Builder 4 as a number crunching engine

Additional Requirements

Pixel Bender Toolkit

• Pixel Bender Toolkit 2.0 for Macintosh (DMG, 77 MB)
• Pixel Bender Toolkit 2.0 for Windows (ZIP, 48.8 MB)

Note: It is possible to use Pixel Bender with Flex Builder or the Flex SDK; however, the examples in this tutorial are built with Flash Builder 4.

Featured in Flash Player 10, Adobe Pixel Bender technology was designed to manipulate pixels, but it can also be used as a multi-threaded number crunching engine. You can pass in a list of numbers, and have Pixel Bender perform complex mathematical operations and then return a list of results.

Why would you want to use Pixel Bender for calculations? The short answer is that it improves performance. ActionScript runs as a single thread, so while Flash Player is processing information, it cannot run another thread to do something else. If you need to do some heavy lifting—such as a long series of complex calculations—Flash Player may appear to be stuck until the calculation is complete. With Pixel Bender you can run 32-bit floating point calculations on a separate thread (on another processor core if available) and process the results when they are complete, leaving the main thread to continue unimpeded.

In this article, I will discuss how to use Pixel Bender for calculations and show you how to build a real-life number crunching application with Pixel Bender.

Pixel Bender was designed for image processing algorithms (including filters and effects) and its syntax is based on the OpenGL Shading Language (GLSL).

The easiest way to create a multi-threaded number crunching ActionScript application is with the Adobe Pixel Bender Toolkit. The Pixel Bender Toolkit lets you create Pixel Bender kernel files (.pbk) and graph description files (.pbg), which are supported directly in After Effects CS4 and the Photoshop Pixel Bender extension.

This is how it works. You use Pixel Bender Toolkit to create a kernel, which performs an operation on individual pixels. (Depending on the context, you may hear kernels referred to as filters or shaders.) You then use Pixel Bender Toolkit to export the kernel file as compiled bytecode and store it as PBJ (.pbj) file. Once you have the bytecode, you can load it or embed it in a Shader object in applications that are compiled for Flash Player 10 (see Figure 1).

Additionally, it is possible to skip this whole workflow and create bytecode without the Pixel Bender Toolkit, which I will cover later in Creating a kernel from assembler code.

Using Pixel Bender you can perform simple arithmetic operations such as add, subtract, multiply, and divide as well as complex functions such as sine, cosine, and so on. You cannot, however, implement loops in a Pixel Bender kernel.

In the following example, you'll create a simple kernel that multiplies two numbers and returns the result. You'll then create an application that uses the kernel to multiply a list of numbers.

Create the kernel

Follow these steps to create the kernel:

1. If you have not already done so, download and install the Pixel Bender Toolkit from the beginning of the article.
2. Open the Pixel Bender Toolkit and click Create A New Kernel.
3. Paste the following code into the source editor:

The kernel code consists of three parts: kernel metadata, declarations, and functions.

Kernel metadata

The metadata contains:

• language-version element - This is a required element; the current version is 1.0.
• name - The name of the kernel, in this case it is "SimpleCalculator".
• Namespace – This provides a way for an author to classify kernels.
• Vendor - The name of the company or individual who wrote the kernel. You can use your name here.
• Description – A description of the kernel; this is the only metadata that is optional.
• Version – An integer version number; start with 1.

Declarations

The second part of the kernel defines the inputs and output. Though this kernel does not use them, the declarations can also include parameters, dependent variables, and constants to be used in the functions. You can also import function libraries if needed.

In this example, you will be using two one-channel images as input, named src and src2.

The output is defined as pixel3, a Pixel Bender data type that holds a three-channel pixel.

Pixel Bender supports images and pixels with up to four channels, typically describing red, green, blue, and alpha channels. The data types image1, image2, image3, and image4 describe a one, two, three, and four channel image respectively. Similarly, pixel1, pixel2, pixel3, and pixel4 describe one, two, three, and four channel pixels respectively.

You may wonder why the code uses a three-channel result when only one channel is needed. The Pixel Bender Toolkit requires the output to be a three-channel type. If it is not, you get the following error message when trying to compile: "ERROR: (line 23): 'result' : cannot have 1 or 2 channel outputs".

Declaring functions

The next part of the Kernel contains function definitions. Each kernel must contain an evaluatePixel() function definition, but you can also define other functions such as evaluateDependents(), needed(), and changed(). See the Pixel Bender Reference for more details.

Note: To access the Pixel Bender Reference, open Pixel Bender Toolkit and choose Help > Pixel Bender Language specification.

In this case, the code extracts the multiplier and multiplicand from the two image inputs, performs multiplication, and then returns the result as a pixel3:

You may want to see the kernel in action on actual images. If you click Run, you will see an alert that tells you that you need to load images (see Figure 2).

To run the kernel, follow these steps:

1. Choose File > Load image 1 and select an image file on your local machine.
2. Choose File > Load image 2 and select a second image file on your local machine.
3. Click Run.

You'll see a mash-up the two images in which every pixel is the result of multiplying corresponding pixels from the two images you selected.

To use a kernel with Flash Player, you must first export it to a bytecode program, stored in a binary file with the extension .pbj.

To create the PBJ file, choose File > Export Filter for Flash Player (see Figure 3). Name the file SimpleCalculator.pbj.

Create the Flex Application

Now that you have the PBJ file, you can create an application that will take a list of numbers and apply the simple calculation z=x*y to produce a list of results. Take a look at the application below:

Compile the code above using Flash Builder 4. When you run the application, you will see two lists of numbers. Click the "=" button to have your Pixel Bender kernel calculate the results (see Figure 4).

Now that you've seen what it does, examine the code in more detail. The first key step is to embed the PBJ file in your application just as you would embed any other assets in Flex.

The Embed tag instructs the ActionScript compiler to embed the Pixel Bender kernel when it creates the SWF file. It is used with a variable definition of type Class. In this case,

KernelClass will be an empty class that you will be using for the Shader class.

The result ByteArray will hold the results that the Shader returns.

The initializeHandler() function is called to start the calculation sequence. It takes as input the two lists that will be used to perform the multiplication:

Each list of numbers will need to be sent as a ByteArray, because the Pixel Bender kernel operates on 1-pixel tall images provided as input. These are defined as byteArray and byteArray2:

You will need a Shader object to execute the kernel on each of the pixels in an image (or in this case, a list of numbers instead of an image).

The convertListToByteArray() method converts a list of numbers into a ByteArray:

Next, the code sets up all the data that is need for the shader job:

Once everything is set, you create a ShaderJob object, add an event listener to handle the results, and start the job.

Note that the width of the image is the length of the byteArray divided by 4, since a float number stores data in four bytes. When the job is completed, onComplete()converts the results back into a list of numbers:

Here is convertListToByteArray(), a utility method for converting a list into a ByteArray:

The last part of the code is the UI, which contains two lists of numbers, a button to start the calculation, and a list to hold and display the results:

The Pixel Bender kernel language also includes functions for more complex number crunching, including:

• sin(x) - Returns the sine of x.
• cos(x) – Returns cosine of x.
• tan(x) – Returns the tangent of x.
• asin(x) – Returns the arcsine (inverse sine) of x.
• acos(x) – Returns the arccosine (inverse cosine) of x.
• atan(x) – Returns the arctangent (inverse tangent) of x.
• exp(x) - Returns ex.
• log(x) – Returns the natural logarithm of x.
• pow(x, y) - Returns xy.
• sqrt(x) – Returns the positive square root of x.

For the second example, you will create and use a kernel that uses cos().

Create the kernel

Following the same procedure you used to create SimpleCalculator.pbj in the previous section, use the code below to create CosCalculator.pbj.

Create the Flex Application

Create a new Flex application using the code below. This code is very similar to the previous example; the main difference is that it passes one list of numbers to the kernel instead of two.

When you have built the application, run it in your browser and click Calculate to see the results (see Figure 5).

Note: The input to cos() is provided in radians.

The next example is a little more practical. The application you will create is a track mixer, which will take two audio tracks, mix them together, and return a result that is one-track mash-up.

The mixing kernel

The kernel uses the Pixel Bender mix(x, y, a) function, which returns a linear interpolation between x and y (specifically, it returns x * (1.0 - a) +y * a).

Here is the kernel code:

Notice that this code has a parameter, distort, which can be used to adjust the level for each track during runtime.

The mixing API

You can download the TrackMixer class that implements an API for mixing two audio files using Pixel Bender from Google code.

The class is relatively simple. It starts by defining the variables it will use:

The start() method loads the tracks that will be mixed.

After loading the tracks, the next step is to set up the ShaderJob object and parameters, and then start the job.

The mixing application

The next step is to create an application that uses the mixing API (see Figure 6).

The application embeds the Pixel Bender kernel and passes it to the TrackMixer. It then starts the TrackMixer, passing it URLs to the two tracks. When the slider moves, the code updates the mixer balance. The full code is below:

So far, you've seen how easy it is to create PBJ files using the Pixel Bender Toolkit. It is possible, however, to create PBJ files directly from assembler code.

Note: To use this technique you will need to know how to write assembler code. Further, the technique relies on unsupported tools and an undocumented syntax. See this post for more details.

Why would you want to do this? The Pixel Bender Toolkit comes with some limitations. For instance, in the music mixer application, you can only mix two tracks, and you may want to mix six tracks.

When you try to add a third image, you get the following error: "This version of Adobe Pixel Bender Toolkit does not support kernel with more than 2 inputs". Additionally, you may want to have more control over the code and ensure that it is optimized for maximum performance.

Tinix Uro put together an unofficial assembler and disassembler that allow you to write assembler code and create a PBJ file. Please note that the code is not officially supported by Adobe.

You can download the C++ code from here:

http://www.kaourantin.net/source/pbjtools/apbj.cpp

http://www.kaourantin.net/source/pbjtools/dpbj.cpp

On Windows, you can download pre-built binaries of these two command line tools:

http://www.kaourantin.net/source/pbjtools/apbj.zip

http://www.kaourantin.net/source/pbjtools/dpbj.zip

On Mac OS X, you will need to compile the C++ code yourself. Type the following commands to create the binaries:

Note: You need the Xcode suite installed on your Mac in order to use g++, which is in the Gnu Compiler Collection.

Using the apbj tool, you can create a PBJ file from assembler code. For example:

And using the dpbj tool you can create assembler from a PBJ file:

To create a PBJ file using assembler, you first need to understand its structure.

I used dpbj to disassemble TwoTracksMixer.pbj, the PBJ file use in the mixing example. Here is the resulting assembler code:

The first portion of the code is the kernel definition; just as with the Pixel Bender Toolkit you need to create the metadata:

The next part defines the inputs, outputs, and parameters.

The texture lines define the source images:

A kernel can take any number of parameters of any type. When you define parameters, you supply metadata, such as minimum, maximum, and default values. You can store several parameters in the same register by using different channels. For instance, the _OutCoord parameter stores the (X,Y) pixel position in the R and G channels of register 0. Other parameters might use the B and A channel:

Each parameter takes three arguments:

• The type of the parameter;
• A register, which indicates where the parameter will be accessible
• In or Out, which identifies the parameter as an input or an output

There are four metadata values that you can set: minValue, maxValue, defaultValue, and description. For example:

The last part of the code includes the instructions that are used to perform the operations. For example, mov is used to move a value into a register, mul is used to multiply values, and sub is used to subtract values.

If you are handy with assembler code, you can fine-tune your code, and then create a new, improved PBJ file using the apbj tool. This capability opens new opportunities for using Pixel Bender for more complex calculations.

Where to go from here

This article has provided a brief introduction to the number crunching capabilities of Pixel Bender.

To learn more, see the following resources:

Pixel Bender Technology Center

Pixel Bender basics for Flash

A Pixel Bender assembler that uses haXe

James Ward's ActionScript 3.0 Pixel Bender Shader Library