Requirements
     
Prerequisite knowledge
Required products    

To build the examples described in this tutorial you need to know how to compile Flex applications using Flex Builder or the Flex SDK.

Flash Player 10

Flash Builder 4  (Download trial)

   
User level      
Intermediate      

 
Additional Requirements

 
Pixel Bender Toolkit
Note: It is possible to use Pixel Bender with Flex Builder or the Flex SDK; however, the examples in this tutorial are built with Flash Builder 4.
 
Featured in Flash Player 10, Adobe Pixel Bender technology was designed to manipulate pixels, but it can also be used as a multi-threaded number crunching engine. You can pass in a list of numbers, and have Pixel Bender perform complex mathematical operations and then return a list of results.
 
Why would you want to use Pixel Bender for calculations? The short answer is that it improves performance. ActionScript runs as a single thread, so while Flash Player is processing information, it cannot run another thread to do something else. If you need to do some heavy lifting—such as a long series of complex calculations—Flash Player may appear to be stuck until the calculation is complete. With Pixel Bender you can run 32-bit floating point calculations on a separate thread (on another processor core if available) and process the results when they are complete, leaving the main thread to continue unimpeded.
 
In this article, I will discuss how to use Pixel Bender for calculations and show you how to build a real-life number crunching application with Pixel Bender.
 

 
The Pixel Bender Toolkit

Pixel Bender was designed for image processing algorithms (including filters and effects) and its syntax is based on the OpenGL Shading Language (GLSL).
 
The easiest way to create a multi-threaded number crunching ActionScript application is with the Adobe Pixel Bender Toolkit. The Pixel Bender Toolkit lets you create Pixel Bender kernel files (.pbk) and graph description files (.pbg), which are supported directly in After Effects CS4 and the Photoshop Pixel Bender extension.
 
This is how it works. You use Pixel Bender Toolkit to create a kernel, which performs an operation on individual pixels. (Depending on the context, you may hear kernels referred to as filters or shaders.) You then use Pixel Bender Toolkit to export the kernel file as compiled bytecode and store it as PBJ (.pbj) file. Once you have the bytecode, you can load it or embed it in a Shader object in applications that are compiled for Flash Player 10 (see Figure 1).
 
Pixel Bender workflow using the Pixel Bender Toolkit
Figure 1. Pixel Bender workflow using the Pixel Bender Toolkit
 
Additionally, it is possible to skip this whole workflow and create bytecode without the Pixel Bender Toolkit, which I will cover later in Creating a kernel from assembler code.
 

 
Performing basic operations

Using Pixel Bender you can perform simple arithmetic operations such as add, subtract, multiply, and divide as well as complex functions such as sine, cosine, and so on. You cannot, however, implement loops in a Pixel Bender kernel.
 
In the following example, you'll create a simple kernel that multiplies two numbers and returns the result. You'll then create an application that uses the kernel to multiply a list of numbers.
 
 
Create the kernel
Follow these steps to create the kernel:
 
  1. If you have not already done so, download and install the Pixel Bender Toolkit from the beginning of the article.
  2. Open the Pixel Bender Toolkit and click Create A New Kernel.
  3. Paste the following code into the source editor:
<languageVersion : 1.0;> kernel SimpleCalculator < namespace : "PixelBender"; vendor : "Elrom LLC"; version : 1; description : "Simple calculator"; > { input image1 src; input image1 src2; output pixel3 result; void evaluatePixel() { float x = pixel1( sample(src, outCoord()) ); float y = pixel1( sample(src2, outCoord()) ); float z = x*y; result = pixel3( z, 0.0, 0.0); } }
The kernel code consists of three parts: kernel metadata, declarations, and functions.
 
 
Kernel metadata
The metadata contains:
 
  • language-version element - This is a required element; the current version is 1.0.
  • name - The name of the kernel, in this case it is "SimpleCalculator".
  • Namespace – This provides a way for an author to classify kernels.
  • Vendor - The name of the company or individual who wrote the kernel. You can use your name here.
  • Description – A description of the kernel; this is the only metadata that is optional.
  • Version – An integer version number; start with 1.
<languageVersion : 1.0;> kernel SimpleCalculator < namespace : "PixelBender"; vendor : "Elrom LLC"; version : 1; description : "Simple calculator"; >
 
Declarations
The second part of the kernel defines the inputs and output. Though this kernel does not use them, the declarations can also include parameters, dependent variables, and constants to be used in the functions. You can also import function libraries if needed.
 
In this example, you will be using two one-channel images as input, named src and src2.
 
input image1 src; input image1 src2;
The output is defined as pixel3, a Pixel Bender data type that holds a three-channel pixel.
 
output pixel3 result;
Pixel Bender supports images and pixels with up to four channels, typically describing red, green, blue, and alpha channels. The data types image1, image2, image3, and image4 describe a one, two, three, and four channel image respectively. Similarly, pixel1, pixel2, pixel3, and pixel4 describe one, two, three, and four channel pixels respectively.
 
You may wonder why the code uses a three-channel result when only one channel is needed. The Pixel Bender Toolkit requires the output to be a three-channel type. If it is not, you get the following error message when trying to compile: "ERROR: (line 23): 'result' : cannot have 1 or 2 channel outputs".
 
 
Declaring functions
The next part of the Kernel contains function definitions. Each kernel must contain an evaluatePixel() function definition, but you can also define other functions such as evaluateDependents(), needed(), and changed(). See the Pixel Bender Reference for more details.
 
Note: To access the Pixel Bender Reference, open Pixel Bender Toolkit and choose Help > Pixel Bender Language specification.
 
In this case, the code extracts the multiplier and multiplicand from the two image inputs, performs multiplication, and then returns the result as a pixel3:
 
void evaluatePixel() { float x = pixel1( sample(src, outCoord()) ); float y = pixel1( sample(src2, outCoord()) ); float z = x*y; result = pixel3( z, 0.0, 0.0); }
You may want to see the kernel in action on actual images. If you click Run, you will see an alert that tells you that you need to load images (see Figure 2).
 
To run the kernel, follow these steps:
 
  1. Choose File > Load image 1 and select an image file on your local machine.
  2. Choose File > Load image 2 and select a second image file on your local machine.
  3. Click Run.
You'll see a mash-up the two images in which every pixel is the result of multiplying corresponding pixels from the two images you selected.
 
The Pixel Bender Toolkit load images alert
Figure 2. The Pixel Bender Toolkit load images alert
 
To use a kernel with Flash Player, you must first export it to a bytecode program, stored in a binary file with the extension .pbj.
 
To create the PBJ file, choose File > Export Filter for Flash Player (see Figure 3). Name the file SimpleCalculator.pbj.
 
Export filter for Flash Player menu item
Figure 3. Export filter for Flash Player menu item
 
Create the Flex Application
Now that you have the PBJ file, you can create an application that will take a list of numbers and apply the simple calculation z=x*y to produce a list of results. Take a look at the application below:
 
<?xml version="1.0" encoding="utf-8"?> <s:Application xmlns:fx="http://ns.adobe.com/mxml/2009" xmlns:s="library://ns.adobe.com/flex/spark" xmlns:mx="library://ns.adobe.com/flex/mx" minWidth="1024" minHeight="768"> <fx:Script> <![CDATA[ import mx.collections.IList; [Embed(source="SimpleCalculator.pbj", mimeType="application/octet-stream")] private var KernelClass:Class; private var result:ByteArray; protected function initializeHandler(list:IList, list2:IList):void { var byteArray:ByteArray = new ByteArray(); var byteArray2:ByteArray = new ByteArray(); var shader:Shader = new Shader(new KernelClass()); var shaderJob:ShaderJob; var height:int; var width:int; byteArray = convertListToByteArray( list ); byteArray2 = convertListToByteArray( list2 ); width = byteArray.length >> 2; height = 1; shader.data.src.width = width; shader.data.src.height = height; shader.data.src.input = byteArray; shader.data.src2.width = width; shader.data.src2.height = height; shader.data.src2.input = byteArray2; result = new ByteArray(); result.endian = Endian.LITTLE_ENDIAN; shaderJob = new ShaderJob(shader, result, width, height); shaderJob.addEventListener(Event.COMPLETE, onComplete); shaderJob.start(); } protected function onComplete(event:Event):void { var length:int = result.length; var num:Number; var i:int; result.position = 0; for(i = 0; i < length; i += 4) { num = result.readFloat(); if(i % 3 == 0) listDest.dataProvider.addItem( num ); } } /** * Static method to convert the list * object into byte array object. * * @param array * @return * */ private static function convertListToByteArray(list:IList):ByteArray { var retVal:ByteArray = new ByteArray(); var number:Number; var len:int = list.length; var i:int; retVal.endian = Endian.LITTLE_ENDIAN; for (i; i<len; i++) { retVal.writeFloat( Number(list[i]) ); } retVal.position = 0; return retVal; } ]]> </fx:Script> <s:List id="listSrc" width="57" height="158"> <s:ArrayCollection> <fx:Object>1</fx:Object> <fx:Object>2</fx:Object> <fx:Object>3</fx:Object> <fx:Object>4</fx:Object> <fx:Object>5</fx:Object> <fx:Object>6</fx:Object> <fx:Object>7</fx:Object> </s:ArrayCollection> </s:List> <s:List id="listSrc2" width="57" height="158" x="122" y="0"> <s:ArrayCollection> <fx:Object>2</fx:Object> <fx:Object>5</fx:Object> <fx:Object>2</fx:Object> <fx:Object>3</fx:Object> <fx:Object>4</fx:Object> <fx:Object>5</fx:Object> <fx:Object>6</fx:Object> </s:ArrayCollection> </s:List> <s:Button x="210" y="56" label="=" click="initializeHandler(listSrc.dataProvider, listSrc2.dataProvider)"/> <s:List id="listDest" width="110" height="158" x="314" y="-1"> <s:ArrayCollection /> </s:List> <s:RichText x="88" y="61" text="X"/> </s:Application>
Compile the code above using Flash Builder 4. When you run the application, you will see two lists of numbers. Click the "=" button to have your Pixel Bender kernel calculate the results (see Figure 4).
 
The simple calculator example running in a browser
Figure 4. The simple calculator example running in a browser
 
Now that you've seen what it does, examine the code in more detail. The first key step is to embed the PBJ file in your application just as you would embed any other assets in Flex.
 
[Embed(source="SimpleCalculator.pbj", mimeType="application/octet-stream")]
The Embed tag instructs the ActionScript compiler to embed the Pixel Bender kernel when it creates the SWF file. It is used with a variable definition of type Class. In this case,
 
KernelClass will be an empty class that you will be using for the Shader class.
 
private var KernelClass:Class;
The result ByteArray will hold the results that the Shader returns.
 
private var result:ByteArray;
The initializeHandler() function is called to start the calculation sequence. It takes as input the two lists that will be used to perform the multiplication:
 
protected function initializeHandler(list:IList, list2:IList):void
Each list of numbers will need to be sent as a ByteArray, because the Pixel Bender kernel operates on 1-pixel tall images provided as input. These are defined as byteArray and byteArray2:
 
var byteArray:ByteArray = new ByteArray(); var byteArray2:ByteArray = new ByteArray();
You will need a Shader object to execute the kernel on each of the pixels in an image (or in this case, a list of numbers instead of an image).
 
var shader:Shader = new Shader(new KernelClass()); var shaderJob:ShaderJob; var height:int; var width:int;
The convertListToByteArray() method converts a list of numbers into a ByteArray:
 
byteArray = convertListToByteArray( list ); byteArray2 = convertListToByteArray( list2 );
Next, the code sets up all the data that is need for the shader job:
 
// the width of data source "image" is the length of the // byteArray and height is set to one. width = byteArray.length >> 2; height = 1; shader.data.src.width = width; shader.data.src.height = height; shader.data.src.input = byteArray; shader.data.src2.width = width; shader.data.src2.height = height; shader.data.src2.input = byteArray2; result = new ByteArray(); result.endian = Endian.LITTLE_ENDIAN;
Once everything is set, you create a ShaderJob object, add an event listener to handle the results, and start the job.
 
shaderJob = new ShaderJob(shader, result, width, height); shaderJob.addEventListener(Event.COMPLETE, onComplete); shaderJob.start(); }
Note that the width of the image is the length of the byteArray divided by 4, since a float number stores data in four bytes. When the job is completed, onComplete()converts the results back into a list of numbers:
 
protected function onComplete(event:Event):void { var length:int = result.length; var num:Number; var i:int; result.position = 0; for(i = 0; i < length; i += 4) { num = result.readFloat(); if(i % 3 == 0) listDest.dataProvider.addItem( num ); } }
Here is convertListToByteArray(), a utility method for converting a list into a ByteArray:
 
private static function convertListToByteArray(list:IList):ByteArray { var retVal:ByteArray = new ByteArray(); var number:Number; var len:int = list.length; var i:int; retVal.endian = Endian.LITTLE_ENDIAN; for (i; i<len; i++) { retVal.writeFloat( Number(list[i]) ); } retVal.position = 0; return retVal; }
The last part of the code is the UI, which contains two lists of numbers, a button to start the calculation, and a list to hold and display the results:
 
<s:List id="listSrc" width="57" height="158"> <s:ArrayCollection> <fx:Object>1</fx:Object> <fx:Object>2</fx:Object> <fx:Object>3</fx:Object> <fx:Object>4</fx:Object> <fx:Object>5</fx:Object> <fx:Object>6</fx:Object> <fx:Object>7</fx:Object> </s:ArrayCollection> </s:List> <s:List id="listSrc2" width="57" height="158" x="122" y="0"> <s:ArrayCollection> <fx:Object>2</fx:Object> <fx:Object>5</fx:Object> <fx:Object>2</fx:Object> <fx:Object>3</fx:Object> <fx:Object>4</fx:Object> <fx:Object>5</fx:Object> <fx:Object>6</fx:Object> </s:ArrayCollection> </s:List> <s:Button x="210" y="56" label="=" click="initializeHandler(listSrc.dataProvider, listSrc2.dataProvider)"/> <s:List id="listDest" width="110" height="158" x="314" y="-1"> <s:ArrayCollection /> </s:List> <s:RichText x="88" y="61" text="X"/> </s:Application>

 
Performing more complex calculations

The Pixel Bender kernel language also includes functions for more complex number crunching, including:
 
  • sin(x) - Returns the sine of x.
  • cos(x) – Returns cosine of x.
  • tan(x) – Returns the tangent of x.
  • asin(x) – Returns the arcsine (inverse sine) of x.
  • acos(x) – Returns the arccosine (inverse cosine) of x.
  • atan(x) – Returns the arctangent (inverse tangent) of x.
  • exp(x) - Returns ex.
  • log(x) – Returns the natural logarithm of x.
  • pow(x, y) - Returns xy.
  • sqrt(x) – Returns the positive square root of x.
For the second example, you will create and use a kernel that uses cos().
 
 
Create the kernel
Following the same procedure you used to create SimpleCalculator.pbj in the previous section, use the code below to create CosCalculator.pbj.
 
<languageVersion : 1.0;> kernel CosCalculator < namespace : "pixelBender"; vendor : "Elad Elrom"; version : 1; description : "Cos Calculator"; > { input image1 src; output pixel3 result; void evaluatePixel() { pixel1 value = pixel1(cos(sample(src, outCoord()))); result = pixel3(value, 0.0, 0.0); } }
 
Create the Flex Application
Create a new Flex application using the code below. This code is very similar to the previous example; the main difference is that it passes one list of numbers to the kernel instead of two.
 
<?xml version="1.0" encoding="utf-8"?> <s:Application xmlns:fx="http://ns.adobe.com/mxml/2009" xmlns:s="library://ns.adobe.com/flex/spark" xmlns:mx="library://ns.adobe.com/flex/mx" minWidth="1024" minHeight="768"> <fx:Script> <![CDATA[ import mx.collections.IList; [Embed(source="CosCalculator.pbj", mimeType="application/octet-stream")] private var KernelClass:Class; private var result:ByteArray; protected function initializeHandler(list:IList):void { var byteArray:ByteArray = new ByteArray(); var shader:Shader = new Shader(new KernelClass()); var shaderJob:ShaderJob; var height:int; var width:int; byteArray = convertListToByteArray( list ); width = byteArray.length >> 2; height = 1; shader.data.src.width = width; shader.data.src.height = height; shader.data.src.input = byteArray; result = new ByteArray(); result.endian = Endian.LITTLE_ENDIAN; shaderJob = new ShaderJob(shader, result, width, height); shaderJob.addEventListener(Event.COMPLETE, onComplete); shaderJob.start(); } protected function onComplete(event:Event):void { var length:int = result.length; var num:Number; var i:int; result.position = 0; for(i = 0; i < length; i += 4) { num = result.readFloat(); if(i % 3 == 0) listDest.dataProvider.addItem( num ); } } /** * Static method to convert the list object into byte array object. * * @param array * @return * */ private static function convertListToByteArray(list:IList):ByteArray { var retVal:ByteArray = new ByteArray(); var number:Number; var len:int = list.length; var i:int; retVal.endian = Endian.LITTLE_ENDIAN; for (i; i<len; i++) { retVal.writeFloat( Number(list[i]) ); } retVal.position = 0; return retVal; } ]]> </fx:Script> <s:List id="listSrc" width="57" height="158"> <s:ArrayCollection> <fx:Object>1</fx:Object> <fx:Object>2</fx:Object> <fx:Object>3</fx:Object> <fx:Object>4</fx:Object> <fx:Object>5</fx:Object> <fx:Object>6</fx:Object> <fx:Object>7</fx:Object> </s:ArrayCollection> </s:List> <s:Button x="71" y="12" label="Calculate &gt;" click="initializeHandler(listSrc.dataProvider)"/> <s:List id="listDest" width="200" height="158" x="160" y="0"> <s:ArrayCollection /> </s:List> </s:Application>
When you have built the application, run it in your browser and click Calculate to see the results (see Figure 5).
 
Note: The input to cos() is provided in radians.
 
The cosine example running in a browser

Figure 5. The cosine example running in a browser
 
Mixing audio files

The next example is a little more practical. The application you will create is a track mixer, which will take two audio tracks, mix them together, and return a result that is one-track mash-up.
 
 
The mixing kernel
The kernel uses the Pixel Bender mix(x, y, a) function, which returns a linear interpolation between x and y (specifically, it returns x * (1.0 - a) +y * a).
 
Here is the kernel code:
 
<languageVersion : 1.0;> kernel sound < namespace : "elad"; vendor : "Elad Elrom"; version : 1; description : "track mixer"; > { input image4 src0; input image4 src1; output float4 dst; parameter float distort < minValue:float(0); maxValue:float(1.0); defaultValue:float(1.0); >; void evaluatePixel() { float4 s1 = sampleNearest(src0,outCoord()); float4 s2 = sampleNearest(src1,outCoord()); dst = mix(s1,s2,distort); } }
Notice that this code has a parameter, distort, which can be used to adjust the level for each track during runtime.
 
 
The mixing API
You can download the TrackMixer class that implements an API for mixing two audio files using Pixel Bender from Google code.
 
The class is relatively simple. It starts by defining the variables it will use:
 
// pixel bender class and shader private var KernelClass:Class; private var shader:Shader; private var shaderJob:ShaderJob; // num of tracks private var numOfTracks:Number = 0; // counter private var trackDownloadCounter:int = 0; // buffer & sound objects private var buffer:Vector.<ByteArray> = new Vector.<ByteArray>; private var sound:Vector.<Sound> = new Vector.<Sound>;
The start() method loads the tracks that will be mixed.
 
public function start(urls:Array):void { if (urls.length >2) { this.dispatchErrorEvent( "API only supports two tracks at this point." ); } numOfTracks = urls.length; for (var i:int = 0; i< numOfTracks; i++) { sound[i] = new Sound(new URLRequest(urls[i])); sound[i].addEventListener(Event.COMPLETE, onSoundLoaded); sound[i].addEventListener(IOErrorEvent.IO_ERROR, onError); } }
After loading the tracks, the next step is to set up the ShaderJob object and parameters, and then start the job.
 
private function onSampleDataHandler(event:SampleDataEvent):void { var width:int = 1; var height:Vector.<int> = new Vector.<int>(numOfTracks); for (var i:int = 0; i < numOfTracks; i++) { buffer[i] = new ByteArray(); sound[i].extract(buffer[i],BUFFER_SIZE * 4); height[i] = buffer[i].length >> 4; buffer[i].position = 0; shader.data["src"+i]["input"] = buffer[i]; shader.data["src"+i]["width"] = width; shader.data["src"+i]["height"] = height[i]; } shader.data.distort.value = [this.balance]; shaderJob = new ShaderJob(shader, event.data,width, height[0]); shaderJob.start(true); }
 
The mixing application
The next step is to create an application that uses the mixing API (see Figure 6).
 
The track mixing application
Figure 6. The track mixing application
 
The application embeds the Pixel Bender kernel and passes it to the TrackMixer. It then starts the TrackMixer, passing it URLs to the two tracks. When the slider moves, the code updates the mixer balance. The full code is below:
 
<?xml version="1.0" encoding="utf-8"?> <s:Application xmlns:fx="http://ns.adobe.com/mxml/2009" xmlns:s="library://ns.adobe.com/flex/spark" xmlns:mx="library://ns.adobe.com/flex/mx" xmlns:local="*" minWidth="1024" minHeight="768" creationComplete="creationCompleteHandler(event)"> <fx:Script> <![CDATA[ import com.elad.framework.sound.events.TrackMixerErrorEvent; import com.elad.framework.sound.TrackMixer; import mx.events.FlexEvent; [Embed("assets/pbj/TwoTracksMixer.pbj", mimeType="application/octet-stream")] private var Kernel:Class; private var trackMixer:TrackMixer; protected function creationCompleteHandler(event:FlexEvent):void { trackMixer = new TrackMixer(Kernel); trackMixer.addEventListener(TrackMixerErrorEvent.TRACK_MIXER_ERROR, function(e:*):void { trace(e.message); } ); trackMixer.start( ["assets/tracks/FeelinGood.mp3", "assets/tracks/Sunshine.mp3"] ); } protected function balanceSliderChangeHandler(event:Event):void { trackMixer.balance = event.currentTarget.value ; } ]]> </fx:Script> <mx:Slider x="28" y="171" id="balanceSlider" labels='["Track1","Mix","Track2"]' minimum="0" maximum="1" liveDragging="true" value="0.5" change="balanceSliderChangeHandler(event)" /> <local:Visualization type="wave" bars="32" width="300" height="137" x="28" y="14"/> </s:Application>

 
Creating a kernel from assembler code

So far, you've seen how easy it is to create PBJ files using the Pixel Bender Toolkit. It is possible, however, to create PBJ files directly from assembler code.
 
Note: To use this technique you will need to know how to write assembler code. Further, the technique relies on unsupported tools and an undocumented syntax. See this post for more details.
 
Why would you want to do this? The Pixel Bender Toolkit comes with some limitations. For instance, in the music mixer application, you can only mix two tracks, and you may want to mix six tracks.
 
When you try to add a third image, you get the following error: "This version of Adobe Pixel Bender Toolkit does not support kernel with more than 2 inputs". Additionally, you may want to have more control over the code and ensure that it is optimized for maximum performance.
 
Tinix Uro put together an unofficial assembler and disassembler that allow you to write assembler code and create a PBJ file. Please note that the code is not officially supported by Adobe.
 
You can download the C++ code from here:
 
On Windows, you can download pre-built binaries of these two command line tools:
 
On Mac OS X, you will need to compile the C++ code yourself. Type the following commands to create the binaries:
 
g++ apbj.cpp -o apbj g++ dpbj.cpp -o dpbj
Note: You need the Xcode suite installed on your Mac in order to use g++, which is in the Gnu Compiler Collection.
 
Using the apbj tool, you can create a PBJ file from assembler code. For example:
 
./apbj input.pba -o output.pbj
And using the dpbj tool you can create assembler from a PBJ file:
 
./dpbj input.pbj -o output.pba
To create a PBJ file using assembler, you first need to understand its structure.
 
I used dpbj to disassemble TwoTracksMixer.pbj, the PBJ file use in the mixing example. Here is the resulting assembler code:
 
version 1 name "sound" kernel "namespace", "elad" kernel "vendor", "Elad Elrom" kernel "version", 1 kernel "description", "track mixer" parameter "_OutCoord", float2, f0.rg, in texture "src0", t0 texture "src1", t1 parameter "dst", float4, f1, out parameter "distort", float, f0.b, in meta "minValue", 0 meta "maxValue", 1 meta "defaultValue", 0.5 ;---------------------------------------------------------- texn f2, f0.rg, t0 mov f3, f2 texn f2, f0.rg, t1 mov f4, f2 mov f2.r, f0.b mov f2.g, f0.b mov f2.b, f0.b mov f2.a, f0.b set f5, 1 sub f5, f2 mov f6, f3 mul f6, f5 mov f7, f4 mul f7, f2 add f6, f7 mov f1, f6
The first portion of the code is the kernel definition; just as with the Pixel Bender Toolkit you need to create the metadata:
 
version 1 name "sound" kernel "namespace", "elad" kernel "vendor", "Elad Elrom" kernel "version", 1 kernel "description", "track mixer"
The next part defines the inputs, outputs, and parameters.
 
The texture lines define the source images:
 
texture "src0", t0 texture "src1", t1
A kernel can take any number of parameters of any type. When you define parameters, you supply metadata, such as minimum, maximum, and default values. You can store several parameters in the same register by using different channels. For instance, the _OutCoord parameter stores the (X,Y) pixel position in the R and G channels of register 0. Other parameters might use the B and A channel:
 
parameter "_OutCoord", float2, f0.rg, in … parameter "distort", float, f0.b, in
Each parameter takes three arguments:
 
  • The type of the parameter;
  • A register, which indicates where the parameter will be accessible
  • In or Out, which identifies the parameter as an input or an output
There are four metadata values that you can set: minValue, maxValue, defaultValue, and description. For example:
 
parameter "distort", float, f0.b, in meta "minValue", 0 meta "maxValue", 1 meta "defaultValue", 0.5
The last part of the code includes the instructions that are used to perform the operations. For example, mov is used to move a value into a register, mul is used to multiply values, and sub is used to subtract values.
 
texn f2, f0.rg, t0 mov f3, f2 texn f2, f0.rg, t1 mov f4, f2 mov f2.r, f0.b mov f2.g, f0.b mov f2.b, f0.b mov f2.a, f0.b set f5, 1 sub f5, f2 mov f6, f3 mul f6, f5 mov f7, f4 mul f7, f2
If you are handy with assembler code, you can fine-tune your code, and then create a new, improved PBJ file using the apbj tool. This capability opens new opportunities for using Pixel Bender for more complex calculations.
 

 
Where to go from here

This article has provided a brief introduction to the number crunching capabilities of Pixel Bender.
 
To learn more, see the following resources: