back

Learn to augment reality using
a webcam and Flash CS4

by Samuel Asher Rivello

Augmented reality (AR) is a field of computer science that involves combining the physical world and an interactive, three-dimensional virtual world.

While mainstream audiences are now becoming aware of AR, it is not new. Its background is intertwined with decades of computer science development. Virtual reality (VR), AR's more familiar counterpart, is the replacement of the user's physical reality (particularly that which is experienced through sight and hearing) with a computer-generated reality. The idea of a virtual experience is exciting — creating entertaining and educational sensory encounters that do not exist in our everyday lives.

From a consumer standpoint, it seems that AR advances have come out of nowhere to surpass VR advances. The acceleration in AR technology is due to two major factors: First, users are still experiencing reality, so believability is easier to achieve. Adding simple graphics (such as text or simple shapes) and color effects (such as night vision or thermal vision) to reality creates a better user experience. The user is still seeing a mostly familiar world. Second, this more subtle use of computer graphics is less expensive with today's technology, making it more feasible than VR.

Let's take a look at augmented reality, its current uses, and its future potential. Then we'll dig in and see how to apply this exciting technology using Adobe Flash CS4 Professional, Adobe Flash Player 10, and a webcam.

Applications of augmented reality

The video game industry has released major augmented reality products for more than a decade. The Eye Toy for Sony PlayStation 2 and 3 takes input from an inexpensive video camera and composites the live video feed with CG onto the TV screen. This toy detects the user's body position in front of the camera as an alternative input method to the typical joystick or gamepad, deepening the user's immersion into the game world. Detection schemes continue to advance, allowing for more engaging interaction.

There are AR applications outside of console games, including military and consumer products, too. Night-vision goggles and targeting-assistance technology help marksmen in battle, and children's stories come to life with AR-enhanced books. The uses are vast.

With AR-enhanced books, each page of a real-world book is outfitted with a marker — a pattern that is detectable by AR. As the child turns each page, a computer is able to place a virtual 3D animation on top of a physical image printed on paper. In this article, we will learn to create this type of AR experience. Let's dig in!

Augmented reality with Flash CS4

Let's create a project in which a webcam will capture the user, holding a marker image in an arbitrary position, and the application will augment the webcam feed with a 3D model overlaid onto the marker's position in a believable way. You can get this project up and running in less than an hour (download source files, 1.5MB).

With Flash Player 10, developers have the most robust toolset yet for rich application development. Flash Player handles the heavy lifting such as video input, pixel-level image manipulation, and the heavy number crunching afforded with ActionScript 3. All you need is a webcam, a few open source libraries, and Flash CS4 to get the job done.

New project setup

Before coding a Flash CS4 project, you need to prepare the libraries and assets. Once the project is set up, most changes will happen within the document class (see Figure 1). To get started:

  1. Create a new project in Flash CS4 by choosing File > New > Flash File (ActionScript 3.0), and save the project as AugmentedReality.fla inside an AugmentedReality folder on your desktop.
  2. Create a document class by choosing File > New > ActionScript File, and save it as AugmentedReality.as within the AugmentedReality folder.
  3. Set this file as the project's document class to the AugmentedReality class file by choosing Window > Property > Class. The class file will be empty at this point.
  4. Create a libs folder inside the AugmentedReality folder, and add it to the project's library path by choosing File > Publish Settings > Flash > ActionScript 3.0 > Library Path > [Folder Icon].
  5. Download the three libraries discussed in the next section, and add each SWC to the libs folder.
  6. Create an Assets folder and add the 3D Collada model file and two AR data files.
Augmented reality project folder after setup.

Figure 1. Augmented reality project folder after setup.

External library setup

A great benefit of ActionScript 3's object-oriented programming model is that developers can import existing libraries of code to save ramp-up time and vastly expand the development power of Flash. This implementation of AR takes advantage of three such libraries, each with a specific role in the project:

Asset preparation

Tower marker graphic.

Figure 2. Tower marker graphic.

The Flash Augmented Reality Toolkit (FLARToolkit) library holds much of the magic for this application. It will introspect the webcam image for a particular graphic image and judge where in real space to map the 3D model. The FLARCameraParams.dat file, which comes with the toolkit, will be used as-is. It corrects webcam distortion and helps the toolkit with its general detection routine. The graphic marker image information is held within the FLARPattern.pat file. The file is made to work with a particular marker graphic image (see Figure 2). If you want to use a different graphic, create a JPEG image with your favorite drawing program, and use the FLAR Marker Generator AIR application to generate a new pattern file.

The marker image is a graphic drawn, printed, and shown to the end application as it runs. FLAR, with help from the marker data file and parameters file, will detect this shape via webcam. To design a graphic, fit the desired shape within a white square that is centered within a larger black square. Keep your shape simple for best results.

Note the sun shape in the graphic. This helps the application detect which way is up for the marker and assists its angle and rotation detection. Your design doesn't need to have this shape per se, but it should be asymmetrical in some way for best results.

Also, you'll need a 3D model in the Collada format that Papervision3D requires. Collada models can be created and exported using many popular 3D modeling tools, including the open source Blender3D.

The document class

The AugmentedReality.as document class is where all of the project's code lives. Figure 3 shows the completed class. As you can see, much of the complexity of FLAR and Papervision3D for the project is hidden within the imported libraries and is therefore out of the way.

package {
      
    //--------------------------------------
    // Imports
    //--------------------------------------
    import flash.display.BitmapData;
    import flash.display.Sprite;
    import flash.events.Event;
    import flash.media.Camera;
    import flash.media.Video;
    import flash.utils.ByteArray;
    
    import org.libspark.flartoolkit.core.FLARCode;
    import org.libspark.flartoolkit.core.param.FLARParam;
    import org.libspark.flartoolkit.core.raster.rgb.FLARRgbRaster_BitmapData;
    import org.libspark.flartoolkit.core.transmat.FLARTransMatResult;
    import org.libspark.flartoolkit.detector.FLARSingleMarkerDetector;
    import org.libspark.flartoolkit.pv3d.FLARBaseNode;
    import org.libspark.flartoolkit.pv3d.FLARCamera3D;
    
    import org.papervision3d.lights.PointLight3D;
    import org.papervision3d.materials.shadematerials.FlatShadeMaterial;
    import org.papervision3d.materials.utils.MaterialsList;
    import org.papervision3d.objects.parsers.DAE;
    import org.papervision3d.objects.primitives.Cube;
    import org.papervision3d.render.BasicRenderEngine;
    import org.papervision3d.scenes.Scene3D;
    import org.papervision3d.view.Viewport3D;
    
    
    //--------------------------------------
    // Class Definition
    //--------------------------------------
    public class AugmentedReality extends Sprite
    {
        
        //--------------------------------------
        // Class Properties
        //--------------------------------------

        //		1. WebCam
        private var video    : Video;        
        private var webcam    : Camera;    

        //  	2. FLAR Marker Detection
        private var flarBaseNode          : FLARBaseNode;        
        private var flarParam            : FLARParam;
        private var flarCode            : FLARCode;
        private var flarRgbRaster_BitmapData    : FLARRgbRaster_BitmapData;
        private var flarSingleMarkerDetector    : FLARSingleMarkerDetector;
        private var flarCamera3D          : FLARCamera3D;        
        private var flarTransMatResult       : FLARTransMatResult;
        private var bitmapData           : BitmapData;
        private var FLAR_CODE_SIZE         : uint         = 16;
        private var MARKER_WIDTH          : uint         = 80;

        
        [Embed(source="./assets/FLAR/FLARPattern.pat", mimeType="application/octet-stream")]
        private var Pattern : Class;
        
        [Embed(source="./assets/FLAR/FLARCameraParameters.dat", mimeType="application/octet-stream")]
        private var Params	 : Class;
                    
        //  	3. PaperVision3D
        private var basicRenderEngine    : BasicRenderEngine;
        private var viewport3D        : Viewport3D;        
        private var scene3D         : Scene3D;
        private var collada3DModel      : DAE;
    
        //  	Fun, Editable Properties
        private var VIDEO_WIDTH        : Number = 640;        //Set 100 to 1000 to set width of screen
        private var VIDEO_HEIGHT       : Number = 480;        //Set 100 to 1000 to set height of screen
        private var WEB_CAMERA_WIDTH     : Number = VIDEO_WIDTH/2;   //Smaller than video runs faster
        private var WEB_CAMERA_HEIGHT     : Number = VIDEO_HEIGHT/2;  //Smaller than video runs faster
        private var VIDEO_FRAME_RATE     : Number = 30;         //Set 5 to 30. Higher values = smoother video
        private var DETECTION_THRESHOLD   : uint  = 80;      		//Set 50 to 100. Set to detect marker more accurately.
        private var DETECTION_CONFIDENCE   : Number = 0.5;        //Set 0.1 to 1. Set to detect marker more accurately.
        private var MODEL_SCALE        : Number = 0.8;        //Set 0.1 to 5. Set higher to enlarge model
        
        // 	Fun, Editable Properties: Load a Different Model
        private var COLLADA_3D_MODEL     : String = "./assets/models/licensed/hummer/models/hummer.dae";
        
        
        //--------------------------------------
        // Constructor
        //--------------------------------------
        
        /**
         * 	The constructor is the ideal place 
         * 	for project setup since it only runs once.
         * 	Prepare A,B, & C before repeatedly running D.
        **/
        public function AugmentedReality ()
        {
            //		Prepare
            prepareWebCam ();         //Step A
            prepareMarkerDetection ();    //Step B
            preparePaperVision3D ();     //Step C
            
            //   	Repeatedly call the loop method
            //   to detect and adjust the 3D model.
            addEventListener(Event.ENTER_FRAME, loopToDetectMarkerAndUpdate3D); //Step D
        }
        
        
        //--------------------------------------
        // Methods
        //--------------------------------------
        
        /**
         * A. 	Access the user's webcam, wire it 
         *  	to a video object, and display the
         *  	video onscreen.
        **/
        private function prepareWebCam () : void
        {
            video  = new Video(VIDEO_WIDTH, VIDEO_HEIGHT);
            webcam = Camera.getCamera();
            webcam.setMode(WEB_CAMERA_WIDTH, WEB_CAMERA_HEIGHT, VIDEO_FRAME_RATE);
            video.attachCamera(webcam);
            addChild(video);
        }    
        
        
        /**
         * B. 	Prepare the FLAR tools to detect with
         * 	parameters, the marker pattern, and
         *   a BitmapData object to hold the information
         *   of the most recent webcam still-frame.
        **/
        private function prepareMarkerDetection () : void
        {
            //		The parameters file corrects imperfections
            //   In the webcam's image. The pattern file
            //   defines the marker graphic for detection
            //   by the FLAR tools.
            flarParam = new FLARParam	();
            flarParam.loadARParam		(new Params() as ByteArray);
            flarCode = new FLARCode 	(FLAR_CODE_SIZE, FLAR_CODE_SIZE);
            flarCode.loadARPatt			(new Pattern());
            
            
            //   A BitmapData is Flash's version of a JPG image in memory.
            //   FLAR studies this image every frame with its
            //   marker-detection code.
            bitmapData = new BitmapData(VIDEO_WIDTH, VIDEO_HEIGHT);
            bitmapData.draw (video);
            flarRgbRaster_BitmapData = new FLARRgbRaster_BitmapData(bitmapData);
            flarSingleMarkerDetector = new FLARSingleMarkerDetector (flarParam, flarCode, MARKER_WIDTH);
        }    
        
        
        /**
         * C. 	Create PaperVision3D's 3D tools including
         *  	a scene, a base node container to hold the
         *   3D Model, and the loaded 3D model itself. 
        **/        
        private function preparePaperVision3D () : void
        {
            //		Basics of the empty 3D scene fit for
            //		FLAR detection inside a 3D render engine.
            basicRenderEngine     = new BasicRenderEngine();
            flarTransMatResult     = new FLARTransMatResult();
            viewport3D         = new Viewport3D();
            flarCamera3D        = new FLARCamera3D(flarParam);
            flarBaseNode        = new FLARBaseNode();
            scene3D           = new Scene3D();
            scene3D.addChild(flarBaseNode);
            
            //		Load, scale, and position the model
            //		The position and rotation will be
            //		adjusted later in method D below.
            collada3DModel = new DAE ();
            collada3DModel.load (COLLADA_3D_MODEL);
            collada3DModel.scaleX = collada3DModel.scaleY = collada3DModel.scaleZ = MODEL_SCALE;
            collada3DModel.z = 5;       	//Moves Model 'Up' a Line Perpendicular to Marker
            collada3DModel.rotationX = -90; 	//Rotates Model Around 2D X-Axis of Marker
            collada3DModel.rotationY = 180; 	//Rotates Model Around 2D Y-Axis of Marker
            collada3DModel.rotationZ = 90;   //Rotates Model Around a Line Perpendicular to Marker

            //  	Add the 3D model into the 
            //  	FLAR container and add the 
            //  	3D cameras view to the screen
            //		so the user can view the result
            flarBaseNode.addChild(collada3DModel);
            addChild (viewport3D);
        }    
        
        
        /**
         * D. 	Detect the marker in the webcamera. If
         *   found: move, scale, and rotate the 
         *   3D model to composite it over the marker
         *   in the user's physical space.
        **/
        private function loopToDetectMarkerAndUpdate3D (aEvent : Event) : void
        {
            
            //		Copy the latest still-frame of the webcam video
            //		into the BitmapData object for detection
            bitmapData.draw (video);

            try {
                
                //		Detect *IF* the marker is found in the latest still-frame
                if (flarSingleMarkerDetector.detectMarkerLite (flarRgbRaster_BitmapData, DETECTION_THRESHOLD) && 
                   flarSingleMarkerDetector.getConfidence() > DETECTION_CONFIDENCE                            ) {
                        
                    //		Repeatedly Loop and Adjust 3D Model to Match Marker
                    flarSingleMarkerDetector.getTransformMatrix (flarTransMatResult);
                    flarBaseNode.setTransformMatrix (flarTransMatResult);
                    basicRenderEngine.renderScene	 (scene3D, flarCamera3D, viewport3D);
                }
            } catch (error : Error) {}
        }
    }
}

Figure 3. The complete document class file.

Stepping through the document class

ActionScript 3 class basics

ActionScript 3 class files are structured into several sections, and each has a specific role in this project.

  • Imports: Typically your class will take advantage of existing code libraries — libraries you may not have coded yourself. Each import statement loads in the needed class or package from an existing library. You need libraries from Flash, FLARToolkit, and Papervision3D for this project.
  • Class definition: The bulk of your classes code should be in the class definition section.
  • Class properties: Declare all public, private, and static variables. Arguments and local variables are declared within the constructor body and the method bodies. For this project, you need only private variables; no outside classes need access.
  • Constructor: This function runs when the class is created. The function's name must match the class name. It runs only once so it's an ideal place to initialize your class. In the following steps, setup code will be placed here. In this project, code is placed to set up the camera, FLARToolkit, and Papervision3D and to prepare a repeating loop to handle marker detection.
  • Methods: Methods are class functions or reusable chunks of code. This project uses only four methods to handle setup and marker detection.

Imports and properties

Starting at the top of the AugmentedReality.as document class, we see import statements for the needed libraries. Within the Properties section, variables are declared. Most of this section is pretty standard, except the Pattern and Params property declarations from lines 58–62. This section is shown again in detail in Figure 4.

[Embed(source="../assets/FLAR/FLARPattern.pat", mimeType="application/octet-stream")]
private var Pattern: Class;

[Embed(source="../assets/FLAR/FLARCameraParameters.dat", mimeType="application/octet-stream")]
private var Params : Class;

Figure 4. The Flex Embed metatag within Flash CS4.

Adobe Flex uses a special type of code called the Embed metatag. Embed tags, as they are commonly called, enable a property to be initialized with a value loaded from an external file. A unique aspect is that the byte loading happens at compile time (as the project is published) rather than at runtime (while the user interacts with the application). The bytes for the loaded file are baked into the SWF file. Flash CS4 now supports this feature as well with help from the flex.SWC, which is included in this project's libs folder.

Class constructor

Since the constructor of a class is called only once, it is an ideal location for one-time-only setup code. Starting on line 93, the preparation methods are called for each of the following: the webcam, the marker detection, and Papervision3D. In addition to the setup methods, the loopToDetectMarkerAndUpdate3D is set up to run again and again. By listening to the ENTER_FRAME event built into Flash, this loop will be called repeatedly as long as the application is running.

Webcam setup

Flash features great webcam support. As the application runs and a camera is detected, Flash Player will automatically prompt the user to enable it as line 96 of the code executes. The Video object is the window onscreen where the video will be shown, and the Camera object records and feeds its live footage to that Video object.

FLAR setup

A cornerstone of this project is marker detection. As the marker graphic is detected via webcam, its position, rotation, and scale in the user's physical space are calculated. This calculation will be used in the next steps. Starting with line 97, FLAR will search each still-frame of webcam footage for the predefined marker graphic (Pattern object). The detection scheme uses an ActionScript BitmapData object, which contains only the latest still-frame of video at any given time (the BitmapData Draw function).

Papervision3D setup

For the most part, the 3D setup at line 98 is typical of any Papervision3D project. A BasicRenderEngine object will handle the bulk of the vector math, converting 3D geometry to 2D graphics for display on the screen. The Collada 3D model is loaded and inserted into the 3D scene, and the view into that scene (ViewPort object) is added to the Stage (see Figure 5). Luckily this is all done for you. In the final project, this view is layered on top of the 2D webcam Video object shown earlier so the composite of 2D and 3D will look believable.

Collada model loaded and rendered with Papervision3D.

Figure 5. Collada model loaded and rendered with Papervision3D.

Repeated detect-and-update routine

As mentioned earlier, by using the ENTER_FRAME you ensure this loopToDetectMarkerAndUpdate3D function on lines 102 will be called 30 times per second. Each time, the BitmapData object will copy in the latest still-frame from the webcam footage. FLAR runs detection on the still-frame. If the marker is detected, FLAR returns updated position, rotation, and scale data (TransformMatrix object). The container (FLARBaseNode object) holding the 3D model is updated to that position so the model appears to be at the same position in the user's physical space as the marker. The resulting effect is nothing short of amazing for first-time viewers.

Running the completed project

Print out the graphic marker. Show the marker to the running application via the webcam (see Figure 6). The red line is for illustration purposes to denote successful initial marker detection. Much like Arnold Schwarzenegger's Terminator character, the project can now read in the surrounding environment, visually scan and detect relevant objects, and make calculations based on that information. It works best to have a simple backdrop of a solid wall in the application because complex scenes may confuse the detection scheme.

The application detecting the graphic marker via webcam.

Figure 6. The application detecting the graphic marker via webcam.

Next, seeing the marker graphic through the webcam, the application superimposes an animated 3D model onto the user's physical space (see Figure 7). The model now rotates and scales believably as the user moves the marker.

The application replaces the marker with a 3D model.

Figure 7. The application replaces the marker with a 3D model.

Conclusion

There you have it. Augmented reality is within the grasp of all Flash developers. The FLAR Toolkit can detect multiple markers to bring even more 3D into your world. As technology matures, you will be able to detect more complex graphic markers within more complex scenes. The choices are endless. What will you choose to do with augmented reality: print and show a marker to an online application as we've done here, buy and play with a real-world AR-enabled toy, or detect and destroy like the Terminator?

‹ Back


Samuel Asher Rivello is the principal of Rivello Multimedia Consulting (RMC). RMC's Flash and Flex services include software architecture, consulting, development, and training. Sam has a decade of experience creating games and applications, and is currently traveling the globe to collaborate with top companies. Follow Sam on Twitter at @srivello.