Augmented reality using a webcam and Flash
Created
8 September 2009
Requirements
Additional required products
External code libraries:
Optional drawing/rendering programs:
- FLAR Marker Generator [AIR, 322K]
- Blender3D
User Level
Beginning
Required products
- Flash Professional (Download trial)
Sample files
- augmented_reality_source.zip (1509 KB)
Augmented reality (AR) is a field of computer science that involves combining the physical world and an interactive, three-dimensional virtual world. This article takes a look at augmented reality, its current uses, and its future potential. Then you'll dig in and see how to apply this exciting technology using Adobe Flash CS4 Professional, Adobe Flash Player 10, and a webcam.
While mainstream audiences are now becoming aware of AR, it is not new. Its background is intertwined with decades of computer science development. Virtual reality (VR), AR's more familiar counterpart, is the replacement of the user's physical reality (particularly that which is experienced through sight and hearing) with a computer-generated reality. The idea of a virtual experience is exciting—creating entertaining and educational sensory encounters that do not exist in our everyday lives.
From a consumer standpoint, it seems that AR advances have come out of nowhere to surpass VR advances. The acceleration in AR technology is due to two major factors: First, users are still experiencing reality, so believability is easier to achieve. Adding simple graphics (such as text or simple shapes) and color effects (such as night vision or thermal vision) to reality creates a better user experience. The user is still seeing a mostly familiar world. Second, this more subtle use of computer graphics is less expensive with today's technology, making it more feasible than VR.
Applications of augmented reality
The video game industry has released major augmented reality products for more than a decade. The EyeToy for Sony PlayStation 2 and 3 takes input from an inexpensive video camera and composites the live video feed with CG onto the TV screen. This toy detects the user's body position in front of the camera as an alternative input method to the typical joystick or gamepad, deepening the user's immersion into the game world. Detection schemes continue to advance, allowing for more engaging interaction.
There are AR applications outside of console games, including military and consumer products, too. Night-vision goggles and targeting-assistance technology help marksmen in battle, and children's stories come to life with AR-enhanced books. The uses are vast.
With AR-enhanced books, each page of a real-world book is outfitted with a marker—a pattern that is detectable by AR. As the child turns each page, a computer is able to place a virtual 3D animation on top of a physical image printed on paper. In this article, you will learn to create this type of AR experience. Let's dig in!
Augmented reality with Flash CS4
Let's create a project in which a webcam will capture the user, holding a marker image in an arbitrary position, and the application will augment the webcam feed with a 3D model overlaid onto the marker's position in a believable way. You can get this project up and running in less than an hour.
With Flash Player 10, developers have the most robust toolset yet for rich application development. Flash Player handles the heavy lifting such as video input, pixel-level image manipulation, and the heavy number crunching afforded with ActionScript 3. All you need is a webcam, a few open source libraries, and Flash CS4 to get the job done.
New project setup
Before coding a Flash CS4 project, you need to prepare the libraries and assets. Once the project is set up, most changes will happen within the document class (see Figure 1). To get started:
- Create a new project in Flash CS4 by choosing File > New > Flash File (ActionScript 3.0), and save the project as AugmentedReality.fla inside an AugmentedReality folder on your desktop.
- Create a document class by choosing File > New > ActionScript File, and save it as AugmentedReality.as within the AugmentedReality folder.
- Set this file as the project's document class to the AugmentedReality class file by choosing Window > Property > Class. The class file will be empty at this point.
- Create a libs folder inside the AugmentedReality folder, and add it to the project's library path by choosing File > Publish Settings > Flash > ActionScript 3.0 > Library Path > [Folder Icon].
- Download the three libraries discussed in the next section, and add each SWC to the libs folder.
- Create an Assets folder and add the 3D Collada model file and two AR data files.
External library setup
A great benefit of ActionScript3's object-oriented programming model is that developers can importexisting libraries of code to save ramp-up time and vastly expand thedevelopment power of Flash. This implementation of AR takes advantageof three such libraries, each with a specific role in the project:
- Flex SDK code library: Processes the Embed metatag needed to import FLAR data files.
- Spark Project's Flash Augmented Reality code library (FLARToolkit): Handles marker graphic detection.
- Papervision3D code library (PV3D): Handles the importing, positioning, and rendering of the 3D model.
Asset preparation
TheFlash Augmented Reality Toolkit (FLARToolkit) library holds much of themagic for this application. It will introspect the webcam image for aparticular graphic image and judge where in real space to map the 3Dmodel. The FLARCameraParams.dat file, which comes with the toolkit,will be used as-is. It corrects webcam distortion and helps the toolkitwith its general detection routine. The graphic marker imageinformation is held within the FLARPattern.pat file. The file is madeto work with a particular marker graphic image (see Figure 2). If youwant to use a different graphic, create a JPEG image with your favoritedrawing program, and use the FLAR Marker Generator (AIR, 322K) to generate a new pattern file.
The marker image is a graphic drawn, printed, and shown to the endapplication as it runs. FLAR, with help from the marker data file andparameters file, will detect this shape via webcam. To design agraphic, fit the desired shape within a white square that is centeredwithin a larger black square. Keep your shape simple for best results.
Notethe sun shape in the graphic. This helps the application detect whichway is up for the marker and assists its angle and rotation detection.Your design doesn't need to have this shape per se, but it should beasymmetrical in some way for best results.
Also, you'll needa 3D model in the Collada format that Papervision3D requires. Colladamodels can be created and exported using many popular 3D modelingtools, including the open source Blender3D.
The document class
TheAugmentedReality.as document class is where all of the project's codelives. The following code shows the completed class. As you can see,much of the complexity of FLAR and Papervision3D for the project ishidden within the imported libraries and is therefore out of the way:
package {
//--------------------------------------
// Imports
//--------------------------------------
import flash.display.BitmapData;
import flash.display.Sprite;
import flash.events.Event;
import flash.media.Camera;
import flash.media.Video;
import flash.utils.ByteArray;
import org.libspark.flartoolkit.core.FLARCode;
import org.libspark.flartoolkit.core.param.FLARParam;
import org.libspark.flartoolkit.core.raster.rgb.FLARRgbRaster_BitmapData;
import org.libspark.flartoolkit.core.transmat.FLARTransMatResult;
import org.libspark.flartoolkit.detector.FLARSingleMarkerDetector;
import org.libspark.flartoolkit.pv3d.FLARBaseNode;
import org.libspark.flartoolkit.pv3d.FLARCamera3D;
import org.papervision3d.lights.PointLight3D;
import org.papervision3d.materials.shadematerials.FlatShadeMaterial;
import org.papervision3d.materials.utils.MaterialsList;
import org.papervision3d.objects.parsers.DAE;
import org.papervision3d.objects.primitives.Cube;
import org.papervision3d.render.BasicRenderEngine;
import org.papervision3d.scenes.Scene3D;
import org.papervision3d.view.Viewport3D;
//--------------------------------------
// Class Definition
//--------------------------------------
public class AugmentedReality extends Sprite
{
//--------------------------------------
// Class Properties
//--------------------------------------
// 1. WebCam
private var video : Video;
private var webcam : Camera;
// 2. FLAR Marker Detection
private var flarBaseNode : FLARBaseNode;
private var flarParam : FLARParam;
private var flarCode : FLARCode;
private var flarRgbRaster_BitmapData : FLARRgbRaster_BitmapData;
private var flarSingleMarkerDetector : FLARSingleMarkerDetector;
private var flarCamera3D : FLARCamera3D;
private var flarTransMatResult : FLARTransMatResult;
private var bitmapData : BitmapData;
private var FLAR_CODE_SIZE : uint = 16;
private var MARKER_WIDTH : uint = 80;
[Embed(source="./assets/FLAR/FLARPattern.pat", mimeType="application/octet-stream")]
private var Pattern : Class;
[Embed(source="./assets/FLAR/FLARCameraParameters.dat", mimeType="application/octet-stream")]
private var Params : Class;
// 3. PaperVision3D
private var basicRenderEngine : BasicRenderEngine;
private var viewport3D : Viewport3D;
private var scene3D : Scene3D;
private var collada3DModel : DAE;
// Fun, Editable Properties
private var VIDEO_WIDTH : Number = 640; //Set 100 to 1000 to set width of screen
private var VIDEO_HEIGHT : Number = 480; //Set 100 to 1000 to set height of screen
private var WEB_CAMERA_WIDTH : Number = VIDEO_WIDTH/2; //Smaller than video runs faster
private var WEB_CAMERA_HEIGHT : Number = VIDEO_HEIGHT/2; //Smaller than video runs faster
private var VIDEO_FRAME_RATE : Number = 30; //Set 5 to 30. Higher values = smoother video
private var DETECTION_THRESHOLD : uint = 80; //Set 50 to 100. Set to detect marker more accurately.
private var DETECTION_CONFIDENCE : Number = 0.5; //Set 0.1 to 1. Set to detect marker more accurately.
private var MODEL_SCALE : Number = 0.8; //Set 0.1 to 5. Set higher to enlarge model
// Fun, Editable Properties: Load a Different Model
private var COLLADA_3D_MODEL : String = "./assets/models/licensed/hummer/models/hummer.dae";
//--------------------------------------
// Constructor
//--------------------------------------
/**
* The constructor is the ideal place
* for project setup since it only runs once.
* Prepare A,B, & C before repeatedly running D.
**/
public function AugmentedReality()
{
//Prepare
prepareWebCam(); //Step A
prepareMarkerDetection(); //Step B
preparePaperVision3D(); //Step C
// Repeatedly call the loop method
// to detect and adjust the 3D model.
addEventListener(Event.ENTER_FRAME, loopToDetectMarkerAndUpdate3D); //Step D
}
//--------------------------------------
// Methods
//--------------------------------------
/**
* A. Access the user's webcam, wire it
* to a video object, and display the
* video onscreen.
**/
private function prepareWebCam() : void
{
video = new Video(VIDEO_WIDTH, VIDEO_HEIGHT);
webcam = Camera.getCamera();
webcam.setMode(WEB_CAMERA_WIDTH, WEB_CAMERA_HEIGHT, VIDEO_FRAME_RATE);
video.attachCamera(webcam);
addChild(video);
}
/**
* B. Prepare the FLAR tools to detect with
* parameters, the marker pattern, and
* a BitmapData object to hold the information
* of the most recent webcam still-frame.
**/
private function prepareMarkerDetection() : void
{
// The parameters file corrects imperfections
// In the webcam's image. The pattern file
// defines the marker graphic for detection
// by the FLAR tools.
flarParam = new FLARParam();
flarParam.loadARParam(new Params() as ByteArray);
flarCode = new FLARCode(FLAR_CODE_SIZE, FLAR_CODE_SIZE);
flarCode.loadARPatt(new Pattern());
// A BitmapData is Flash's version of a JPG image in memory.
// FLAR studies this image every frame with its
// marker-detection code.
bitmapData = new BitmapData(VIDEO_WIDTH, VIDEO_HEIGHT);
bitmapData.draw(video);
flarRgbRaster_BitmapData = new FLARRgbRaster_BitmapData(bitmapData);
flarSingleMarkerDetector = new FLARSingleMarkerDetector(flarParam, flarCode, MARKER_WIDTH);
}
/**
* C. Create PaperVision3D's 3D tools including
* a scene, a base node container to hold the
* 3D Model, and the loaded 3D model itself.
**/
private function preparePaperVision3D() : void
{
// Basics of the empty 3D scene fit for
// FLAR detection inside a 3D render engine.
basicRenderEngine = new BasicRenderEngine();
flarTransMatResult = new FLARTransMatResult();
viewport3D = new Viewport3D();
flarCamera3D = new FLARCamera3D(flarParam);
flarBaseNode = new FLARBaseNode();
scene3D = new Scene3D();
scene3D.addChild(flarBaseNode);
// Load, scale, and position the model
// The position and rotation will be
// adjusted later in method D below.
collada3DModel = new DAE();
collada3DModel.load(COLLADA_3D_MODEL);
collada3DModel.scaleX = collada3DModel.scaleY = collada3DModel.scaleZ = MODEL_SCALE;
collada3DModel.z = 5; //Moves Model 'Up' a Line Perpendicular to Marker
collada3DModel.rotationX = -90; //Rotates Model Around 2D X-Axis of Marker
collada3DModel.rotationY = 180; //Rotates Model Around 2D Y-Axis of Marker
collada3DModel.rotationZ = 90; //Rotates Model Around a Line Perpendicular to Marker
// Add the 3D model into the
// FLAR container and add the
// 3D cameras view to the screen
// so the user can view the result
flarBaseNode.addChild(collada3DModel);
addChild(viewport3D);
}
/**
* D. Detect the marker in the webcamera.
* If found: move, scale, and rotate the
* 3D model to composite it over the marker
* in the user's physical space.
**/
private function loopToDetectMarkerAndUpdate3D(aEvent : Event) : void
{
// Copy the latest still-frame of the webcam video
// into the BitmapData object for detection
bitmapData.draw(video);
try {
// Detect *IF* the marker is found in the latest still-frame
if(flarSingleMarkerDetector.detectMarkerLite(flarRgbRaster_BitmapData, DETECTION_THRESHOLD) &&
flarSingleMarkerDetector.getConfidence() > DETECTION_CONFIDENCE) {
// Repeatedly Loop and Adjust 3D Model to Match Marker
flarSingleMarkerDetector.getTransformMatrix(flarTransMatResult);
flarBaseNode.setTransformMatrix(flarTransMatResult);
basicRenderEngine.renderScene(scene3D, flarCamera3D, viewport3D);
}
} catch(error : Error) {}
}
}
}
Stepping through the document class
ActionScript 3 class basics
ActionScript 3 class files are structured into several sections, and each has a specific role in this project.
- Imports:Typically your class will take advantage of existing codelibraries—libraries you may not have coded yourself. Each importstatement loads in the needed class or package from an existinglibrary. You need libraries from Flash, FLARToolkit, and Papervision3Dfor this project.
- Class definition: The bulk of your classes code should be in the class definition section.
- Class properties:Declare all public, private, and static variables. Arguments and localvariables are declared within the constructor body and the methodbodies. For this project, you need only private variables; no outsideclasses need access.
- Constructor: Thisfunction runs when the class is created. The function's name must matchthe class name. It runs only once so it's an ideal place to initializeyour class. In the following steps, setup code will be placed here. Inthis project, code is placed to set up the camera, FLARToolkit, andPapervision3D and to prepare a repeating loop to handle markerdetection.
- Methods: Methods are classfunctions or reusable chunks of code. This project uses only fourmethods to handle setup and marker detection.
Imports and properties
Startingat the top of the AugmentedReality.as document class, we see importstatements for the needed libraries. Within the Properties section,variables are declared. Most of this section is pretty standard, exceptthe Pattern and Params property declarations from lines 58–62. Thissection showing the Flex Embed metatag within Flash CS4 appears here:
[Embed(source="../assets/FLAR/FLARPattern.pat", mimeType="application/octet-stream")]
private var Pattern: Class;
[Embed(source="../assets/FLAR/FLARCameraParameters.dat", mimeType="application/octet-stream")]
private var Params : Class;
Adobe Flex uses a special type of code called the Embed metatag.Embed tags, as they are commonly called, enable a property to beinitialized with a value loaded from an external file. A unique aspectis that the byte loading happens at compile time (as the project ispublished) rather than at runtime (while the user interacts with theapplication). The bytes for the loaded file are baked into the SWFfile. Flash CS4 now supports this feature as well with help from theflex.SWC, which is included in this project's libs folder.
Class constructor
Sincethe constructor of a class is called only once, it is an ideal locationfor one-time-only setup code. Starting on line 93, the preparationmethods are called for each of the following: the webcam, the markerdetection, and Papervision3D. In addition to the setup methods, the loopToDetectMarkerAndUpdate3Dfunction is set up to run again and again. By listening to theENTER_FRAME event built into Flash, this loop will be called repeatedlyas long as the application is running.
Webcam setup
Flashfeatures great webcam support. As the application runs and a camera isdetected, Flash Player will automatically prompt the user to enable itas line 96 of the code executes. The Video object is the windowonscreen where the video will be shown, and the Camera object recordsand feeds its live footage to that Video object.
FLAR setup
Acornerstone of this project is marker detection. As the marker graphicis detected via webcam, its position, rotation, and scale in the user'sphysical space are calculated. This calculation will be used in thenext steps. Starting with line 97, FLAR will search each still-frame ofwebcam footage for the predefined marker graphic (Pattern object). Thedetection scheme uses an ActionScript BitmapData object, which containsonly the latest still-frame of video at any given time (the BitmapDataDraw function).
Papervision3D setup
For the mostpart, the 3D setup at line 98 is typical of any Papervision3D project.A BasicRenderEngine object will handle the bulk of the vector math,converting 3D geometry to 2D graphics for display on the screen. TheCollada 3D model is loaded and inserted into the 3D scene, and the viewinto that scene (ViewPort object) is added to the Stage (see Figure 3).Luckily this is all done for you. In the final project, this view islayered on top of the 2D webcam Video object shown earlier so thecomposite of 2D and 3D will look believable.
Repeated detect-and-update routine
As mentioned earlier, by using the ENTER_FRAME you ensure this loopToDetectMarkerAndUpdate3Dfunction on lines 102 will be called 30 times per second. Each time,the BitmapData object will copy in the latest still-frame from thewebcam footage. FLAR runs detection on the still-frame. If the markeris detected, FLAR returns updated position, rotation, and scale data(TransformMatrix object). The container (FLARBaseNode object) holdingthe 3D model is updated to that position so the model appears to be atthe same position in the user's physical space as the marker. Theresulting effect is nothing short of amazing for first-time viewers.
Running the completed project
Printout the graphic marker. Show the marker to the running application viathe webcam (see Figure 4). The red line is for illustration purposes todenote successful initial marker detection. Much like ArnoldSchwarzenegger's Terminator character, the project can now read in thesurrounding environment, visually scan and detect relevant objects, andmake calculations based on that information. It works best to have asimple backdrop of a solid wall in the application because complexscenes may confuse the detection scheme.
Next, seeing the marker graphic through the webcam, the applicationsuperimposes an animated 3D model onto the user's physical space (seeFigure 5). The model now rotates and scales believably as the usermoves the marker.
Where to go from here
There you have it. Augmentedreality is within the grasp of all Flash developers. The FLARToolkitcan detect multiple markers to bring even more 3D into your world. Astechnology matures, you will be able to detect more complex graphicmarkers within more complex scenes. The choices are endless. What willyou choose to do with augmented reality: print and show a marker to anonline application as we've done here, buy and play with a real-worldAR-enabled toy, or detect and destroy like the Terminator?
To see another example of augmented reality, check out the MAX 2009 example with links to AR sessions at the conference.
Note: This article appeared originally in the Edge newsletter.
More Like This
- Controlling the appearance of text elements with the Text Layout Framework
- Using the Flash OSMF Media Player template
- Preloading TLF runtime shared libraries in Flash Professional CS5.5
- Getting started with the Adobe SiteCatalyst extension for Flash Professional CS5
- Optimizing content for Apple iOS devices
- Using screen orientation APIs for smartphone application development
- Guide for Apple App Store submissions
- Saving state in AIR applications for iOS devices
- Creating ActionScript 3.0 components in Flash – Part 2: MenuBar component prototype
- Exploring the new 3D features in Flash CS4 Professional
| 04/23/2012 | Auto-Save and Auto-Recovery |
|---|---|
| 04/23/2012 | Open hyperlinks in new window/tab/pop-up ? |
| 04/21/2012 | PNG transparencies glitched |
| 04/01/2010 | Workaround for JSFL shape selection bug? |
| 02/13/2012 | Randomize an array |
|---|---|
| 02/11/2012 | How to create a Facebook fan page with Flash |
| 02/08/2012 | Digital Clock |
| 01/18/2012 | Recording webcam video & audio in a flv file on local drive |
