Mapping Flash video to 3D objects
Modified
9 July 2007
Requirements
Prerequisite knowledge
Basic knowledge of ActionScript 3.0, including how to set up projects and compile SWFs, is required to complete this tutorial. Understanding of 3D modeling is beneficial.
User Level
Intermediate
Required products
- Flex Builder (Download trial)
- Flash Professional (Download trial)
Sample files
- flash_video_3d.zip (44937 KB)
Additional Requirements
Code editor of choice
Flex SDK / ActionScript 3.0 compiler
3D Modeling package Blender, 3D Studio Max, etc. (optional)
Subversion client (optional)
Flash video has made a tremendous impact on the web. From sites like Google Video and YouTube to advertisements, Adobe Flash dominates the web video landscape.
One of the major advantages to deploying video from Flash is the amount of creative control you have over the video. The ability to tightly integrate video with existing content makes Flash a compelling platform for video. With the introduction of a new 3D engine, Papervision 3D, the creative control you can exert over Flash video has gotten a whole lot wilder.
This tutorial steps you through what it takes to map your videos to 3D objects using the Papervision 3D engine. Even though this is a bit more advanced than what you are creating in this tutorial, take a look at my Flash video and 3D demo to get some idea of what you'll be building.
Understanding Papervision 3D
Papervision 3D is an open-source project released under the MIT license. It was started by Carlos Ulloa, who later joined forces with John Grden and Ralph Hauwert. If you haven't had the opportunity to see some of their stunning demonstrations of the engine, I encourage you to seek them out. Some of the most amazing Flash work to date is being done using the Papervision engine.
Affine texture mapping
Papervision 3D includes a typical set of objects you'd expect from a 3D engine: scenes, cameras, meshes, materials, and more—not a whole lot of shock value there. What makes this engine so successful is its method of texture mapping.
Instead of trying to attempt expensive, setPixel-based, perspective corrected texture mapping, the engine uses what is known as affine texture mapping. To understand affine texture mapping, it's easiest to see it in action and in comparison to perspective corrected texture mapping. Figures 1–3 show a sample source image and its mapping using the two techniques.
As you can see from comparing Figures 2 and 3, affine texture mapping is not as accurate as perspective corrective texture mapping. But what you lose in accuracy, you gain in performance—and as you begin to subdivide your meshes, the distortion effects become less apparent (see Figure 4).
Affine texture mapping can be frustrating for those who don't expect its results. It is especially noticeable when text or hard lines are present in the texture. However, considering the performance limitations of a software-rendered 3D engine, it is a reasonable compromise.
Collada support
Another powerful feature provided by Papervision 3D is Collada support—an industry-standard XML schema used to describe a wide range of 3D features. It's used as an intermediate format for sharing assets between 3D packages. Specifically, it was designed to support the handoff between 3D content creation packages—such as Maya, Softimage, and Blender—and runtime 3D engines, such as those used in gaming.
Papervision 3D doesn't support the full range of features described in the Collada schema. However, what it does support is an excellent start and plenty enough for what we need. The team is also hard at work implementing some of the missing features.
Setting up your workspace
As with any new project, a little forethought goes a long way and can save valuable time down the line. Before you get your hands dirty with code, you will need to do a little setup to ensure a clean workspace for moving forward. I use Flex Builder to do my work, but similar considerations apply regardless of your specific development environment.
When you set up your project, it is important to think at a high level about the varying levels of abstraction and responsibility involved. You'll be using two Eclipse projects to divide your workspace for the tutorial at the appropriate levels. The first, a Flex Library Project, manages the Papervision 3D library (see Figure 5). Keeping Papervision 3D in its own project enables you to manage it individually and reuse it as a library across projects.
If you're working with the sample files, you can use the existing project, which has already been set up for you. If you're working from scratch, you'll need to set up the project. First, create the project folder and check out the latest Papervision 3D code from the Subversion repository. I use Tortoise SVN, a popular Subversion client for Windows that integrates with Microsoft Windows Explorer. You are free, of course, to use whatever SVN client suits your needs.
Note: The Papervision 3D library may have changed since this writing which could impact the validity of the sample code.
Typically, I like to place my source in a project subdirectory corresponding to the language of the source code: \as3, \as2, or \java, for example. However, because you are working with a third-party library, it makes more sense to play by its rules and let it govern the directory structure a bit.
The Papervision 3D repository already includes an as3 directory and it does not correspond to the root of the ActionScript source. Because of this, the main source folder in the build path for the library project's properties needs to be changed to as3\trunk\src. When you've done this step, be sure to check the entire contents of the src directory for inclusion in the library.
The second project, an ActionScript Project (see Figure 6), will be for your primary application files.
Again, if you're working from the sample files, you can use the existing project that is included. If not, you'll need to create the project. When creating the new project, set the main source folder to as3 and the output folder to deploy. While I like using bin in typical circumstances, you'll be storing more than just binaries in here. It makes more sense to name it something that better reflects its purpose. You also need to add the Papervision 3D library to the build path. Click the Library Path tab and add the Papervision library project.
With your workspace set up, you are ready to proceed. Let's start by looking at the 3D model.
Building the 3D model
You'll be mapping your video to a custom mesh that I've modeled for you in Blender (see Figure 7). Blender is a free, open-source, 3D modeling, animation, and rendering package licensed under the GNU General Public License. While a simple plane or cube would suffice for your mesh, using a custom model adds visual interest.
Modeling in Blender isn't terribly difficult, but the process is outside the scope of this tutorial. If you are interested in using Blender for your Papervision 3D adventures, refer to the Blender site. They have plenty of tutorials to get you started.
I do want to note one thing before moving on. Before exporting your model from Blender, convert your mesh's quads to triangles. This is done, in edit mode, by selected the vertices and choosing Mesh > Faces > Convert Quads to Triangles. Papervision 3D doesn't seem to know how to handle the UV coordinates in the exported mesh if this step is not done.
Importing and viewing your model in Papervision 3D
It's time to start getting into some code. By now you should have your workspace set up and ready to go. Open your default application class file, Video3D.as. If you're working from the sample files, simply follow along.
If you haven't already formatted the class to your liking, go ahead and do so. If you need some inspiration, look at the provided sample source. You'll need to do a little setup on the Stage object, and what better to handle this than a setupStage method?
/**
* Configures the Stage object
*/
private function setupStage(): void
{
this.stage.scaleMode = StageScaleMode.NO_SCALE;
this.stage.align = StageAlign.TOP_LEFT;
}
If this was a real, commercial project, I would more than likely have an abstract application class that I would be extending. If I didn't have one, I'd be writing one. It would be handling simple responsibilities like this so I wouldn't have to code it over and over again. However, for the purpose of brevity and simplicity, this kind of setup will go right into the default application class.
Now it's really time to get your hands dirty. You need to create the class that is responsible for setting up the appropriate Papervision 3D objects in order to load and render your mesh. Don't get carried away; in honor of the meticulous, take the time to properly package the new class. I've used spitzer.video3d for my package. Short, sweet, and to the point. The class name is equally important. I've chosen Video3DScene.
The Video3DScene class extends Sprite. This is done for two reasons. First, the object will be added as a child to the default application, Video3D. Second, the object will act as a container to which Papervision will render its scene.
The first task the Video3DScene class needs to perform is setting up a core set of Papervision objects. Create a setupScene method for this:
/**
* Sets up and configures the 3D scene.
*/
private function setupScene(): void
{
this.scene = new Scene3D(this);
this.camera = new FreeCamera3D();
This.camera.z = -600;
}
The scene is a container for all of the 3D assets in Papervision. It takes the instance of the Video3DScene class as its single argument and uses it to render its contents to the display list. The camera will be our view into the scene. You'll move the camera away from the screen to ensure the mesh is fully visible.
Note that I haven't shown the member declarations. If you see the use of this, know that there is a declaration for the property. If you're having trouble, refer to the provided sample source.
Papervision 3D includes two types of camera objects. The FreeCamera3D object, which you are using for this tutorial, gives you manual control over how the camera is positioned. The alternative, Camera3D, uses a look-at target that specifies the direction in which the camera points or looks. The look-at target takes the form of a DisplayObject3D and is passed to the Camera3D's constructor. The look-at target is useful for tracking or orbiting around objects in your scene.
The second task the Video3DScene class performs is instructing Papervision 3D to load the Collada file. This is done through Papervision's DisplayObject3D class. A DisplayObject3D object represents a 3D mesh or collection of meshes. Technically, it consists of a collection of faces where each face consists of a collection of vertices.
To perform the task of loading, add a setupCollada method:
/**
* Instructs Papervision to load the Collada file
* which contains the mesh to which video will be
* mapped.
*/
private function setupCollada(): void
{
var material: ColorMaterial = new ColorMaterial(0xcecece);
material.doubleSided = true;
var materials: MaterialsList = new MaterialsList();
materials.addMaterial(material, "Material");
var screen: DisplayObject3D = new DisplayObject3D();
screen.addCollada("screen.dae", materials);
this.scene.addChild(screen, "Screen");
}
Lots of new and definitely interesting code here. Let's step through it.
The first thing that happens is that a material is created. This will be applied to the mesh. To keep things simple, you'll start with a ColorMaterial. Once you ensure that the Collada file is loading properly, you'll move to mapping the video onto the mesh with a BitmapMaterial.
Next, a MaterialsList is created. This list provides all the materials in use within the Collada file. Since there is only one material used, you add only a single material to the list. Note, however, the second argument of the addMaterial method. This specifies the name of the material. This name correlates to the name of the material assigned to the mesh in Blender and exported in the Collada file.
Next, a DisplayObject3D object is created and its addCollada method is called. The methods takes two arguments. The first is the path to the Collada file and the second is the materials list. The last line of code adds the DisplayObject3D object to the scene with the name Screen.
Now it's time to add some interaction. You'll be mapping the rotation of the DisplayObject3D object containing your Collada contents to the mouse. This happens in an ENTER_FRAME event. Add the event handler:
/**
* Handles the ENTER_FRAME event and updates the 3D scene,
* mapping the rotation of the mesh to the mouse.
*/
private function handleEnterFrame(event: Event): void
{
var screen: DisplayObject3D = this.scene.getChildByName("Screen");
var rotationY: Number = -(this.mouseX / this.stage.stageWidth * 720);
screen.rotationY += (rotationY - screen.rotationY) / 10;
this.scene.renderCamera(this.camera);
}
First you ask the scene for the DisplayObject3D object, Screen. Next you calculate a rotation offset, and then you apply the offset to the DisplayObject3D's rotationY property. The division by 10 gives the rotation some deserved style. Finally, you instruct the scene to render the camera.
Very cool. Excited? You should be. It's time to wire things up and test. The Video3DScene class needs a constructor that calls your setup methods and wires up the ENTER_FRAME event:
/**
* Constructor. Configures and kicks off the
* rendering of a 3D mesh with video.
*/
public function Video3DScene()
{
this.setupScene();
this.setupCollada();
this.addEventListener(Event.ENTER_FRAME, this.handleEnterFrame);
}
The default application class, Video3D, also needs some wiring in its constructor:
/**
* Constructor. Configures and kicks off the application.
*/
public function Video3D()
{
this.setupStage();
var scene: Video3DScene = new Video3DScene();
scene.x = this.stage.stageWidth / 2;
scene.y = this.stage.stageHeight / 2;
this.addChild(scene);
}
At this point you should be ready to test your work thus far. Drum roll, please. Not bad, eh? Ready to add some video?
Adding video to the 3D mesh
If you've made it this far, you're obviously committed and probably more than ready to see your videos rocking the face of a 3D mesh. Time to get to work.
Your first task is to prepare your video. Because of security restrictions introduced in Flash Player 9, FLV video content cannot be captured to a bitmap when streamed via RTMP. So instead of streaming your video, embed it into a SWF.
Create a new FLA file and import the FLV video onto the Timeline. Pretty simple, right? Then publish the FLA to the deploy directory with the name video.swf.
To handle the loading of the video SWF, create a new method, setupVideo:
/**
* Loads the video SWF.
*/
private function setupVideo(): void
{
var url: String = "video.swf";
var request: URLRequest = new URLRequest(url);
var loader: Loader = new Loader();
var loaderInfo: LoaderInfo = loader.contentLoaderInfo;
loaderInfo.addEventListener(ProgressEvent.PROGRESS, this.handleLoadProgress);
loader.load(request);
}
This should be fairly straightforward to understand. You set up a URLRequest and Loader object to load the SWF file. Notice the handler for the PROGRESS event; you need to add this method as well:
/**
* Handles the load PROGRESS event from the LoaderInfo object
* associated with loading the video SWF
*/
private function handleLoadProgress(event: ProgressEvent): void
{
var percent: Number = event.bytesLoaded / event.bytesTotal * 100;
if(percent >= 25)
{
var loaderInfo: LoaderInfo = event.target as LoaderInfo;
this.video = loaderInfo.content;
loaderInfo.removeEventListener(ProgressEvent.PROGRESS, this.handleLoadProgress);
}
}
The PROGRESS event handler checks what percentage of the SWF has loaded. If the percent is greater than or equal to 25, the listener is removed and a reference to the loaded content is stored in a member variable. You may have noticed that I neglected to add a handler for the IO_ERROR event type. If this were a commercial project, I would not have left it out. To keep things simple in this instance, I left it out on purpose.
You are so close to having that video mapped onto your mesh. The final step is to replace the ColorMaterial with a BitmapMaterial and update the bitmap contained in the material with a bitmap capture from the video at each frame:
/**
* Instructs Papervision to load the Collada file
* which contains the mesh to which video will be
* mapped.
*/
private function setupCollada(): void
{
//var material: ColorMaterial = new ColorMaterial(0xcecece);
this.bitmapData = new BitmapData(400, 300, false, 0xcecece);
var material: BitmapMaterial = new BitmapMaterial(this.bitmapData);
material.doubleSided = true;
var materials: MaterialsList = new MaterialsList();
materials.addMaterial(material, "Material");
var screen: DisplayObject3D = new DisplayObject3D();
screen.addCollada("screen.dae", materials);
this.scene.addChild(screen, "Screen");
}
Now for the bitmap capture of the video:
/**
* Handles the ENTER_FRAME event and updates the 3D scene
* mapping the rotation of the mesh to the mouse.
*/
private function handleEnterFrame(event: Event): void
{
var screen: DisplayObject3D = this.scene.getChildByName("Screen");
var rotationY: Number = -(this.mouseX / this.stage.stageWidth * 720);
screen.rotationY += (rotationY - screen.rotationY) / 10;
if(this.video != null)
{
this.bitmapData.draw(this.video);
}
this.scene.renderCamera(this.camera);
}
As you can see, you first need to check for the existence of the video object before attempting to capture its contents. The loading of the SWF file is asynchronous and the video object may not be defined by the time the first frame elapses.
You are one line of code away. The last line is a call to the setupVideo method from the Video3DSceen constructor. After adding the method call, test your work. Hooray, you made it!
Where to go from here
Flash, video, 3D: it's a brave new world. It is quite extraordinary to see this kind of thing done in Flash—especially with such excellent performance. A year ago I would have never thought this possible. I have been happily proven wrong. I hope this has been a joyful and educational experience for you.
I encourage you to expand on the idea of using video and 3D together. There are a wide range of possibilities and plenty of opportunities to take Flash to places it has never gone before. Map your video to a sphere or different kinds of meshes, map live video from a user's camera, or sync the audio with the location of the mesh in 3D space. Be creative and don't be afraid to experiment.
For more reading, be sure to visit the Papervision 3D site and wiki:
The wiki, in particular, has several links to demonstrations and additional tutorials. You can also find more examples here:
- Flash 9 phong shading (Ralph Hauwert)
- 3D bump-mapped objects demo (Ralph Hauwert)
- First Papervision 3D game (John Grden)
More Like This
- Exploring the new 3D features in Flash CS4 Professional
- 3D view controller in Flash using ActionScript 3
- Using the FLVPlayback component with Flash Player 9 Update 3
- 3D moving stars in Flash using ActionScript 3
- Flash CS4 Missing Manual excerpts: Video, testing and debugging, optimization, and sound
- Designing and animating characters in Flash – Part 2: 2.5D animation
- Web video template: Spokesperson presentation with synchronized graphics
- Flash video template: Video presentation with navigation
- Examining the ActionScript 3 Flash video gallery source files
- Building streaming video players in Flash with Open Source Media Framework
| 04/23/2012 | Auto-Save and Auto-Recovery |
|---|---|
| 04/23/2012 | Open hyperlinks in new window/tab/pop-up ? |
| 04/21/2012 | PNG transparencies glitched |
| 04/01/2010 | Workaround for JSFL shape selection bug? |
| 02/13/2012 | Randomize an array |
|---|---|
| 02/11/2012 | How to create a Facebook fan page with Flash |
| 02/08/2012 | Digital Clock |
| 01/18/2012 | Recording webcam video & audio in a flv file on local drive |
