Table of contents

Created

25 March 2013

Requirements

Prerequisite knowledge

This tutorial is designed for developers with an intermediate to advanced understanding of ActionScript 3, object oriented programming, and the Away3D engine.

Additional required other products

User level

Intermediate

Every 3D engine needs to solve a very basic problem: what’s under the mouse pointer? In 3D graphics, this is commonly referred to as picking. Although it may sound like a simple and straightforward problem, it actually involves nontrivial mathematics, requires very efficient algorithms, and is definitely much more complex than its 2D counterpart. Detecting what object is beneath the cursor in a 3D scene can be a programmatically expensive task. But don’t worry–Away3D offers a series of carefully crafted features to deal with this problem, taking into account Flash Player capabilities and limitations, and giving the user plenty of flexibility to find the best balance between precision and performance for each particular implementation.

This article explores the Away3D engine’s picking feature set, shows how easy it is to use these features, and provides a basis for mastering their subtleties to achieve an efficient implementation on advanced and demanding projects.

Hello picking

To get started, consider the 3D scene in Example 1.

Note: The full source code for this example and subsequent examples is available in the sample files for this article.

The key parts of the code are:

anObject.mouseEnabled = true; anObject.addEventListener( MouseEvent3D.MOUSE_OVER, onObjectMouseOver ); private function onObjectMouseOver( event:MouseEvent3D ):void { // do stuff… }

It is as simple as that.

Of course, you can use all the other mouse event types, just like in 2D. The idea is that you know the interface and are already familiar with it. There is nothing new here. But it is 3D picking though, and things can get more complicated, as I noted before.  You’ll see that more clearly when you look under the hood and learn what’s going on in the 3D engine.

Entity properties

Picking in Away3D can be managed by a set of properties within 3D entities.

MouseEnabled

So, how is the engine doing the picking? Basically, the engine is casting a ray from the camera, through the mouse position on the screen, and straight into the scene (see Figure 1). This ray could potentially hit several objects within the scene. One way to determine which objects are hit is using raw ray collision mathematics. This involves calculating ray-sphere collisions, ray-aabb collisions, ray-triangle collisions, and so on. The most expensive part of this process is finding out if the ray hits the inside of a triangle in a mesh, which could potentially contain a high polygon count (or poly count).

Ray tracing is not the only method used for picking. Away3D in fact has two different methods and uses this one by default for reasons I will cover later in this tutorial.

When you set mouseEnabled to true on any object, you make it eligible for these collision evaluations. This property is set to false by default on all objects in order to avoid potentially complex calculations.

It is possible to have an object mouse-enabled but with no listeners attached to it. The result of this is simply occlusion. The object collides with the ray, but triggers no mouse events. Example 2 shows this in action. It uses the same basic code as Example 1, but the attachment of listeners to one of the objects is commented out.

If the occluding object was not mouse enabled, the picking ray would just pass through it, ignoring it completely. Again, this is the default behavior for all scene objects unless you specifically set the mouseEnabled property to true. This is pretty much the way it works on the regular Flash 2D display API, but in that case objects are mouse-enabled by default because picking is so much cheaper in two dimensions.

Picking collider

You may have noticed with the previous examples that (after rotating the camera to specific angles) picking around the sphere is not very accurate. In Example 1 or 2, if you hover the mouse pointer close enough to the sphere, it will trigger a mouse over, even though the mouse is not actually over the object. That is not good.

The engine sets the picking precision for all objects to BOUNDS_ONLY by default (see Figure 2). In Figure 1, the ray is being "stopped" at the mesh’s bounding volume surface and collision calculations with its potentially hundreds or thousands (or even millions) of triangles is avoided by default. If you do need more precision, you just need to tell the ray to keep going. You change this using the pickingCollider property:

anObject.pickingCollider = PickingColliderType.BOUNDS_ONLY; // default //anObject.pickingCollider = PickingColliderType.AS3_FIRST_ENCOUNTERED; //anObject.pickingCollider = PickingColliderType.AS3_BEST_HIT; // etc…

BOUNDS_ONLY is the computationally cheapest picking collider that the engine offers and is accurate enough for many cases. If you want more precision, you can choose a different one:

  • PickingColliderType.BOUNDS_ONLY (default value) - Calculates ray vs. bounding volume collisions.  
  • PickingColliderType.AS3_FIRST_ENCOUNTERED - Calculates ray vs. all mesh triangles collisions. This costs considerably more than BOUNDS_ONLY  and is more expensive as the number of triangles in the mesh increases. The collider stops all tests as soon as a collision with a triangle is found, despite the fact that the hit surface may not be the mesh's closest surface along the ray to the camera.
  • PickingColliderType.AS3_BEST_HIT - Like the previous collider, but more detailed, evaluating which of the triangles that collide with the ray is closest to the ray's origin. This is necessary if you want to accurately know the collision point on a mesh that may have more than one possible collision surface with the ray, such as a mesh representing a coffee cup, for example. AS3_FIRST_ENCOUNTERED could incorrectly resolve a collision with the inner walls of the cup whereas this collider will not stop tests on the first hit, but will instead find all collisions and evaluate the best one overall.  
  • PickingColliderType.PB_FIRST_ENCOUNTERED - This is the same as AS3_FIRST_ENCOUNTERED  but uses Pixel Bender instead of pure ActionScript for ray collision calculations. This method proves to be faster for higher-poly meshes and slower for lower-poly meshes. Pixel Bender takes advantage of multithreading if it can, meaning that such a collider will tend to be faster on desktops, but may not be on simpler CPUs like those found on mobile devices. Furthermore, AIR for iOS simply does not support Pixel Bender at this time, which causes this collider to fail silently.  
  • PickingColliderType.PB_BEST_HIT - Pixel bender equivalent to AS3_BEST_HIT.
  • PickingColliderType.AUTO_FIRST_ENCOUNTERED - Like AS3_FIRST_ENCOUNTERED, but automatically decides whether to use the ActionScript or Pixel Bender version of the collider depending on the poly count of the mesh.  
  • PickingColliderType.AUTO_BEST_HIT - Same as AUTO_FIRST_ENCOUNTERED  but with the behavior of AS3_BEST_HIT

As you can see, there are many options! Deciding which picking collider to use requires an understanding of what each available type does so that you can choose the best balance between performance and precision. For instance, you would likely choose PB_BEST_HIT on an application that paints on a high poly mesh where you want very accurate picking precision, but would definitely not choose such a collider for a game where you need to be able to click on a low poly mesh that occupies a small portion of the screen. Such a poor choice would be a waste of resources, especially if you enable this type of mouse interactivity on many low poly objects. To solve the problem illustrated in Figure 2, you could just set a different collider type on the object:

sphere.pickingCollider = PickingColliderType.AS3_FIRST_ENCOUNTERED;

Or, you could simply change the sphere's bounding volume to a BoundingSphere instead of an AxisAlignedBoundingBox (Away3D sets all bounding volumes to AxisAlignedBoundingBox by default). This would allow you to get the precision you want without the need to enter the volume and perform costly ray triangle collision calculations. Of course, this is a very specific, almost theoretical case, but it is important to keep in mind that the core of this picking technique is bounding volumes, and different bounding volume shapes can be very helpful. In code, this is how you change a bounding volume:

sphere.bounds = new BoundingSphere();

Either way you choose to resolve the precision problem, the result can be appreciated in Example 3.

View properties

In addition to object properties, picking in Away3D also involves global properties.

Shader vs. raycast mouse pickers

As I mentioned earlier, Away3D offers a completely different approach for picking calculations. So far you have been learned about using the properties of individual objects, but you can also change the way picking works globally; for example:

view.mousePicker = PickingType.SHADER;

You can use any of the following PickingType values:

  • PickingType.RAYCAST_FIRST_ENCOUNTERED (default value) - Uses ray tracing as a picking method, stopping tests at the first successful renderable level collision.
  • PickingType.RAYCAST_BEST_HIT - Uses ray tracing as a picking method, evaluating the best (closest) hit between all the colliding renderables.
  • PickingType.SHADER - Uses a shader technique as a picking method, always evaluating the best hit.

The difference between first two options is quite subtle, so for now focus on the difference between the first two and the third option, which represents a completely different picking technique. While the first two are based on evaluating collisions between a ray and mathematical entities within the scene, the third option (SHADER) does not evaluate ray-geometry collisions at all. Instead it renders a portion of the scene around the mouse with specific colors into a temporary buffer, and then analyzes the color directly under the mouse to find which object is hit. This technique is harder to visualize, and is in fact what most high-end 3D engines use, because it has proved to be efficient both in accuracy and performance. Unfortunately, this technique is not as performant in Stage3D at this time. This is because it involves transferring images from the GPU to the CPU via the Context3D drawToBitmapData() method, which is slow. For more detail on this, see Jackson Dunstan's article on Stage3D Readback Performance. The method introduces a severe bottleneck, which is why Away3D offers a raycast alternative and sets it as the default method. The RAYCAST method proves to be faster than the SHADER method in Flash Player. Keep in mind that you can only have one of the two methods per view, but not both. So, at this time you cannot have shader based picking on some objects and raycast based picking on other objects.
 
You may be wondering why Away3D still offers shader based picking if it is slower. It is because shader picking is still necessary in some situations. For example, it is needed when dealing with GPU animations and transformations, in which geometry can be altered after the CPU stage, in the GPU, where the raycast approach could not possibly determine where the vertices are, and will not be able to properly calculate ray-geometry collisions. Since the shader method is not ray-geometry collision based and deals with what is actually visible on screen, it is the method of choice when dealing with animated objects where high picking precision is desired. Example 4 shows the effect of using ray versus shader based picking on an animated object. This example uses a sphere to trace where the mouse ray hits the mesh, as well as a line segment to trace the mesh's normal at that point.

Notice how the raycast method does not see the GPU level vertex transformations and instead hits the static ghost mesh. This problem is eliminated with the shader method. Furthermore, the shader based method ignores the .pickingCollider property of objects since this property is only relevant for the raycast method. The shader picker does, however, care about the object's .shaderPickingDetails property:

anObject.shaderPickingDetails = true;

This property simply asks the shader picker to evaluate not just whether the mouse events exist, but also the data involved in them such as position, normals, and so on. Calculating this information is not free as it is in the raycast approach, so you should only request it when needed. If this property is set to false, the mouse events will trigger, but some of the event properties will be null or invalid. The value is set to false by default. In Example 4, it is true so that the code can retrieve the location of the mouse event in the scene when using the shader picker.

The two raycast mouse pickers

Now that you know more about the shader picker, I want to return to the difference between the two available raycast mouse pickers, RAYCAST_FIRST_ENCOUNTERED and  RAYCAST_BEST_HIT. These are actually the same picker, but with slightly different settings. The differentiation exists for the same reason best hit and first encountered options exist in an object's picking collider, that is for stopping the collision test as soon as it is needed in terms of precision. In this case, the test stops checking on the criteria of renderables instead of triangles.

  • PickingType.RAYCAST_FIRST_ENCOUNTERED (default value) - Uses ray tracing as a picking method, stopping tests at the first successful renderable level collision.  
  • PickingType.RAYCAST_BEST_HIT - Uses ray tracing as a picking method, evaluating the best (closest) hit between all the colliding renderables.  

But what is a renderable? In short, it is something that can be drawn to screen. The raycast picking system relies on an entity's bounding volume, but an entity can contain multiple submeshes. These submeshes share a single bounding volume. A submesh is a renderable. Submeshes exist because Stage3D limits the number of elements that can be put in a single buffer. So, if the poly count increases beyond this limit in a mesh, a new set of buffers is created within a new submesh object. Such buffers represent vertices, normals, uv coordinates and so on. With RAYCAST_FIRST_ENCOUNTERED, the picking system does not care about which colliding subrenderable is closest to the camera and stops collision checks as soon as a positive collision is found.

RAYCAST_BEST_HIT, in contrast, finds all collisions amongst renderables and picks the one closest to the camera. It exists for the case in which large meshes are subdivided into multiple submeshes.

This is a subtle difference, and to be honest, one not easy to grasp, but it's good to be aware of it. Use of RAYCAST_BEST_HIT is typically rare, but it is there in case it is needed. For example, you will not have a problem if you align a set of cubes along the path of the mouse ray; RAYCAST_FIRST_ENCOUNTERED will do just fine with that. However, if you import a mesh that contains a set of aligned cubes into the same geometry (admittedly, a weird case), you will need RAYCAST_BEST_HIT for proper picking. Example 5 illustrates this. I added another cone tracer pointing towards the hit location just to help show where it is.

As you can see, if the mouse ray hits more than one submesh with RAYCAST_FIRST_ENCOUNTERED, it will fail to identify the appropriate collision. However, RAYCAST_BEST_HIT always finds the right one. Notice how I used an AS3_FIRST_ENCOUNTERED collider on the mesh; otherwise I would be checking collisions with its bounding volume and not with the triangles of its submeshes, which produces even worse results! Also, notice how I shuffled the z positions of the submeshes. If I didn’t do this, the order of the submeshes in the tree would coincide with the order in which they collide with the ray, hence producing correct results by mere coincidence.

So far you have seen the three available mouse pickers, as well as the seven available picking colliders for independent objects. That covers most of the topic of picking in the Away3D engine, and it provides a great deal of flexibility in setting up mouse interactivity as efficiently as possible. Picking can be confusing if you don’t understand what each feature does, but once you do, the decision becomes pretty obvious for each implementation

UV painting

There is one more aspect of the picking system that you should be aware of: the MouseEvent3D class. The scene position property of MouseEvent3D used in Examples 4 and 5 made it possible to position a tracer mesh where the mouse ray collides with the object under it. MouseEvent3D also offers a series of other helpful properties, including:

  • scenePosition - The position of the event's collision in scene space.
  • localPosition - Same as scenePosition but in the hit object's local space.
  • sceneNormal - The object's normal at the point of collision, in scene space.
  • localNormal - The object's normal at the point of collision, in object space.
  • uv - The interpolated uv coordinates at the point of collision. This will not be available with raycast bounds collisions and shader collisions with details disabled.
  • screenX and screenY - The position in screen space of the mouse event.
  • material - The material of the colliding renderable.  

Example 6 uses the scenePosition property, as well as the sceneNormal and uv properties to draw a line segment representing the event's normal and to paint on the object's material.

In this example, I’ve used PB_BEST_HIT as the mesh’s picking collider. This seemed appropriate given the considerable poly count of this mesh.

As you can see, MouseEvent3D provides useful information on the involved event, providing all the information needed for complex mouse interaction with the 3D scene.

Where to go from here

As you have seen, 3D picking is a complex topic. It must be properly understood and mastered to develop quality 3D applications. Fortunately, Away3D offers a flexible feature set for this important task.

To explore the Away3D engine further, visit the growing Away3D tutorials wiki.
The Away3D forum is another great resource with an active community.
Also, be sure to check out the GitHub repository of Away3D examples.