Requirements

Prerequisite knowledge

To benefit from this article, you should know how to achieve uniformly colored shading using your 3D program's material system and how to output it using the renderer. You should also know how to view, separate, reassign, and blend channels in After Effects using the respective effects.

User level

Intermediate

How can you take the output of your 3D applications into Adobe After Effects? This two-part series shows you how.

In Part 1 you saw how to export UV data from your 3D apps for use in After Effects. In Part 2, I show you how to create mattes in your 3D applications that you can export to After Effects. Multichannel OpenEXR files are an ideal format for storing your mattes.

This article concludes by showing you how to install and use the OpenEXR plug-ins from fnord to enable you to handle this format in After Effects.

Using mattes to isolate areas of your image

A regular issue you will face when integrating 3D rendered imagery with other work is how to separate elements. Why would you want that? There are many reasons.

First, clearly you do not intend to process parts of the image that you don't want processed. Returning to the UV pass example and its use inside After Effects as explained in the UV maps section of this series, in Figure 1 you can see that the UV pass not only covers the filmstrip but also the spool, which you don't want to cover up. You somehow need to cut away the parts that you don't need.

Second, you will often use identical instances of the same object in your 3D scene. Since they are merely clones of an original, they will all have the same textures. As a result, they also share the same UV information and other material attributes (see Figure 2). When you only want to change a few of them, you need to isolate them somehow.

The last, and perhaps the most common, application of generating mattes in 3D programs is selective color correction. By isolating areas, you can exactly control the color and shading in parts of the image while leaving the rest unchanged. This can be important when, for example, trying to match a specific color in a logo animation or integrating synthetic elements with existing camera footage.

Obviously, you can achieve that to some degree using masks or paint tools, but not only are these methods not very precise, they are also prone to errors when working with complex setups and long sequences. They're also quite tedious and not very elegant, so I'll focus on using mattes that you can generate from your 3D programs.

Alpha mattes

The most straightforward way of getting a clean matte is to simply contain it in the alpha channel. Using either object-specific settings, materials, or render overrides, many 3D programs enable you to manipulate the alpha channel of the resulting rendered image where everything is visible in the RGB image, but only one object contributes opaque areas to the alpha channel. An example would be Matte Object settings in Lightwave 3D or the Compositing Tag in Cinema 4D.

Object ID/matte passes

The second, and probably most widely used, approach is based around creating a set of specific matte passes. These can either be separate image sequences or embedded into multichannel file formats such as RLA, RPF, IFF, OpenEXR, and to some extent even PSDs.

This method has the advantage of not needing an extra render process, but requires the 3D program to provide special means of designating these mattes. This is done either through the material system or by marking the respective scene items using special attributes. Examples for this kind of matte would again be Cinema 4D's Compositing Tag that allows you to assign an Object ID, Maya's Render Layers, and 3DStudio MAX's Matte Materials.

If special tools for matte creation are not available in your 3D program, the alternative way is to set up multiple scenes with the same objects, but to use the standard material system to give them black and white colors according to which object is supposed to be isolated. As a result, you will end up with extra image sequences. Obviously, this requires a lot of discipline and thinking ahead and has the disadvantage that you need to do it all over again. Once the mattes have been generated, you can use them in the following ways in After Effects:

  • If the files are already separate images or image sequences, simply place them in your composition as you would other footage.
  • If the mattes are part of a layered PSD file or PSD sequence, it is easiest to import the sequence as a composition using File > Import and choosing As Composition – Cropped Layers. After the composition has been created, you can easily arrange the layers in the right order and remove the ones that you don't need. This method is particularly recommended if you are really pushing it and are using all possible passes because it saves a lot of time.
  • If the mattes are part of other multichannel files, you need to apply utility effects to extract them. For RLA, RPF, and IFF, this can be done using the ID Matte effect, which can be found in the Effects menu under the 3D Channel category. Apply the effect and, in the ID Selection, choose the matte channel as shown in Figure 3. Be aware that, depending on the program, the number and order of available channels may vary.
  • For multichannel OpenEXR files, you need to use a special free plug-in. For information on how to obtain, install, and use it, refer to the Using the OpenEXR plug-ins from fnord section of this document.
  • When you have successfully extracted your mattes, you can apply them as luma mattes or alpha mattes. To do so, place them above the UV pass (see Figure 4). Select the UV layer and, from its blending mode options, choose the correct matte type. If you followed the procedures correctly, your setup should look like Figure 5.

RGB mattes

A somewhat old-fashioned, but nonetheless very effective, method is the use of so-called RGB mattes. Instead of just using the alpha channel or a simple black and white image, this technique uses the color channels of an image. For each object or surface in the scene, an exactly defined color is used. In addition to the base colors (red, green, and blue), mixed colors with fixed ratios can be used. As shown in Figure 6, full red (100% red, 0% green, 0% blue) is used for the floor and yellow (100% red, 100% green, 0% blue) is used for the cube.

In order to extract a proper matte from this mix of colors, you need to manipulate and remap the channels. The simplest way to do that is to use the Shift Channels effect. It can be found in the Channels category. To turn the green channel into a matte for the cube, you select it as the alpha channel. Turning the other channels off is optional, but it helps to visualize things (see Figure 7).

In theory, there is an infinite number of mattes you can create this way, but there is one caveat: As soon as you start using colors that do not have the full 100% on one channel, you need to use additional effects. If you use, say, 50% red, 0% green, and 50% blue, you will get a medium magenta (see Figure 8). When extracting a channel directly to the alpha, it will only be half-opaque. Therefore you need to apply an Alpha Levels effect to expand the ranges (see Figure 9).

Another problem is that mixed colors often share color components with other colors. Returning to the example, that means that if you have, for instance, yellow (100% red, 100% green, 0% blue), cyan (0% red, 100% green, 100% blue), and magenta (100% red, 0% green, 100% blue), there will always be two areas that cannot be isolated in a single step. The solution to this problem lies in combining the different extracted mattes against each other.

So, for instance, subtracting a matte generated based on the red channel from the matte on the green channel will remove the yellow areas completely but leave the cyan parts intact. Ideally, you would create all those mattes in subcompositions to be able to use them any place you need them (see Figure 10).

However, as you can imagine, this can become quite complex and it will be hard to keep track of the various compositions. Even if you only use the three base colors and the three mixed colors already mentioned, you will need at least six additional compositions in your project. At some point it may simply become impractical, which you can certainly imagine if you look at Figure 11.

In such situations it would be more manageable to create extra mattes using the other methods.

Concluding thoughts on mattes

Although they do work, all of the above methods have one very big potential drawback: There is a good chance they might degenerate your alpha channels, meaning that when you combine mattes over and over, pixels at the edges between transparent and opaque areas will show the wrong opacity after a time, leading to fringes or gaps. This is due to the antialiasing employed by your 3D program plus the matting procedures of After Effects. With some careful planning, many of the problems can be minimized, however. Here are some simple things you can do:

  • Render your footage with straight (unmultiplied) alpha channels.
  • Use 16 bpc or even 32 bpc image formats, even for mattes. This will give better transparency handling when expanding and compressing levels.
  • When you render extra passes, turn off antialiasing and dithering when possible. UV passes, normal passes, object buffers, and a few others do not need it anyway.
  • Do not render with motion blur and depth of field.

If the above tips are not applicable (for example, because you cannot influence the way your 3D program generates those mattes), then try to tackle the issues in After Effects:

  • Make sure your footage interpretation is correct. There is a crucial difference between premultiplied and straight (unmultiplied) alpha, and setting this wrong can mess up things from the start.
  • If needed, manually unmultiply your channels by using effects such as Shift Channels, Channel Combiner, Channel Mixer, Compound Arithmetic, or third-party plug-ins.
  • Using the same effects, create clean edges on your mattes by removing the antialiasing.
  • Use the Minimax and Matte Choker effects with very small settings to erode offending areas or restore edges that have been thinned down in a matting operation.

Also, experiment with other tools and particularly blending modes. For example, if you plan to use Add or Screen blending modes, that will make black areas disappear anyway, so you do not need to spend hours trying to remove black fringes on a matte. Remember: everything is allowed as long as the result is what you expect.

Generating RGB and alpha mattes in Maxon Cinema 4D

Using RGB and alpha mattes in Maxon Cinema 4D is not really the preferred way of working because Cinema 4D provides a much more elegant way with its multipass rendering and object buffers. (See the Rendering object buffers in Maxon Cinema 4D section of this document.) However, you can still use RGB and alpha mattes in Cinema 4D, and it's easy to set up.

Alpha mattes

The "retro" way of generating multiple alpha mattes is to render the scene multiple times. By using the visibility dots in the Object Manager to hide or reveal items, it is possible to exclude them from appearing in the alpha channel and the RGB information (see Figure 12). This also includes lights and special scene objects, so it is possible to create different versions easily. The downside, of course, is that you will spend a lot of time toggling the visibility for each pass.

To facilitate the process, you should use the layer system introduced in Cinema 4D R10. By assigning them to different layers (see Figure 13), you can generate different render passes easily. Simply toggle the renderability attribute of the layer as you would with the object or hierarchies.

RGB mattes with materials

Rendering a separate RGB matte or luma matte pass can be done by assigning differently colored materials. To give solid colors, you need to only use the Luminance channel, but have all other shading channels turned off (see Figure 14).

Generating different mattes from one scene is achieved either by saving different versions of it or making use of the layer system again. For the latter, you will have to duplicate the hierarchy and assign different materials to the cloned objects (see Figure 15). At first sight, this has no benefits over the other method, but it could be easier to manage if your scene contains animation that is used in every layer and you only need to isolate other parts not affected by this.

As a last way, you can render a RGB matte or luma matte by using the Compositing Tag's Matte Object option (see Figure 16).

Rendering object buffers in Maxon Cinema 4D

This method is of particular interest, as it is very flexible without having to create separate scenes. Tagging the objects is very straightforward. All you need to do is apply a compositing tag and enable the object buffer output (see Figure 17). Since the number of available slots is limited, you will want to label objects only where you are certain that they need to be treated further in compositing. Also, keep in mind that this is not an exclusive setting: objects can be part of multiple object buffers depending on your needs.

When using multipass output, you need to consider a few things to use object buffers successfully in After Effects. First and most important, similar to the issue discussed in the Outputting a UV pass from Maxon Cinema 4D section of this series, the object buffer will inherit transparencies. Turn it off if it stops you from getting the matte that you need (see Figure 18).

Second, and just as critical, you cannot use the Multi-Layer File option (see Figure 19). Unfortunately, it cannot output to multichannel OpenEXR. As with other formats, such as PSD, you will run into problems because the data will be stored as extra channels, not as separate layers. These custom channels cannot be extracted directly from the PSD file in After Effects. More or less, such a file would be useless to you.

When you have successfully set all options, you should end up with a clean black and white object buffer that can be used as a luma matte to isolate the respective areas of your other passes (see Figure 20).

An additional advantage is that object buffers can also be used with the standard render output if you choose a format that supports multiple channels, like RLA or RPF (see Figure 21). They can later be extracted in After Effects using the ID Matte effect in the 3D Channel category.

Generating mattes in NewTek Lightwave 3D

Because it was originally developed as a content generation program for NewTek's VideoToaster product range, NewTek Lightwave always had multiple ways of manipulating the alpha channel directly without additional steps so you could load your rendered Targa files on the Toaster and play them back immediately. Parts of this system still can be used to your advantage, as well.

A very simple and straightforward way of generating a matte is to use the Matte Object option on the Render tab of the Object Properties panel (see Figure 22). Either you assign a solid color to it to use it for an RGB matte or luma matte (see Figure 23) or you play with the alpha settings.

A setting similar to the per-object alpha can be found in the Surface Editor. Using the alpha options on the Advanced tab, you can control the way that transparencies are evaluated. For a clean matte, set it to Constant and dial in a value between 0 (no alpha) and 255 (full alpha), as shown in Figure 24.

When you view the alpha channel, areas occupied by an object or a material on parts of the object will appear black (empty) even if they show up in the RGB channels (see Figure 25).

Generating RGB mattes in Luxology modo 301

At this point, generating mattes in Luxology modo is a mainly manual process. The software lacks a native render layer system, and therefore, render passes always are tied in with the whole scene, requiring the user to hide and show relevant items and shaders manually.

Surface ID pass

A built-in means of generating an RGB matte is the Surface ID pass. You have used this pass already in the general Using mattes to isolate areas of your image section of this document. The Surface ID pass assigns a color out of a predefined palette of colors that are either pure base colors (red, green, blue) or a mix of them with a fixed percentage. Which color is assigned to which surface is based on the order in which the materials in your scene were created and assigned. There is no way to change this post facto.

In order to make use of the Surface ID pass, you need to add it as an additional Render Output to the Shadertree. Simply click the respective button and pick it from the list (see Figure 26).

All additional render outputs will be added as Final Color outputs, so you need to switch them to whatever you require by using the right-click mouse menu (see Figures 27 and 28). In case of the Surface ID, no additional steps are required except for setting an output path, if you so desire. Dithering, clamping, and all the other settings have no influence on the content of the pass, and therefore can be left at their defaults, unlike in the UV pass process described in the Outputting a UV pass from Luxology modo 301 section of this document.

RGB mattes with materials

Just like in any other program, you can always generate an RGB matte conventionally by using materials and setting their properties to give a flat shading. As in Lightwave, materials in Luxology modo are properties of a given set of polygons, not the object containing the polygons. Therefore it is easily possible to mix multiple materials on one object by defining selections.

To do so, draw a selection in Polygon mode and then assign a new material. The default keyboard shortcut for this is M; the internal command is called poly.setMaterial. When the dialog box appears (see Figure 29), you are given the opportunity to define a few basic parameters, which at this point in particular have relevance for discerning polygon areas in shaded mode and can later be refined in the Shadertree for rendering.

Once the new material/shader group has been assigned, it will show up in the program's various lists (see Figure 30). This will not only serve as a criterion for rendering, but also enable you to recall those selections with a click of the mouse for modeling operations.

One thing that is slightly different about modo's materials is that the renderer is based to some degree on correct physical principles, as opposed to the simplified shading algorithms that have been common in the industry until a few years ago. Because of that, it is slightly more difficult to coerce it into rendering a flat-shaded pass because it wouldn't be possible in a real environment.

To avoid having to tweak too many parameters, I will pick out the luminosity parameter. It functions based on the energy emission of the surface and therefore operates in floating-point values. At first, this is confusing, but for your purposes there is one simple rule: any value beyond 1.0 W/srm2 will register as a fully opaque color (see Figure 31). This is because the energy will exceed the color range that is of interest to you, and you therefore will have no use for higher values because you don't intend to create your mattes with float values.

Even though the luminous shading will also affect the Final Color pass, you cannot use it for creating your mattes. Luckily, modo makes provision for rendering the luminous shading to a separate pass, also. You just need to add it in the usual manner already explained (see Figures 32 and 33).

Selection-based mattes using shader group masks in Luxology modo 301

In order to make use of shader group masks, you must have polygonal selections in your scene. Matters are a bit more complex, however, as there are multiple ways to store those selections persistently. In addition to vertex maps (UV sets, weight maps, vertex color maps, etc.), modo uses so-called polygon tagging (PTags) to associate custom information with geometry. In theory, there can be an infinite number of them, just as there could be an infinite number of vertex maps, but the three types implemented in modo are more than sufficient.

Preparing the geometry

The first, and the one you will encounter most frequently, is the Material. By default, all polygons will have a Default material, but by following the procedures laid out in the Generating RGB mattes in Luxology modo 301 section of this document, you can assign a custom material to any polygon selection.

The second method involves creating Parts. With the polygons selected, you go to the menu and choose Geometry > Polygon > Set Part. The internal command is poly.setPart.

Creating Selection Sets is the last option. As the name implies, it is meant as a helper utility during modeling, so it can be found in the menu under Select > Assign Selection Set. In addition to polygon selections, point and edge selections can also be stored this way, but the latter two are of no use for rendering. Be careful to not mix them up. The command to access this function via scripts and macros or assign it to a shortcut key is select.editSet.

Once you have created your custom tags, they will show up in the respective subsections of the Statistics window (see Figure 34).

A special case of material selections are the Item Masks. This is a function that can be found in the context menu of the Item List (see Figure 35). As I will illustrate later, it adds a dedicated group mask that includes all items on a Mesh layer or geometry group.

Creating the group masks in the Shadertree

When you use materials and item masks, you do not need to take any additional steps to propagate them to the Shadertree. Luxology modo will take care of everything and create a group mask and a material contained inside it.

However, all of the groups created this way will reside in the root level of the Shadertree. This is OK most of the time for simple scenes, but for more complex setups and for controlling the matte, you will often want to create nested groups and organize things manually. You can do so easily by selecting the Group entry from the Add Layer popup menu (see Figure 36). Depending on what you had selected before, the new group will be created inside an existing one or at the root level. You can easily reorder it by dragging and dropping, which is also the means of choice to put other groups in place.

In hierarchical groups it is crucial to set them all to the correct PTag types in order not to break the evaluation during rendering. It is not enough that only the nested groups have the correct properties; all groups above them need to share the same settings to allow pass-through. The other way around, you can use the Apply to Sub-Group switch to force lower-level groups to inherit the settings of the parent group.

Figures 37–47 show this process for each of the types. For illustration purposes, red and blue materials were used, where red matches the selection illustrated in the viewport. Gray areas are the background or the default material.

Item masks

 

Materials

 

Selection sets

 

Parts

 

Once the group mask hierarchies have been created, you can combine them as you see fit by simply toggling their visibility (see Figure 48). Because they are merely masks, you are free to do whatever you want inside them. Most of the time you will simply use materials to create luma mattes or RGB mattes, but of course you can also add separate shaders, textures, or render outputs.

Using the OpenEXR plug-ins from fnord

Many 3D programs will allow you to render your images to the OpenEXR format. Using it in After Effects is possible ever since version 7. Unfortunately, After Effects does not directly support multichannel OpenEXR files that can contain additional channels like depth, object IDs, UVs, ambient occlusion, and so on all in one file (similar to RLA/RPF files). Programs like Luxology modo are able to generate these channels in multichannel OpenEXR files and, as you may well imagine, this capability greatly facilitates managing those files.

In order to use multichannel OpenEXR files in After Effects, you need a freely available third-party plug-in set:

fnord OpenEXR plug-in

The plug-ins can be used with After Effects. To install and use them without getting a conflict with the program's own file I/O plug-in, you first need to disable the native After Effects OpenEXR plug-in. This can be done most safely by putting parentheses around the name of the plug-in, which means After Effects will ignore it. Assuming you installed it in the default directory on Windows, the plug-in is located in the following folder:
C:\Program Files\Adobe\After Effects CS3\Support Files\Plug-ins\Format

On Mac OS, the corresponding directory can be found here:
HD:Applications:Adobe After Effects CS3:Support Files:Plug-ins:Format

Simply rename OpenEXR.8bi to (OpenEXR.8bi). After that, you can extract the plug-ins from the downloaded ZIP files and place them in the Plug-ins directory. I recommend that you create a dedicated subfolder for them. Then you can launch After Effects.

The effect in which you are most interested is called EXtractoR, and it will show up in the 3D Channel category of the Effects menu (see Figure 49). It is applied directly to layers based on multichannel EXR footage. By default, it will show you the RGB and alpha channels, and therefore not look much different than other footage. To use other channels, you click in the info area.

On doing so, you are presented with a dialog box that allows you to remap other custom channels to the RGB and alpha channels (see Figure 50). The names of the channels may look confusing at first. Beyond the initial channels, no fixed naming convention exists. Therefore, programmers are free to name additional channels whatever they like. Since most likely you will only work with one or two 3D programs at a time, you will have learned to make sense of this quickly.

A special case is presented with the (copy) option. This will simply copy values from the previous channel or use the default channel. This can be useful, for example, if you want to extract only the depth channel and need a clean grayscale representation on a single channel (see Figure 51).

All the changes are properly reflected on the interface, of course. Depending on the data you chose, the other controls may be of interest to you. If you choose an object buffer, for example, you may wish to create an alpha channel from it instead of using it as a luma matte. This can be done by simply using the UnMult option.

Since many channels in EXR files are stored as HDR floating-point numbers, their ranges are not always normalized to fall between 0 and 1. This allows them to retain all the fine details and exposure. However, it may not always be possible to work with such data directly. Therefore, you can change the minimum and maximum range by adjusting the Black Point and White Point.

Depending on what channels you extract, it will be necessary to work in 16 bpc or 32 bpc to get access to the full value ranges, so make sure to set your project to the appropriate bit depth. This will always avoid wrong results due to conversion errors and clipping when going from floating-point 32 bpc to fixed-point 8 bpc, for example.

Where to go from here

Now that you have an understanding of how to generate mattes in your 3D applications, and have seen how to use the fnord plug-ins to import your mattes as multichannel OpenEXR files in After Effects, you're equipped to use these techniques in your After Effects projects.