Rotoscoping using After Effects and Photoshop

Use the selection and painting tools in Photoshop to efficiently rotoscope. Exchange files between Photoshop and After Effects.

• View full-size on Adobe TV
• View more videos like this

Requirements

To complete the tasks demonstrated in this tutorial, you need the following software and files:

Adobe Photoshop CS4

• Try
• Buy

Adobe After Effects CS4

• Try
• Buy

Sample files

lrvid4107_xp.zip (ZIP, 317MB)

Prerequisite knowledge

Intermediate knowledge of painting and masking inside Photoshop and After Effects.

Rotoscoping using After Effects and Photoshop

Both Adobe® Photoshop® and Adobe After Effects® allow you to rotoscope footage. Rotoscoping is the art of altering footage frame-by-frame. In this tutorial, you will learn the art of rotoscoping using animated paint tools in Photoshop and animated masks in After Effects.

Rotoscoping in Photoshop using the Clone Stamp tool

The Clone Stamp tool in Photoshop allows you to paint over unwanted elements in your footage by copying or cloning from other parts of the frame. To use the Clone Stamp tool:

1. Open a movie file in Photoshop. Choose Window > Animation. You will see a layered timeline in the Animation (Timeline) panel at the bottom of the Photoshop window.
2. To focus on a specific portion of the animation, move the current-time indicator to a point in the Timeline.
3. Move the Work Area Start slider to the beginning of the focus area and the Word Area End slider to the end.
4. To view a looped preview of the selected work area, click the Play button at the bottom of the Animation panel.

   

   Figure 1: Playing a looped preview
5. Use the Zoom tool to enlarge the portion of footage you are working on.
6. Choose the Clone Stamp tool from the toolbox.

   

   Figure 2: The Clone Stamp tool
7. Choose Window > Clone Stamp to open the Clone Source panel. Select Show Overlay to display a transparent image of the source point while you are painting with the Clone Stamp tool.
8. Look for an area of your footage to sample for the source point. Use an area of the footage that does not contain the element you want to remove, but that will match the area you will be painting over. Press Option/Alt-click to set your source point.
9. Line up the source point with the footage you want to remove. Click to set the position relative to the source point.
10. Once you have set your position, paint over the areas of the frame that you want to remove. Move the current-time indicator backwards and forwards through the frames and continue painting to remove all traces of the element.

    Tip: Use the Select Next/Previous Frame buttons at the bottom of the Animation panel to ensure that you have reviewed every frame and painted over every aspect of the element being removed from the footage.

    

    Figure 3: Reviewing footage
11. Change the brush size and brush type as needed to complete the cloning process. You can change the brush diameter and hardness using the Brush Preset picker in the Options bar when the Clone Stamp tool is selected.

    

    Figure 4: Changing the brush diameter and hardness
12. You can review your changes in the Clone Source panel. Click the Toggle The Clone Source Panel icon in the upper-right corner of the Options bar to open and close the panel.

    

    Figure 5: The Clone Source panel

Using animated masks in After Effects

Creating animated masks from imported Photoshop documents is another way to disguise unwanted elements in your footage. The advantage of animated masks is that you don't have to paint frame-by-frame to remove an element from a scene. To create an animated mask:

Importing a document from Photoshop

To create a mask, you first need to import a Photoshop document into After Effects. To import the document:

1. In Photoshop, choose File > Save As to save the movie you were working with in the previous section as a .psd file. When the Save As dialog box opens, select Photoshop as the format and click Save.
2. In After Effects, choose File > Import. When the Import File dialog box opens, select All Acceptable File Types from the Enable menu, select the .psd file you saved in Step 1 and click OK. Note that the Import File dialog box resembles the native OS window, either Finder (Mac OS) or Windows Explorer (Windows). Metadata about the file you selected is displayed in Column or Detail view, including file size, kind, and created and modified dates.

   

   Figure 6: The Layer Options dialog box
3. Once you click OK, the Layer Options dialog box opens. Choose Footage from the Import Kind menu and make sure the Merged Layers radio button is selected under Layer Options. Click OK. The file appears in the Project panel of After Effects.

   

   Figure 7: Import File dialog box
4. Drag the imported file to the New Composition icon in the Project panel. You have successfully created a new After Effects composition that has the same length, frame rate, and size of the source file.

   

   Figure 8: Creating a new composition

Creating a mask

Creating an animated mask in After Effects is an excellent way to rotoscope footage. To create a mask:

1. Press Command/Ctrl+D to duplicate the composition on a new layer. You'll make your adjustments to this layer, so you don't alter the original footage.
2. Move the current-time indicator to the point at which the element that you would like to mask appears. Use the Zoom tool as needed to get a precise view of the element.
3. Trim the layer so that only those frames containing the element are included, by dragging the In and Out points to the correct start and end points in the Timeline.

   

   Figure 9: Expanding to full-screen view
4. Press the tilde key to view the frame in full-screen view. Select Fit Up To 100% from the Magnification Ratio menu in the lower-left corner of the After Effects window to get the maximum screen size possible.
5. Choose the Pen tool from the toolbox.

   

   Figure 10: Selecting the Pen tool
6. Begin drawing around the part of the footage that contains the element you want to mask. Note that your drawing here does not have to be precise. A simple outline around the element will do.
7. Add a second mask (Mask 2) that separates the set from the elements you want to keep in your footage. Make sure you do not include any points from the first mask when drawing the second mask.

   

   Figure 11: Drawing animated masks
8. Use the tilde key to get out of the full-screen view so you start to add keyframes and animate the second mask layer.

   Tip: You can use the arrow-shaped Pointer tool from the toolbox to drag the anchor points of the second mask's outline and adjust its position.

Animating the mask

Because you are working with moving footage, the masks you create in your composition also have to move. To animate a mask:

1. Click the colored square icon next to one of the masks. A Mask Color dialog box opens.

   

   Figure 12: Changing the color of a mask
2. Select a color that differs from that of the other mask and click OK. You will now see the two mask outlines in two different colors in the Composition panel.
3. Click the triangle next to Mask 2, the mask you are going to use to hide another footage element, to reveal the mask properties. The property you will be animating is the Mask Path.
4. Move the current-time indicator to the beginning of the trimmed piece of footage and click to add a keyframe to the layer in the Timeline representing the Mask 2 Mask Path property. Then move the current-time indicator to the Out point of the trimmed layer and click again to set a second keyframe.
5. Make sure the second Mask Path keyframe is selected, zoom out and double-click the mask. Drag it to move it to the right, outside the footage area into pasteboard. You have just animated the mask. Scrub the current-time indicator to see Mask 2 move.
6. To ensure that this mask moves at the same rate as the rest of the elements in the footage, you can add an ease. To add an ease, choose Animation > Keyframe Assistant > Easy Ease Out.

   

   Figure 13: Adding an Ease
7. Adjust the starting and ending keyframes as needed to ensure that Mask 2 stays in line with the elements it contains.
8. Set the mask mode of Mask 2 to Subtract by clicking the menu next to the mask's name in the Timeline panel.
9. Collapse the Timeline so that only the layer names are showing. Select the layer containing the masks.
10. To remove the element in Mask 1 from the footage, hover the cursor over the layer outside of the trimmed portion. A double-ended arrow allows you to slide the timing of the layer. Slide the layer such that the element in Mask 1 is entirely removed from the trimmed section of the footage.
11. Scrub the Timeline to ensure that the element is no longer included in your footage. You may need to nudge some keyframes to ensure the correct position of the masks.
12. To reduce the harshness of the masking effect even further, reveal the Mask Feather property for Mask 2 by clicking the arrow next to the mask name.

    

    Figure 14: Revealing Mask properties
13. Adjust the Feather value to 25 pixels.

    

    Figure 15: Adjusting the feather value

Exporting an After Effects composition to Photoshop

In some cases, you may need to export an After Effects file back to Photoshop for further modifications. To export your composition:

1. Choose Composition > Make Movie.
2. Choose a lossless render option that does not add any compression and that is compatible with Photoshop. The default Lossless setting meets these criteria.
3. You can now transfer compositions seamlessly between After Effects and Photoshop as many times as necessary.

Where to go from here

• Photoshop CS4 Help and Support
• Learn Photoshop CS4
• Introducing Photoshop CS4, by Deke McClelland
• Introducing After Effects CS4, by Chad Perkins
• Exporting from After Effects to Flash , by Todd Perkins
• Lynda.com

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License

About the authors