Refine edges of a Roto Brush selection

Learn how refine the harsh edges of a Roto Brush selection.

Published December 16, 2025

Download files

00:01

Introduction

The Refine Edge tool is a tool designed to work in conjunction with the Roto Brush tool.

And if we look at our current project here you can see I've got a waterfall.

And if I toggle my Switches/Modes here and turn off the effects, you can see this is the original waterfall.

And when I turn my effects back on, you can see I've created an initial Roto Brush selection.

To double check that, I'll go ahead and double click on Layer 1 to open it here in the Layer panel.

Now in the bottom of the Layer panel, you should see this purple line here, letting me know that the propagation is frozen.

I did that so everybody had the same selection that I had originally created.

So, since the initial Roto Brush selection was created

00:44

Preparing an existing Roto Brush selection for refinement

and I want to modify it by using the Refine Edge tool, I'm going to come over here and click the Freeze button to unfreeze those frames.

Now it'll go ahead reanalyze what's going on in the scene.

Now I'm only working on one second of the video because the workflow is the same whether you're doing one second or multiple seconds.

So, let's start by going up to our Roto Brush tool here

01:10

Using the Refine Edge tool to improve edge detail

and clicking and holding, and choose the Refine Edge tool.

Now, if I look in the Composition panel here, you'll see the edges need help.

That is a very rough selection.

So, I'll make sure I'm in the Layer panel here and with my Refine Edge tool selected, if I need to change the brush size, I can hold down Command on the Mac or Control on Windows and click and drag up or down.

I'm going to choose a brush around this size and I'll just click and drag around the edges of my selection like so.

Then I'll go ahead and let go and you'll notice it's going to take a second, but now I get a black and white representation of transparency where it's analyzed the edges and refine them.

So, I like this.

I want to go ahead and search the rest of the frames in the span.

So, I'll press the Spacebar and let the analysis happen.

Now it's going to take a little while to analyze these frames because what it's doing is rather intensive.

So, through the magic of editing, I'll rejoin you here in a couple of seconds.

Once all the frames in the span have been loaded, you should notice playback speeding up significantly to give you a good preview as to what's actually happened with the refined edges.

02:26

Reviewing refined edges in the composition

Now, this looks pretty good in the Layer panel, but let's go back to the Composition panel.

Now you can see the edges of the waterfall look a lot better.

Let's press the Spacebar to watch these frames inside the Composition panel.

And as you can see, it's doing a pretty darn good job.

Yes, I have some black pixels around the edges, but we can fix that here in just a moment.

All right, I'm going to press the Spacebar to stop playback and with Layer 1 selected, I'll toggle my Switches/Modes

02:53

Cleaning up edges with blend modes and layer duplication

and I'll change the blend mode from Normal to Screen.

And that's going to cut out a lot of the dark pixels.

Now I'll select that layer and press Control D on Windows or Command D on the Mac to duplicate, and I'll change the upper most duplicate from Screen back to Normal, and I'll press T to open the Opacity, and I'll crank that down to something like 23% or 25%.

Now this looks much more like a clean selection.

So, to resize this and position it, I'll go to the Parent & Link column,

03:28

Resizing and positioning the refined element

click on the pick whip, and point it to the lower layer which is Layer 2 and I'll select Layer 2, and press S to open the scale, and I'll just click and drag to scale it down.

And once I get something that I think might work, I can go ahead and reposition the element in the scene.

I just want to make sure to grab my Selection tool first and here I'll make sure my Scale is set to something like 50%.

OK and then I'll go ahead and just position this like so.

Here we go.

You know what?

I'll do 55%.

That looks a little bit better... position it and we can press the Spacebar.

And you can see the Refine Edge tool has definitely taken the initial Roto Brush selection and added much more realism to it by analyzing the edge pixels and creating those fine details that could really only be achieved with that tool.

Download a sample file to practice with. This sample file is an Adobe Stock asset you can use to practice what you learn in this tutorial. If you want to use the sample file beyond this tutorial, you can purchase a license on Adobe Stock. Check out the ReadMe file in the folder for the terms that apply to your use of this sample file.

What you learned: How to refine the edges of a Roto Brush selection

Refine the edges of a Roto Brush selection with the Refine Edge tool

Preview the Roto Brush selection on the video clip.
Double-click the video layer in the Timeline panel and, if the Roto Brush selection is frozen, click the Freeze button in the Layer panel to unfreeze the selection.
Select the Refine Edge tool from the Tools panel.
Position the current-time indicator to the first frame of the Roto Brush selection, and set a brush size for the Refine Edge tool.
Set the size of the brush for the Refine Edge tool, and draw a line around the edges of the Roto Brush selection.
Press the Spacebar to start and stop the analysis, and make changes to the selection as needed.
Adjust the Refine Edge Matte options in the Effect Controls panel to fine-tune the edge.

Tip: With any brush tool selected in the Tools panel, press and hold the Control (Windows) or Command (MacOS) key and drag up or down in the Layer panel to increase or decrease the size of the brush.

Adobe Stock Contributor: Dustin
Designer: Ian Robinson
Presenter: Ian Robinson

Try these tutorials with After Effects

Create visual effects and motion graphics.

View all plans

Cookie Settings

General information

Cookies are small text files stored by your web browser when you use websites. There are also other technologies that can be used for similar purposes like HTML5 Local Storage and local shared objects, web beacons, and embedded scripts. These technologies help us do things like remembering you and your preferences when you return to our sites, measure how you use the website, conduct market research, and gather information about the ads you see and interact with.

If enabled:

We can improve your experience by tailoring the site and the content to things we think might be of interest
We can better keep track of your preferences — like what language you prefer to use
We will better understand your likely interests so we can provide you more relevant Adobe ads and content on non-Adobe websites and in non-Adobe apps
It will help us improve the performance of our website and those of our partners who use the Adobe Experience Cloud

If disabled:

We won’t be able to remember you from session to session so the experience may not be tailored to your interests
You’ll still have access to the content of the site but certain features that depend on cookies may not function
You’ll still see ads, they just may not be as relevant to you