All Articles After Effects
After Effects: The Roto brush Tool
Richard Lainhart on Wed, March 14th 0 comments
This excellent tutorial on using Adobe After Effect's Roto brush tool is from the unpublished collection of Hub articles by Richard Lainhart, RIP. This is an essential article on compositing. Enjoy.

Rotoscoping, the art of tracing moving characters and objects in film and video footage to isolate them from their backgrounds, has a long history, going back to the earliest days of cartoon animation. (For more about rotoscoping, check out my HUB article "A Brief Technical History of Compositing".

Nowadays, rotoscoping is mainly used in situations where characters need to be isolated but weren't originally shot against green or blue, which would allow them to be easily color-keyed. (Rotoscoping, in fact, is how many current films that were originally shot in 2D are converted into a form of 3D for theatrical screening.)  In such cases, the only choice is generally to trace the outlines of the characters by hand, often frame by frame, which can be a terribly tedious and time-consuming job. 

But there are some tools available to us that can make jobs like that go a little smoother. In this article, I'm going to introduce you to After Effects' Roto brush tool, which can largely automate the process of rotoscoping and let us easily create mattes for compositing footage that isn't suitable for color-keying. Let's get started.

Step 1 -  Set Up Your Project 

For best results with the Roto brush tool, your footage should have its subject well-outlined against a plain solid background, with high contrast between the foreground and background. I'm going to be using a short shot of me speaking against a plain white background, which I've imported and dragged onto the Create a New Composition button to add to a new comp:

Even though this shot is only about 3.5 seconds long, it's still over 100 frames, so it's something we definitely wouldn't want to trace by hand. And even though my head is well-defined against the background, the shot isn't well-lit enough to allow for color or luminance keying. Here's what happens when you try:

trying luminance keying

So, it's on to the Roto brush tool.

Step 2 -  Select and Apply the Roto Brush Tool

With your new comp active, select the Roto brush tool:

Rotobrush tool

Double-click with it on the image in the Comp window to open the footage's Layer window (most Roto brush work happens in the Layer Window).

Step 3 -  Draw Your First Stroke

Next, locate your base frame (a frame where the subject is clearly defined against the background) where you'll be drawing your first Roto brush stroke. The first frame of your shot often works well as a base frame, and that's what I'm going to be using here. 

Now, drag the Roto brush tool (which should display as a green circle with a plus in it) down through the center of your image from edge to edge, as shown (you don't use the Roto Brush tool to draw around the outline of the image, as you would with the Pen tool)...

An outline

...and release the mouse button to create the segmentation boundary, the magenta line that will define your subject's mask:

Define the subject's mask

As you can see, the Roto brush has pretty smoothly outlined at least part of the image. Our next step is to add to this basic boundary to outline the entire image.

Step 4 -  Add to the Segmentation Boundary

Still working in our Base Frame, position the Roto brush cursor in the areas of the subject that weren't outlined with the first stroke, and just drag through them to add to the boundary until you get a solid magenta line that completely outlines the subject:

A complete outline of the subject

Step 5 -  Refine the Segmentation Boundary

Now you can see, in a couple of places at the top of my head, that the Roto brush has selected a little too much of the background, which won't give up a clean matte. However, if we hold down the Option key (the Roto brush tool will change to a red circle with a minus sign in it) and drag in those areas (zooming in can help here), we can tighten up the boundary by subtracting those areas from the selection:

Subtract unwanted areas

My wispy hair is a bit of a problem here, but we can improve that a little later. For now, let's leave it as is. Continue on if necessary by adding to and subtracting from the Base Frame's boundary until you have a clean, solid selection. Click the Toggle Alpha:

Toggle Alpha

and Toggle Alpha Overlay:

Toggle Alpha Overlay

These will show us our alpha channel alone and the subject with a red overlay in the background, respectively, so we can check the state of the matte.

Step 6 - Propagate the Boundary

Once the Base Frame matte is to our liking, it's time to propagate the current outline to the rest of the footage. This is where the magic happens: when we propagate the rest of the frames, the Roto brush tool will copy and automatically adjust the boundary to match the movement of the subject. 

Now, the range of frames within which the propagation occurs is called the span, and we adjust the span to set the propagation range. With complex subjects, we often want to set a short span, so we can precisely tweak the segmentation boundary. In this case though, because there's relatively little motion in the clip, we can set the span to cover the entire clip, and tweak later if necessary. 

Drag the span (the row of gray blocks under the Time ruler) out to the end of the Time Ruler, then hit the Space bar to begin the propagation. 

Drag the Span

If the boundary loses track of the subject's outline, just stop the propagation and adjust the boundary as you did on the base frame, backing up in the Time Ruler if necessary, then continue the propagation. Eventually, all your frames will be outlined. 

Step 7 - Freeze the Boundary

The last step in the Layer window is to freeze the segmentation, which will cache the boundary data to disk to both prevent further mask updating and to speed up interaction with the comp. To do that, just click the Freeze button at the bottom of the window:

Click the Freeze button

After a bit, you'll see an indication in the Layer window that the segmentation is frozen and you can close the Layer window.

It's frozen

Step 8 - Composite the Layer and Adjust the Matte

Our final step is to composite the layer we just masked and tweak the edge of its alpha channel to smooth it against the background. In my case, I've added a layer of flame imagery behind my masked head to make the edge clear:

Smoothing it out

As you can see, the masked edge of my head is pretty hard and doesn't blend smoothly with the background. To clean that up, we'll use the Roto brush's Matte tools which, since the Roto brush is an effect, are found in the Effect Controls window for the selected layer:

The Roto Brush will smooth it out

Smooth, Feather, and Choke work interactively to refine the edge of matte:

Smooth, as you'd expect, reduces the roughness of the matte edge; Feather softens the matte edge by making it more transparent, so it blends more into the background; Choke pulls the edge of the matte in towards the center of the subject to tighten it up. 

Adjusting these three together will give you a much smoother final composite:

And there you have it.  

The Roto brush tool is one of After Effects' lesser-known features, but a powerful one that anyone doing compositing special effects will find useful. I haven't gone into all the details of the tool by any means, so as always, I encourage you to experiment. And above all, have fun!

Related Videos
Comments (0)

You must be logged in to comment.

Motion Tracking and Stabilization
After Effects CS5 201
Dream It. Do It.
Do you want to learn Motion Tracking and Stabilization?
Yes, I want to learn!
No Thanks, I just want to read the article.
Course Advisor
Don't Know Where To Start?
Ask A Course Advisor
Ask Us!
Copy the link below and paste it into an email, forum, or Facebook to share this with your friends.
Make money when you share our links
Become a Affiliate!
The current affiliate rate is: 50%
Classes Start Next Week!
Live 8-week Online Certification Classes for: