Accurate profiling with Adobe Scout

From the moment you start using Adobe Scout, you're given access to a huge amount of data. This touches on almost every aspect of your content; from ActionScript execution to the individual rendering steps that Flash Player performs. It's tempting to jump in head first, looking for performance problems in your content, but before you do so it's important to understand how this data is collected.

Every piece of data that Flash Player sends to Scout has a cost. In addition to executing your content, Flash Player has to spend time measuring itself and transmitting this data. This means that Flash Player behaves differently when you’re profiling with Scout, compared to its normal execution. Depending on what data you choose to collect, the difference could be negligible or really significant. This may or may not be acceptable, depending on the question you’re trying to answer.

In this article, you’ll learn how Flash Player measures and sends data to Scout, as well as how to decide what data to collect. This will help you to more accurately answer questions about the behavior and performance of your content. By having a much better understanding of what the numbers mean in Scout, you'll increase your confidence when profiling.

Note that this isn’t a guide to the user interface of Scout. If you don’t know how to use Scout, or what the different panels do, then you should begin by reading the getting started guide.

When you start using Scout, you'll notice that as soon as you load any Flash content, a new session opens up and data starts to arrive. This data is called Telemetry. It’s basically a stream of measurements that Flash Player takes while executing the content. For this to work, Scout has to be able to tell Flash Player what data to send, and Flash Player has to be able to connect to Scout and send the data.

How does Scout tell Flash Player to send data?

There's a file named .telemetry.cfg in your home directory, which Scout uses to tell Flash Player what to do. This file stores information about which machine Scout is running on, what port to use, and what data Flash Player should gather (that is, what you selected under Settings for New Sessions in Scout). If you're running Scout on the same machine as Flash Player, then you should never need to touch this file. Scout writes to it whenever you change any settings, and Flash Player will read it whenever it loads a new SWF. You can't change the settings once a session has already started in Scout; they only apply to new sessions.

If you run Scout on a different machine, and want Flash Player to send data to it, you need to manually set up your .telemetry.cfg file. The getting started guide explains how to do this. Remember that you need to configure Scout to listen on the same port as you specify in this file. You can set this in the Preferences dialog box in Scout.

If you want to profile Adobe AIR content on a mobile device, you can download the Scout Companion App for iOS, Android, and Kindle from the appropriate app store. The app connects to an instance of Scout, and communicates with Scout to determine what data settings you selected. This process doesn't use .telemetry.cfg. Instead, the companion app directly relays this information to running instances of Adobe AIR, so they know what data to send to Scout, and where to send it.

How does Flash Player send data to Scout?

When Telemetry is enabled (that is, there's a .telemetry.cfg file in your home directory), Flash Player tries to open a TCP connection to Scout whenever it loads a new SWF. If this fails, Flash Player disables Telemetry and continues running, and your content doesn't experience any profiling overhead. If the connection succeeds, then Flash Player starts streaming data to Scout, which shows up as a new session. The data is sent in a compressed binary format called AMF, which Scout parses before presenting to you. When you save an FLM, you're actually saving the original data that Flash Player sent, in the same binary format. This means that when you open an FLM, Scout is really just replaying the data it received from your content, in fast forward.

Every piece of data that Scout receives has to be sent by Flash Player. The more data you ask for, the more additional work Flash Player has to do, and the more this impacts the performance of your content. Because of this, it's important to think about the type of question you want to answer when you decide what data to collect, so you know which boxes to check under Settings for New Sessions (see Figure 1).

If you want to measure performance, you should limit yourself to gathering low-overhead data: the Basic Telemetry, ActionScript Sampler, and CPU Usage categories. These have been carefully engineered so that Flash Player spends as little time as possible collecting and sending the data to Scout, compared to executing your content. This means that you can trust the time measurements you see in Scout, which is essential when you’re looking for performance problems!

If you want to gather detailed information about rendering, you can also select the settings with a high overhead: DisplayList Rendering Details and Stage3D Recording. These are really useful for debugging graphics-related problems; they let you see the DisplayObjects drawn on each frame and the precise sequence of Stage3D commands executed, respectively. This level of detail comes at a price though. Flash Player has to spend much more time gathering and transmitting the data to Scout, which slows down your content. If you decide to gather this data, then the timing information you see in Scout won't be accurate. Some execution times will appear longer than they really are, because of the extra work Flash Player is doing. Not only that, but the results will be biased because some activities are measured in more detail than others.

Remember, if you want accurate performance data, keep the overhead low! You can’t get detailed rendering data and accurate time measurements simultaneously. It’s like trying to interview Usain Bolt while he’s running the 100m. He’s not going to break any world records while you’re prodding him with a microphone!

Basic Telemetry and ActionScript Sampler

When you're interested in performance, you should turn on the Basic Telemetry and ActionScript Sampler settings. They actually collect two different types of data, which can be a bit confusing at first. When Flash Player executes your content, there are two core pieces that work together:

1. Flash Player, which does the real work and interfaces with the outside world. Among other tasks, it renders to the screen and plays sound and video, as well as handling input events and networking.
2. Your SWF, which is basically a bundle of static resources, such as images, and dynamic resources, such as scripts and timelines. These dynamic resources are executed by Flash Player, and tell it what to do.

The Flash Player codebase is instrumented so that it measures itself in key places. It measures how long it spends executing key blocks of code, and how much memory is used by its key data structures. The important word here is "key". If Flash Player measured the time spent in every single function, it would have too much overhead, and run much slower than when it wasn’t being profiled. Because it only measures activities that are known to take a significant amount of time, the overhead of these measurements is negligible. In other words, the times you see in Scout are as accurate as possible. This data is called Basic Telemetry, and it’s used by Scout in drawing the Frame Timeline, and in the Summary, Activity Sequence, and Top Activities panels. Note that Flash Player only reports activities that take longer than five microseconds; anything shorter than this won’t show up in Scout.

Scout also displays information about the execution of ActionScript 3 in your SWF. You can see it in the ActionScript panel when you turn on the ActionScript Sampler setting. This data is very different from Basic Telemetry. Flash Player doesn’t know ahead of time what your code will be, so it can’t be instrumented in advance. And if Flash Player were to record and send every single function call to Scout, it would add too much overhead to be accurate. Instead, Flash Player samples your ActionScript while it's executing. Every millisecond, it takes a snapshot of the call stack, which represents what your code was doing at that particular instant in time.

In the ActionScript panel in Scout, this data is aggregated over the range of frames you select, letting you see what percentage of time you're spending in each function, on average. This data is only approximate (it's only shown to the nearest millisecond, because that's the sampling frequency), but it gets increasingly accurate the more samples you look at (for example, by selecting more frames). You should always pay attention to the data quality indicator in the ActionScript panel. A sad face (see Figure 2) tells you that you're not looking at enough data for it to be statistically significant, whereas a happy face (see Figure 3) means that the data is accurate enough for you to determine which functions are the most expensive.

Understanding the difference between instrumentation and sampling is very important, since the two types of data have different precision. Imagine you’re on a roller coaster, and want to measure how long the ride lasts. With the instrumentation approach, you carry a stopwatch, start it when the ride starts, and stop it when it ends (see Figure 4). This can give you a fairly precise result (for example, to the nearest second), but it takes time to start and stop the stopwatch, and to make a note of each measurement. If you do this for every ride, it'll make your day at the theme park a little bit longer (assuming you can only do one thing at a time). For this example, let’s say you measure a time of 23 seconds.

With sampling, rather than carrying a stopwatch, you set up a camera to take a photo of the ride every 10 seconds (or whatever interval you choose) (see Figure 5). This is less intrusive, because you don’t have to carry around a stopwatch and take measurements, but it’s also less precise. After the ride is over, you look at how many photos you were moving in (in this case, two), and multiply by the time interval between photos (in this case, 10 seconds). This gives you an approximation of 20 seconds for the ride’s duration.

Sampling is less precise than the instrumentation approach, but the longer you repeat it (going round the roller coaster multiple times), the smaller the relative error will be. This is analogous to what the ActionScript Sampler does. The "photos" are snapshots of the call stack (known as samples), and they’re taken every millisecond.

One helpful feature in Scout is the ability to combine the two types of data in really useful ways. For example, the code in Flash Player for receiving a mouse event is instrumented, so it shows up in Basic Telemetry. This gives you a microsecond-precise measurement of how long it took to handle this event. If you have an ActionScript 3 event handler, then Flash Player also gathers samples of your code while it’s executing. In Scout, when you go to the Top Activities panel and click on the event, Scout will filter the ActionScript panel to show only the code that was running while that specific event was being handled (see Figure 6).

Basic Telemetry and ActionScript Sampler data are also combined in the Summary panel. Most of the data in this panel comes from Basic Telemetry, but if you’ve turned on the ActionScript Sampler, you can expand the ActionScript category to see a breakdown of the packages in which the most time was spent. Just remember that this data is approximate, because it’s computed by sampling (unlike the rest of the data in the Summary panel). This is indicated by the "≈" sign (see Figure 7).

Here’s a summary of the two types of data:

• Basic Telemetry (Activity Sequence panel, Top Activities panel) gives you timing measurements about what Flash Player is doing. The data is obtained by lightweight instrumentation of Flash Player, and it has microsecond precision.
• ActionScript Sampler (ActionScript panel) takes periodic snapshots of what your ActionScript 3 code is doing. Every millisecond, it takes a sample of the call stack, and aggregates this data over time to show you what percentage of time you spend in each function, on average. Times are approximate, but get more accurate the more frames you select, so be sure to look for the happy face!

CPU Usage

If you turn on the CPU Usage setting, Flash Player will take periodic measurements of how much CPU time it’s using. It gets this data by querying the operating system, so it corresponds to the figures you would see in Activity Monitor on OS X, or Task Manager on Windows. It's low overhead, so you can turn this on while you're investigating the performance of your content.

DisplayList Rendering Details

If you use the Flash DisplayList in your content, then the DisplayList Rendering Details setting lets you record detailed information about which portions of the screen get redrawn in each rendering pass, and the time taken to render each DisplayObject. You can see this information in the DisplayList Rendering panel in Scout. In particular, the Heat Map mode shows you graphically which portions of the screen took the longest time to render.

The overhead of collecting this data depends on how many objects are on the DisplayList. This can get quite expensive, so you shouldn’t look at the overall performance of your content with this data setting turned on; only the relative time taken to render each DisplayObject. A good workflow is the following:

1. Profile your content with only Basic Telemetry enabled, and look at how much time it spends doing DisplayList Rendering, compared to other activities like executing ActionScript.
2. If DisplayList Rendering is a problem (because it’s taking up too much time), then turn on the DisplayList Rendering Details setting, and profile your content again. You can then use the DisplayList Rendering panel in Scout to determine the source of the problem.

Stage3D Recording

If you use Stage3D in your content (either directly, or using a framework such as Starling or Away3D), Scout can give you detailed information about what’s going on. The Stage3D API in Flash Player is instrumented so that it can send every Stage3D API call and its arguments to Scout. You can then replay the commands inside Scout, to see how each scene is rendered, one draw call at a time. This works because there’s a copy of the Stage3D engine inside Scout itself.

Turning on the Stage3D Recording setting generates a huge amount of data, and has a high overhead. That's because Flash Player has to send every buffer, texture, and AGAL program over to Scout, along with details about every function call you make to the Stage3D API. This is a lot of data, and collecting and sending it will slow down the execution of your content. It's fine to turn on Stage3D Recording when you want to debug rendering problems, but you should never do performance profiling with this data setting enabled. The ActionScript panel will even warn you about it if you try (see Figure 8).

If you’re concerned about performance, you should carry out the following steps:

1. Profile your content with only Basic Telemetry and the ActionScript Sampler enabled. In the Summary panel in Scout, expand the ActionScript category and look at what proportion of time is spent in the Stage3D APIs. The ActionScript panel will show you which specific function calls took the longest.
2. If your content is spending too much time in the Stage3D APIs, you can turn on Stage3D Recording. Scout will show you every Stage3D API call your content makes, in the Stage3D Rendering panel. This is really useful for finding the cause of performance problems, such as issuing too many draw calls or changing the state of the GPU too often.

If you need to debug a rendering issue, like an AGAL program not working the way you expected, you can just jump to the second step. Remember though, when you want to improve the performance of your content, you should first find out if Stage3D is part of the problem before trying to optimize it!

Because Stage3D Recording sends so much data to Scout, it's easy to use up gigabytes of memory within a matter of minutes. Since you might want to debug rendering problems that happen after your content has been running for a while (for example, determining why the dragon boss on Level 22 doesn’t have a shadow), Scout has a delayed recording mode. If you change the Recording setting from Immediate to Delayed, Flash Player will send the same detailed Stage3D data as before, but it won't get loaded into memory in Scout. When you click start recording in the Stage3D Rendering panel, Scout loads only the necessary data into memory (such as your meshes and textures). This lets you replay the commands from that point onwards, without using as much memory.

You’ve already seen why it’s important to choose the right data settings to profile accurately with Scout. But it’s also vital to consider the environment in which you’re profiling. If you don’t, you could end up getting very misleading data, and waste time trying to fix a problem that isn’t really there! Here are some important tips to consider when you’re doing performance profiling with Scout:

• Profile using a release version of Flash Player. Other versions, like the debugger, have different performance characteristics.
• Profile using a release SWF. Debug SWFs run much slower, because their ActionScript bytecode is heavily instrumented to help you debug your content.
• Profile using the same hardware, operating system, and browser that you expect your users to use. Your content might run well on your super-fast developer machine, but how will it run for the average user?
• Don’t run other programs or SWFs while you’re profiling. This interferes with the measurements that Flash Player makes, and it could end up reporting misleading information. Have as few programs running as possible, and close any other browser tabs you might have open.
• Avoid using VMware or device emulators when profiling. Use a native machine or device if possible, since its performance characteristics will be closest to what your users will experience. If you do need to use VMWare, then make sure you set it to use more than one core. It won’t work properly with Scout if it only has one core (this is a known issue).
• If you’re profiling on mobile, make sure your network has enough bandwidth for Flash Player to send its data to Scout. If not, the network connection will become a bottleneck, and this will give you misleading information in Scout.
  

Now that you understand how the data in Scout gets collected, you can profile more accurately and effectively. Remember to only collect the data you need to answer the question you’re currently addressing. Blindly turning on all the settings will end up skewing the data, and could lead you on a wild goose chase where you end up optimizing the wrong thing!

Once you’ve identified a performance problem, the next step is to determine why there’s a problem, and what to do about it. To learn more about how Flash Player works, and what to do about the information that Scout shows you, see Understanding Flash Player with Adobe Scout.