28 January 2013
From the moment you start using Adobe Scout, you're given access to a huge amount of data. This touches on almost every aspect of your content; from ActionScript execution to the individual rendering steps that Flash Player performs. It's tempting to jump in head first, looking for performance problems in your content, but before you do so it's important to understand how this data is collected.
Every piece of data that Flash Player sends to Scout has a cost. In addition to executing your content, Flash Player has to spend time measuring itself and transmitting this data. This means that Flash Player behaves differently when you’re profiling with Scout, compared to its normal execution. Depending on what data you choose to collect, the difference could be negligible or really significant. This may or may not be acceptable, depending on the question you’re trying to answer.
In this article, you’ll learn how Flash Player measures and sends data to Scout, as well as how to decide what data to collect. This will help you to more accurately answer questions about the behavior and performance of your content. By having a much better understanding of what the numbers mean in Scout, you'll increase your confidence when profiling.
Note that this isn’t a guide to the user interface of Scout. If you don’t know how to use Scout, or what the different panels do, then you should begin by reading the getting started guide.
When you start using Scout, you'll notice that as soon as you load any Flash content, a new session opens up and data starts to arrive. This data is called Telemetry. It’s basically a stream of measurements that Flash Player takes while executing the content. For this to work, Scout has to be able to tell Flash Player what data to send, and Flash Player has to be able to connect to Scout and send the data.
There's a file named
.telemetry.cfg in your home directory, which Scout uses to tell Flash Player what to do. This file stores information about which machine Scout is running on, what port to use, and what data Flash Player should gather (that is, what you selected under Settings for New Sessions in Scout). If you're running Scout on the same machine as Flash Player, then you should never need to touch this file. Scout writes to it whenever you change any settings, and Flash Player will read it whenever it loads a new SWF. You can't change the settings once a session has already started in Scout; they only apply to new sessions.
If you run Scout on a different machine, and want Flash Player to send data to it, you need to manually set up your
.telemetry.cfg file. The getting started guide explains how to do this. Remember that you need to configure Scout to listen on the same port as you specify in this file. You can set this in the Preferences dialog box in Scout.
If you want to profile Adobe AIR content on a mobile device, you can download the Scout Companion App for iOS, Android, and Kindle from the appropriate app store. The app connects to an instance of Scout, and communicates with Scout to determine what data settings you selected. This process doesn't use
.telemetry.cfg. Instead, the companion app directly relays this information to running instances of Adobe AIR, so they know what data to send to Scout, and where to send it.
When Telemetry is enabled (that is, there's a
.telemetry.cfg file in your home directory), Flash Player tries to open a TCP connection to Scout whenever it loads a new SWF. If this fails, Flash Player disables Telemetry and continues running, and your content doesn't experience any profiling overhead. If the connection succeeds, then Flash Player starts streaming data to Scout, which shows up as a new session. The data is sent in a compressed binary format called AMF, which Scout parses before presenting to you. When you save an FLM, you're actually saving the original data that Flash Player sent, in the same binary format. This means that when you open an FLM, Scout is really just replaying the data it received from your content, in fast forward.
Every piece of data that Scout receives has to be sent by Flash Player. The more data you ask for, the more additional work Flash Player has to do, and the more this impacts the performance of your content. Because of this, it's important to think about the type of question you want to answer when you decide what data to collect, so you know which boxes to check under Settings for New Sessions (see Figure 1).
If you want to measure performance, you should limit yourself to gathering low-overhead data: the Basic Telemetry, ActionScript Sampler, and CPU Usage categories. These have been carefully engineered so that Flash Player spends as little time as possible collecting and sending the data to Scout, compared to executing your content. This means that you can trust the time measurements you see in Scout, which is essential when you’re looking for performance problems!
If you want to gather detailed information about rendering, you can also select the settings with a high overhead: DisplayList Rendering Details and Stage3D Recording. These are really useful for debugging graphics-related problems; they let you see the DisplayObjects drawn on each frame and the precise sequence of Stage3D commands executed, respectively. This level of detail comes at a price though. Flash Player has to spend much more time gathering and transmitting the data to Scout, which slows down your content. If you decide to gather this data, then the timing information you see in Scout won't be accurate. Some execution times will appear longer than they really are, because of the extra work Flash Player is doing. Not only that, but the results will be biased because some activities are measured in more detail than others.
Remember, if you want accurate performance data, keep the overhead low! You can’t get detailed rendering data and accurate time measurements simultaneously. It’s like trying to interview Usain Bolt while he’s running the 100m. He’s not going to break any world records while you’re prodding him with a microphone!
When you're interested in performance, you should turn on the Basic Telemetry and ActionScript Sampler settings. They actually collect two different types of data, which can be a bit confusing at first. When Flash Player executes your content, there are two core pieces that work together:
The Flash Player codebase is instrumented so that it measures itself in key places. It measures how long it spends executing key blocks of code, and how much memory is used by its key data structures. The important word here is "key". If Flash Player measured the time spent in every single function, it would have too much overhead, and run much slower than when it wasn’t being profiled. Because it only measures activities that are known to take a significant amount of time, the overhead of these measurements is negligible. In other words, the times you see in Scout are as accurate as possible. This data is called Basic Telemetry, and it’s used by Scout in drawing the Frame Timeline, and in the Summary, Activity Sequence, and Top Activities panels. Note that Flash Player only reports activities that take longer than five microseconds; anything shorter than this won’t show up in Scout.
Scout also displays information about the execution of ActionScript 3 in your SWF. You can see it in the ActionScript panel when you turn on the ActionScript Sampler setting. This data is very different from Basic Telemetry. Flash Player doesn’t know ahead of time what your code will be, so it can’t be instrumented in advance. And if Flash Player were to record and send every single function call to Scout, it would add too much overhead to be accurate. Instead, Flash Player samples your ActionScript while it's executing. Every millisecond, it takes a snapshot of the call stack, which represents what your code was doing at that particular instant in time.
In the ActionScript panel in Scout, this data is aggregated over the range of frames you select, letting you see what percentage of time you're spending in each function, on average. This data is only approximate (it's only shown to the nearest millisecond, because that's the sampling frequency), but it gets increasingly accurate the more samples you look at (for example, by selecting more frames). You should always pay attention to the data quality indicator in the ActionScript panel. A sad face (see Figure 2) tells you that you're not looking at enough data for it to be statistically significant, whereas a happy face (see Figure 3) means that the data is accurate enough for you to determine which functions are the most expensive.
Understanding the difference between instrumentation and sampling is very important, since the two types of data have different precision. Imagine you’re on a roller coaster, and want to measure how long the ride lasts. With the instrumentation approach, you carry a stopwatch, start it when the ride starts, and stop it when it ends (see Figure 4). This can give you a fairly precise result (for example, to the nearest second), but it takes time to start and stop the stopwatch, and to make a note of each measurement. If you do this for every ride, it'll make your day at the theme park a little bit longer (assuming you can only do one thing at a time). For this example, let’s say you measure a time of 23 seconds.
With sampling, rather than carrying a stopwatch, you set up a camera to take a photo of the ride every 10 seconds (or whatever interval you choose) (see Figure 5). This is less intrusive, because you don’t have to carry around a stopwatch and take measurements, but it’s also less precise. After the ride is over, you look at how many photos you were moving in (in this case, two), and multiply by the time interval between photos (in this case, 10 seconds). This gives you an approximation of 20 seconds for the ride’s duration.
Sampling is less precise than the instrumentation approach, but the longer you repeat it (going round the roller coaster multiple times), the smaller the relative error will be. This is analogous to what the ActionScript Sampler does. The "photos" are snapshots of the call stack (known as samples), and they’re taken every millisecond.
One helpful feature in Scout is the ability to combine the two types of data in really useful ways. For example, the code in Flash Player for receiving a mouse event is instrumented, so it shows up in Basic Telemetry. This gives you a microsecond-precise measurement of how long it took to handle this event. If you have an ActionScript 3 event handler, then Flash Player also gathers samples of your code while it’s executing. In Scout, when you go to the Top Activities panel and click on the event, Scout will filter the ActionScript panel to show only the code that was running while that specific event was being handled (see Figure 6).
Basic Telemetry and ActionScript Sampler data are also combined in the Summary panel. Most of the data in this panel comes from Basic Telemetry, but if you’ve turned on the ActionScript Sampler, you can expand the ActionScript category to see a breakdown of the packages in which the most time was spent. Just remember that this data is approximate, because it’s computed by sampling (unlike the rest of the data in the Summary panel). This is indicated by the "≈" sign (see Figure 7).
Here’s a summary of the two types of data:
If you turn on the CPU Usage setting, Flash Player will take periodic measurements of how much CPU time it’s using. It gets this data by querying the operating system, so it corresponds to the figures you would see in Activity Monitor on OS X, or Task Manager on Windows. It's low overhead, so you can turn this on while you're investigating the performance of your content.
If you use the Flash DisplayList in your content, then the DisplayList Rendering Details setting lets you record detailed information about which portions of the screen get redrawn in each rendering pass, and the time taken to render each DisplayObject. You can see this information in the DisplayList Rendering panel in Scout. In particular, the Heat Map mode shows you graphically which portions of the screen took the longest time to render.
The overhead of collecting this data depends on how many objects are on the DisplayList. This can get quite expensive, so you shouldn’t look at the overall performance of your content with this data setting turned on; only the relative time taken to render each DisplayObject. A good workflow is the following:
If you use Stage3D in your content (either directly, or using a framework such as Starling or Away3D), Scout can give you detailed information about what’s going on. The Stage3D API in Flash Player is instrumented so that it can send every Stage3D API call and its arguments to Scout. You can then replay the commands inside Scout, to see how each scene is rendered, one draw call at a time. This works because there’s a copy of the Stage3D engine inside Scout itself.
Turning on the Stage3D Recording setting generates a huge amount of data, and has a high overhead. That's because Flash Player has to send every buffer, texture, and AGAL program over to Scout, along with details about every function call you make to the Stage3D API. This is a lot of data, and collecting and sending it will slow down the execution of your content. It's fine to turn on Stage3D Recording when you want to debug rendering problems, but you should never do performance profiling with this data setting enabled. The ActionScript panel will even warn you about it if you try (see Figure 8).
If you’re concerned about performance, you should carry out the following steps:
If you need to debug a rendering issue, like an AGAL program not working the way you expected, you can just jump to the second step. Remember though, when you want to improve the performance of your content, you should first find out if Stage3D is part of the problem before trying to optimize it!
Because Stage3D Recording sends so much data to Scout, it's easy to use up gigabytes of memory within a matter of minutes. Since you might want to debug rendering problems that happen after your content has been running for a while (for example, determining why the dragon boss on Level 22 doesn’t have a shadow), Scout has a delayed recording mode. If you change the Recording setting from Immediate to Delayed, Flash Player will send the same detailed Stage3D data as before, but it won't get loaded into memory in Scout. When you click start recording in the Stage3D Rendering panel, Scout loads only the necessary data into memory (such as your meshes and textures). This lets you replay the commands from that point onwards, without using as much memory.
You’ve already seen why it’s important to choose the right data settings to profile accurately with Scout. But it’s also vital to consider the environment in which you’re profiling. If you don’t, you could end up getting very misleading data, and waste time trying to fix a problem that isn’t really there! Here are some important tips to consider when you’re doing performance profiling with Scout:
Now that you understand how the data in Scout gets collected, you can profile more accurately and effectively. Remember to only collect the data you need to answer the question you’re currently addressing. Blindly turning on all the settings will end up skewing the data, and could lead you on a wild goose chase where you end up optimizing the wrong thing!
Once you’ve identified a performance problem, the next step is to determine why there’s a problem, and what to do about it. To learn more about how Flash Player works, and what to do about the information that Scout shows you, see Understanding Flash Player with Adobe Scout.