Implementing a dual-threshold buffering strategy in Flash Media Server

Fabio Sonnati
10 July 2006
User Level
Required Products
Flash Media Server (Download Trial)
Buffering accomplishes a very important task when streaming video over the web: it absorbs bandwidth fluctuations. In a perfect world, where the effective bandwidth between client and server doesn't change during video streaming, buffering could be very small. In the real world, however, where several factors like line status, sudden changes in a server's load, or simultaneous downloads on the client side may lead to an unstable bandwidth, buffering becomes absolutely necessary because any moderate drop in connection speed under the amount required by the stream could interrupt the user's viewing experience.
Compromised viewing experiences become more evident when serving hour-long streams, as with Internet TV applications. You can detect approximately the client bandwidth and choose the stream size accordingly (see Stefan Richter's article, Delivering Flash Video: Dynamic Bandwidth Detection with Macromedia Flash Communication Server) but there is no certainty about the consistency of detected connection speed across the entire streaming time.
Therefore, buffering is necessary. But what is the optimal length for it? A small buffer (2–3 seconds or less) fills up quickly, allowing the movie to start quickly, but a moderate fall in speed will easily empty the buffer, causing movie playing to pause until the end of a rebuffering session. On the other hand, a very large buffer (20 seconds or longer) assures a high resilience to bandwidth changes but also forces the user to wait during long and annoying prebuffering sessions. In both cases, there is the real possibility of a poor viewing experience.
In this article, I address this important issue by implementing a dual-threshold buffering strategy. This optimization of standard Flash Media Server 3 behavior combines the advantages of a short buffer with the advantages of a very large one. The goal is to achieve a very fast start time while maintaining very high resilience towards bandwidth fluctuations.
After explaining how Adobe Flash Media Server 3 implements buffering of prerecorded videos by default, I explain the principles of the proposed optimized technique. At the end you will see some code in action.
Standard buffering
When Flash Player connects to Flash Media Server 3 to subscribe to a video stream, you can define a buffering amount in seconds using the method netStream.setBufferTime(N).
The buffer is on the client side and its behavior is indeed very simple, yet efficient. Flash Player receives the stream and archives data in the buffer until it is full. When it happens, the movie starts to play and Flash Player tries to keep the buffer full to the chosen length, receiving only a sufficient amount of data from FMS. If the instant bandwidth goes below the value required by the current stream, the amount of data available in the buffer slowly decreases. This is because the data rate at which FMS feeds the buffer becomes lower than the data rate at which Flash Player fetches data from buffer. If the buffer becomes empty, the movie stops playing until the buffer fills up again.
Figure 1 simulates the buffer behavior (blue line) related to a hypothetic graph of the instant client-server bandwidth (orange line). The top graph illustrates a client-server bandwidth profile normalized to the bandwidth required by the stream. When the orange graph goes higher than 100%—say, 120%—the instant available bandwidth at that time is 20% higher than the amount required by the stream. If the graph goes below the 100%, bandwidth is insufficient for the current stream. The bottom graph reports the amount of data in the buffer normalized to buffer length. When the blue line is at 100%, the buffer is completely full, and so on.
Standard buffer behavior and bandwidth availability over time
Figure 1. Standard buffer behavior and bandwidth availability over time
Let's examine these graphs in more detail:
  1. Starting at the beginning, the buffer fills linearly because the movie is not playing yet and the bandwidth is approximately constant.
  2. At time T1 the buffer is full; therefore movie starts to play and the buffer is kept full by a bandwidth greater than that required by the stream (available bandwidth greater than 100%).
  3. At time T2 the available normalized bandwidth falls below 100% and the buffer starts to empty.
  4. At T3 the buffer is completely empty, the movie is paused because there is no data in the buffer to play the movie, and rebuffering begins. The buffer starts to fill almost linearly because of the almost constant bandwidth. (Notice, however, that now the refill time T3–T4 is more than doubled because of the halved bandwidth.)
  5. At the full-buffer state again (T4), the movie exits its pause mode—the amount of buffered data resulting from a buffer fetch (for movie playing) and a buffer feeding (from FMS).
  6. At time T5 the available bandwidth exceeds the nominal value required by the stream and buffering starts to fill up again to the full status (T6).
As you can see from this standard, static buffering method, if bandwidth decreases under the required value, the buffer may be insufficient to compensate for the bandwidth drop and one or more rebuffering sessions may be necessary to deliver the streaming.
Obviously with a larger buffer, this would not happen. Determining the correct buffer depth requires weighing the following options:
  • A buffer should be large to prevent rebuffering. A larger buffer can compensate for a longer bandwidth drop.
  • A too-large buffer means a longer prebuffering time and probably a poor viewing experience.


Dual-threshold buffering
In order to achieve a fast start, you need a short buffer. But to obtain a high resilience to bandwidth drops, you need a much longer buffer. Is it possible to modify the standard buffering procedure to make the video start when the buffer reaches a first threshold value and then keep filling it up to a second, much higher level? The answer is yes.
You know that using the netStream's onStatus event, it's possible to recognize when the buffer is full or empty. Using it, you can set up a small starting buffer (the first threshold value). When this little buffer is full, the movie starts to play and a NetStream.Buffer.Full state is fired. A specific status handler can recognize this state and change the buffer length on the fly to a higher value (the second threshold).
Dual-threshold buffering assures a better allocation of excess bandwidth because it uses it to expand the buffer to a final, higher value while playing the movie at the same time. The movie can start after a short preloading time and the excess bandwidth can be used to build a big reserve of data to counteract the likelihood of future drops in bandwidth. If the buffer runs empty, a NetStream.Buffer.Empty state is fired and you can lower the buffer length again to the starting value. The same can be done if the user seeks in the middle of the movie.
Figure 2 shows the behavior of this modified buffering strategy, using the same bandwidth availability in Figure 1. Compared to the standard behavior, when the starting buffer (SB) is full, the buffer length is enlarged for a full allocation of the entire available bandwidth. Until time T2, when available bandwidth decreases under 100%, the buffer continues to grow. In this specific case, the second threshold is very high and the buffer doesn't even reach it. With this reserve of buffered data now in place, the lowered bandwidth between times T2 and T3 is accommodated for without any video interruption. At time T3, bandwidth returns over 100% and the buffer grows again. The second threshold is the ceiling of this growth, and you can set it arbitrarily high.
Dual-threshold buffer behavior and bandwidth availability over time
Figure 2. Dual-threshold buffer behavior and bandwidth availability over time
The following example helps illustrate the level of resilience that this dual-threshold buffering method can provide. Suppose you have an Internet TV application that delivers hour-long videos. You could set a first threshold of three seconds and a second threshold of 60 seconds. If you are supposed to deliver a 500 kbps stream to a xDSL client with an average, starting bandwidth of around 750 kbps (detected using a bandwidth checking routine), you will reach the first threshold in around two seconds. This will also be the time that the viewer must wait for the movie to start.
Deviating from the standard behavior, the buffer continues to expand beyond this initial value at a rate equal to 750 kbps – 500 kbps = 250 kbps. This ensures half a second of further buffered data for every second of streaming from Flash Media Server, which means that the buffer reaches the extended value after about 120 seconds.
What happens if after four minutes of streaming, the bandwidth suddenly falls in the range of 125–250 kbps? This method ensures about 60–120 seconds' worth of streaming instead of only a few seconds' worth. With such a large buffer in reserve, it's likely that bandwidth will return to normal before the buffer runs empty.
This dynamic buffering technique can guarantee a fast movie start time (using a low starting threshold) while ensuring an arbitrarily high resilience to bandwidth fluctuations. Resilience is limited only by memory usage. A buffer expanded to 100 seconds needs 3.7 MB for storing a 300 kbps stream or 6.2 MB for a 500 kbps stream. Because a seek in the middle of a stream empties the buffer, a too-large buffer can also cause a higher-than-average bandwidth consumption on the server side.
Examining the code
The sample code of this dynamic buffering technique is quite simple:
function initConnection() { // Create a NetConnection nc = new NetConnection(); // Define onStatus event handler nc.onStatus = function(info) { trace("Level: "+info.level+" Code: "+info.code); // Initialize the stream after the connection is successful if (info.code == "NetConnection.Connect.Success") { trace("--- connected to: " + this.uri); startStream(); //Inizialize the stream } }; // try to connect, to be changed with your app path nc.connect("rtmp://"); } function startStream() { //Define stream and buffers parameters stream_name="test_stream"; //your stream startBufferLength=2; //keep this in the range 2-4+ expandedBufferLength=15; //arbitrarily high // create a netstream ns = new NetStream(nc); // Define onStatus event handler ns.onStatus = function(infoObject:Object) { if (infoObject["code"]=="NetStream.Buffer.Full"){ ns.setBufferTime(expandedBufferLength); trace("set expanded buffer"); } if (infoObject["code"]=="NetStream.Buffer.Empty"){ ns.setBufferTime(startBufferLength); trace("set start buffer"); } } // attach the NetStream to a video object // change from videoObj to your istance name if needed videoObj.attachVideo(ns); ns.setBufferTime(startBufferLength);, 0); } initConnection(); stop();
I initialize the NetConnection in the first function. (You have to change the path used in the sample code to connect to your own FMS 3 application.) When the connection is successful, I call the initialization of the NetStream object. In this routine, I define the onStatus event handler with a few key parameters (name of the stream, start buffer length, and expanded buffer length) and then start the stream.
The key code is in the onStatus  event handler. Here the buffer length is expanded when a
NetStream.Buffer.Empty status is detected, and collapsed when a NetStream.Buffer.Full
is detected. If you want to seek in the middle of the video, you must set the buffer length to the starting value before launching the seek command.
Where to go from here
As you can see from this article, you can greatly enhance your FMS application's resilience to bandwidth drops using a dual-threshold buffering strategy. You can use this technique with a bandwidth-detection routine to ensure a better viewing experience. Try different parameters to satisfy your streaming project's demands. Feel free to e-mail me for more information and visit my blog for information about further enhancements.