New buffering strategies in Flash Player 9 and Flash Media Server 3

Tom Green
17 March 2008
User Level
Required Products
Flash Media Server 3 (Download Trial)


Additional Requirements


Flash Player 9 Update 3 (9,0,115,0) or later


Video on the web is growing more popular every day. User-generated content, professional high-quality video, and the ever-increasing bandwidth of connections are pushing the web as the video delivery channel of the future. The video explosion can also be attributed to the compelling evolution of the Adobe video ecosystem.
Adobe Flash Player 9 Update 3, Adobe Flash CS3 Professional, and Adobe Flash Media Server 3 offer a unique combination of ubiquitous media player, powerful streaming media capabilities, and flexible development environment for creating, customizing, and delivering state-of-the-art video experiences to the broadest possible audience.
The Flash Media Server 3 product line offers dramatically improved performance, more secure streaming, live streaming enhancements, industry-standard H.264 and HE-AAC support, and streaming delivery to mobile phones with Adobe Flash Lite 3 and Adobe Media Player software.
Today's web audience are more and more demanding. They expect better video quality, higher bit rates, and overall have very high expectations regarding their video experience. Optimal buffering of incoming video is, even more today, an important goal to achieve. Therefore, knowing exactly how Flash Media Server and Flash Player perform video buffering is critical for developers to fine-tune their streaming applications.
In a previous article, I described a technique to improve Flash Media Server 2 buffering behavior (see Implementing a dual-threshold buffering strategy in Flash Media Server). The information in the article is still applicable to enhance Flash Media Server 3 streaming behavior. However, with the release of Flash Media Server 3 and Flash Player 9, Adobe introduced many more new buffering strategies. In this article, I'll describe these new improved buffering behaviors.
Standard video buffering
Internet connections are characterized by a number of statistically determined features, including bandwidth and latency. Bandwidth and latency values are not guaranteed—in fact, they can fluctuate a great deal depending on the local ISP network load and remote server load, as well as by local applications sharing network resources.
Video delivered using a "deterministic" channel (such as satellite, DVD, cable or, digital TV broadcasting) requires only a small buffer because data arrives to the media player with deterministic delay and rate, and very limited or infrequent data drops.
Video delivered over IP networks is much more problematic. There is no guarantee that the data will flow to the users at the sufficient rate and determined delay. Instead, it arrives with both a rate and delay that can change consistently as the video streams. Therefore, an accurate buffering system for video streams is necessary for web delivery.
Streaming servers manage buffering in a rather simple manner. The server sends data to the client, which stores data in a buffer. The data is sent as quickly as possible. When the buffer is full (because it has reached a predefined threshold of stored data), the video playback starts and the server continues to send data to the client—but it only sends the amount of data necessary to keep the buffer full.
If the data rate becomes insufficient to keep the buffer full, the buffer becomes empty in the span of a few minutes. When this happens, the video playback is stopped and a rebuffering phase begins again.
A small buffer results in instant playback, while a wider buffer provides higher resilience to bandwidth fluctuations. A video delivery application developer must choose between resilience and user experience: if you set a buffer too small, it fills up quickly but is vulnerable to adverse channel conditions. A sudden, temporary drop in data flow can easily empty a small buffer, causing the video playback to stutter. On the other hand, if you set a buffer too big, the playback is very reliable but requires that viewers endure a longer (and annoying) buffering time before the streaming video starts playing. In both cases, for some viewers, the experience could be considered unsatisfactory.
Standard buffering in Flash Media Server
When streaming from Flash Media Server 3 and Flash Media Server 2, Flash Player basically behaves in the manner previously described. However, thanks to the flexibility of the Flash authoring platform, the buffering functionality can be enhanced using client-side ActionScript code.
The standard buffering process is indeed vulnerable to bandwidth drops, as well as being unable to exploit a sudden increase of bandwidth. A dual-threshold buffering strategy addresses this issue, assuring a fast start and—at the same time—providing fairly high resilience to bandwidth fluctuations or other adverse conditions.
When the buffer is filled with a given amount of data (the first threshold), the playback starts as expected. But instead of trying to keep the buffer full to this level, the modified strategy involves trying to fill the buffer to a second, higher threshold. This extra amount of data could be extremely useful later if the Internet connection encounters temporarily bandwidth drops or fluctuations.
Dual-threshold buffering in Flash Media Server
The strategy of working with dual-threshold buffering can be implemented for Flash Media Server 3 using the same code as described in my Flash Media Server 2 article, Implementing a dual-threshold buffering strategy in Flash Media Server. You should be aware that at a low level, Flash Player 9 executes buffering by transmitting a sudden burst of frames (to enhance the scalability of the server). When you test your applications locally using the standard buffering technique, you can see a strong oscillation of the buffer length as you monitor the buffer level. This occurs because of the transmission burst and other related causes. It is important to note that the oscillation of the buffer length is not caused by a native implementation of a dual-threshold buffering strategy.
New buffering features available in Flash Player 9
Flash Player 9 introduces some new buffering functionality that was not implemented in the previous release. The new features include the buffering of H.264 video files and the ability to continue buffering while the stream has been paused. In order to fully leverage these features, you need to connect Flash Player 9 Update 3 to Flash Media Server 3.
Buffering of H.264 encoded video
One of the most exciting new features of Flash Media Server 3 and Flash Player 9 is undoubtedly the support for H.264 streaming media. The process of buffering H.264 encoded video is much more complex than buffering On2 VP6 or Sorenson Spark-encoded video.
Both the VP6 and Spark codec primarily use two types of video frames: intra-compressed (i-frames) and inter-predicted frames (p-frames). Known commonly as "keyframes," i-frames are compressed without reference to other frames. This means that keyframes are independent and represent an anchor when viewers perform seeking, scrubbing, and similar operations. P-frames are encoded as the difference from a previous p-frame or i-frame. P-frames are therefore dependent by the decompression of previous frames.
The Spark codec allows the use of the previous p-frame or i-frame only as reference to a p-frame, while the VP6 codec allows the reference to include two frames—which are always in the past and always reside after the last i-frame. This means a GOP (group of pictures) always extends between two consecutive i-frames. Buffering this type of stream is simple because after the first i-frame is buffered, it is not strictly necessary to buffer an exact amount of subsequent p-frames or i-frames because none of the frames requires future frames in order to be decompressed.
H.264 is a very complex standard. It uses a large number of encoding modes and strategies. A new type of frame, called a b-frame, can have multiple references both in the past and future. P-frames can have multiple references too, and their referencing is not "bounded" by the location of i-frames. Therefore, i-frames are not necessarily keyframes because after an i-frame, you can find a p-frame that refers to a frame preceding that i-frame. In order to identify a safe frame to seek to, H.264 defines a new frame type, called the IDR-frame. An IDR-frame is an i-frame that prevents the next p-frames from using any prior frames as their reference. Two consecutive IDR-frames determine the boundaries of a GOP.
When streaming H.264 videos, depending on the techniques used by the encoder, it might be necessary to buffer several frames before being able to decompress the first frame. When you stream H.264 content you will get an effective buffer length that differs from the settings you applied using the 


method. You may also find it is impossible to set and obtain a zero-length buffer.
This means that streaming H.264 video generally requires a deeper buffer. To help with this, I recommend that you always encode your video using two-pass mode because video encoded using single-pass usually requires a higher bit rate at the beginning—and this can lead to longer waiting times before the initial playback begins.
Background extended buffering while in pause
A new important feature is available in Flash Player 9 Update 3 when the stream playback is paused. It differs from the buffering procedure used in previous versions. When a stream is paused using Flash Player 8, for example, the streaming server fills the buffer regularly. As soon as the stream is unpaused, however, it always clears all the buffered data and starts a new rebuffering session. This process wastes precious bandwidth and causes an annoying rebuffering period every time the viewer chooses to pause and unpause the stream playback.
Flash Player 9 operates differently when the stream playback is paused. When the viewer pauses the stream, Flash Media Server continues to feed the buffer. The buffer size is slowly expanded in the background beyond the set buffer length. In approximately 60 seconds (bandwidth permitting) the buffer is filled with 60 seconds of stream data. This new feature takes advantage of the bandwidth available between the client and server without wasting any of it. This causes the buffer to save the extra data, which is waiting when the viewer chooses to unpause the stream.
This buffering process mimics the buffering that occurs when you deliver a progressively downloaded video but includes the advantages inherent over the control of bandwidth usage and server load. If you encounter situations where you don't wish to use the extended buffering in the background, you can still choose to use the older buffering method. Simply set the stream to stop instead of pause when the user clicks the pause button, or pause the stream and then perform a seek operation. By using these strategies, the buffer is not expanded.
Synchronizing multiple streams


Thanks to this new buffering feature, it is finally possible to synchronize multiple streams using Flash Player 9 and Flash Media Server 3. The process for synchronizing multiple streams using progressive download is very simple. After you launch the load of videos and pause them all, you can monitor the loading status of each video. When a sufficient amount of data for each video is loaded, you can unpause the playback of all the video files at the same time. This method is not possible when using Flash Player 8 to stream video files because the unpause command causes the rebuffering of the streams and the subsequent start of each playback begins with an unpredictable delay.
On the other hand, Flash Player 9 keeps the buffer, so it is now possible to apply the same method as described for progressive downloads to achieve the synchronization of multiple video streams.
Where to go from here
As you can see, Flash Player 9 introduces many helpful new buffering features. Understanding exactly how Flash Player and Flash Media Server perform video buffering is very important for developers to optimize their streaming applications. By taking advantage of the features described in this article, you can be sure to satisfy your audience's expectations and guarantee a state-of-the-art video experience using Flash Player 9 and Flash Media Server 3.
Feel free to e-mail me for more information and visit my blog.