ActionScript guide to dynamic streaming


Abhinav Kapoor

Created

12 January 2009

- Requirements

 

User level

Intermediate

 

Adobe Flash Media Server 3.5 introduces a flexible, fast, and efficient switching capability that enables seamless transition between two streams of the same content but different bit rates (quality) in both recorded video on demand and broadcast (live) streaming scenarios. Likewise, Adobe Flash Player 10 introduces a new set of Quality of Service (QoS) metrics that are exposed by ActionScript 3.0 to provide real-time network and video performance information and enable developers to take control of video playback and adjust the streaming experience accordingly.

 

Dynamic stream switching gives video consumers the best possible experience based on their bandwidth environment and system resources. Also, this feature can be used to dynamically change streams queued in a playlist, replace a stream with another stream of the same content and compatible timeline and different bitrates or resolution, or swap it for another stream with completely different content altogether.

 

This article gives an overview of dynamic stream switching in Flash Media Server 3.5 and provides guidelines on how you can implement this feature using ActionScript 3.0. (There is also brief mention at the end of implementing dynamic streaming in ActionScript 2.0.)

 

Previous workarounds

 

Prior to Flash Player 10 and Flash Media Server 3.5, streaming video and audio was limited to constant bit rate streaming and lacked a native capability to switch bit rates to adjust to the dynamic nature of bandwidth conditions and client capabilities for rendering the video. For example, a server streaming video with a bit rate of 512 Kbps to a user on a DSL connection might work fine in ideal network conditions, but in adverse conditions—such as wireless connections with interference (or any other scenarios causing the bandwidth available to drop down below 512 Kbps)—this would result in the player buffer shrinking, eventually running out of data and causing the playback to stop and requiring the player to wait to buffer again.

 

The lack of a stream switching solution often caused the playback to pause and stutter with fluctuating bandwidth conditions. Even in good network conditions, the player could not always provide clients with the best possible quality or experience.

 

While many ways have been suggested to provide bit rate switching by stopping a playing stream and starting to a play a new one, none of the proposed solutions has provided a seamless transition to the new stream without incurring a significant pause while the stream switches to a new one, thus compromising the user experience. Also, proposed solutions have either estimated the client bandwidth at the start of a session using the server's native client bandwidth detection (which has startup overhead and delays playback) or during playback by monitoring the client-side buffer or empty buffer events.

 

Notification of an empty buffer event is too late to respond with a switch, because the viewer's playback has already stopped. Watching the client-side buffer may be effective for responding to a drop in bandwidth but it does not provide the capability to monitor increases in bandwidth speeds, since the FMS server sends enough data to fill the buffer duration specified in the application, but not beyond.

 

New NetStream method

 

Flash Player provides a flexible API to switch from one stream to another using the new NetStream method: play2. The play2 API is a superset of the old play API and provides new functionality to switch streams. The play2 method takes a single object as aNetStream.play.

 

The API does not make any assumptions about the content of the streams being switched, except that the video timelines of the two streams be related and compatible.

 

This approach makes no assumptions about formatting of content containers. It works with all existing container formats supported by Flash Media Server and Flash Player. The flexible API allows the developer to customize the criteria used to determine when streams will be switched, instead of enforcing or automatically choosing the criteria.

 

Usage

 

public play2(playParam: NetStreamPlayOptions)

 

Parameters

  • playParam: NetStreamPlayOptions. An instance of a new class, NetStreamPlayOptions, containing the parameters for the play command.

The following properties are supported on the object:

  • streamName: String. The name of the stream to play or the new stream to which FMS needs to switch.
  • oldStreamName: String. The name of the initial stream that needs to be switched out. This is not needed and ignored when play2 is used for just playing the stream and not switching to a new stream.
  • start: Number. The start time of the new stream to play, just as supported by the existing play API. and it has the same defaults. This is ignored when the method is called for switching (in other words, the transition is either NetStreamPlayTransition.SWITCH or NetStreamPlayTransitions.SWAP)
  • len: Number. The duration of the playback, just as supported by the existing play API and has the same defaults.
  • transition: String. The transition mode for the playback command. It could be one of the following:
    • NetStreamPlayTransitions.RESET
    • NetStreamPlayTransitions.APPEND
    • NetStreamPlayTransitions.SWITCH
    • NetStreamPlayTransitions.SWAP

    Note: For more information about transition, see the NetStreamPlayOptions documentation in the ActionScript 3.0 Language and Components Reference.

     

For example, the following code will issue a new play command to play a stream called streamX using NetStreamPlayTransitions.RESET:

var param:NetStreamPlayOptions = new NetStreamPlayOptions(); param.oldStreamName = null; ///optional param.streamName = streamX; param.transition = NetStreamPlayTransitions.RESET ns.play2(param);

 

The above code results in resetting any queued up playlists and playing streamX exactly as the NetStream.play() method would.

 

Using playlists and transitions

To add a stream to a playlist, set NetStreamPlayOptions.transition to NetStreamPlayTransitions.APPEND:

var param:NetStreamPlayOptions = new NetStreamPlayOptions(); param.transition = "APPEND"; param.streamName = "streamA"; ns.play2(param);

 

This mode invokes the current NetStream.play function with the "reset" flag set to "false" and thus queues up the streamName at the end of the playlist to be played after the previously queued up streams have played through.

 

To switch from one stream to another, use NetStreamPlayTransitions.SWITCH. use this mode to perform multi-bit-rate streaming by seamlessly switching one bit-rate version of a stream to another when the measured QoS metrics suggest a more compatible bit rate. For example, if an application plays a particular stream or adds streams to a playlist at a particular bit rate, but later realizes a significant change in available bandwidth, the application would then want to update the playlist with the names of streams containing a more compatible bit rate. Another common use case would be tracking the dropped frames value. If the video is dropping a lot of frames due to the client's insufficient CPU resources, the application could perform a switch to a lower bit rate by using this transition mode.

 

If the oldStreamName is null or undefined, this mode would make the server switch to the streamName at the logical point/key frame to avoid any switching artifacts and keep the switch smooth. If the oldStreamName is defined and valid, then the server either switches to the streamName at the next key frame if it's already playing, or updates the playlist if it's queued up in the playlist and hasn't started streaming:

 

var param:NetStreamPlayOptions = new NetStreamPlayOptions(); param.transition = NetStreamPlayTransitions.APPEND; param.streamName = "streamA"; ns.play2(param); ... ... ... ... ///SWITCH to lower bit rate version of streamA param.streamName = streamA_lower; param.transition = NetStreamPlayTransitions.SWITCH; ns.play2(param);

In the above case, streamA is switched to stream_lower seamlessly.

The switch can also be applied to an item in a playlist:

var param:NetStreamPlayOptions = new NetStreamPlayOptions(); param.transition = NetStreamPlayTransitions.APPEND; param.streamName = "streamA"; ns.play2(param); param.streamName = "streamB"; ns.play2(param); param.streamName = "streamC"; ns.play2(param); ... ... ... ... param.oldStreamName = streamA; param.streamName = streamA_lower; param.transition = NetStreamPlayTransitions.SWITCH; ns.play2(param); param.oldStreamName = streamB; param.streamName = streamB_lower; ns.play2(param); param.oldStreamName = streamC; param.streamName = streamC_lower; ns.play2(param);

 

In the above case, the playlist originally had streamA, streamB, streamC; but after the switch, it has streamA_lower, streamB_lower, streamC_lower.

Like NetStreamPlayTransitions.SWITCH, NetStreamPlayTransitions.SWAP replaces oldStreamName with streamName and keeps the rest of the playlist queue the same. However, in this case, the server makes no assumptions about the content of the streams and treats them as different content. Therefore, it will do the switch either at the stream boundary or never; in other words, if the server has already started sending the bits for the oldStreamName, then it will not switch to streamName and instead send a NetStream.Play.Failed event. If the oldStreamName is still queued up and not sent yet, then the server behaves exactly like the .SWITCH case and sends the bits for the streamName from the beginning. A good use case for this is when the streams are not related to each other and have different content or lengths (for example, swapping a commercial with another based on user tracking and past commercial viewing statistics). Obviously, this mode is useful only in a playlist scenario, not when only a single stream is playing:

ns.play(streamA); ns.play(streamB); ns.play(streamC); .. .. .. var param:NetStreamPlayOptions = new NetStreamPlayOptions(); param.streamName = streamX; param.oldStreamName = streamB; param.transition = NetStreamPlayTransitions.SWAP ns.play2(param);

 

In the above case, the original playlist was streamA, streamB, streamC; but after the swap, it is either streamA, streamX, streamC if the call was made before streamB started streaming from the server, or it would stay as streamA, streamB, streamC.

 

NetStreamPlayTransitions.STOP is equivalent to the current NetStream.play(false) call. It stops playing the current stream, and any additional parameters that are sent in the parameter object are ignored. For example:

var param:NetStreamPlayOptions = new NetStreamPlayOptions(); param.streamName = "streamA"; ns.play2(param); .. .. .. param.transition = NetStreamPlayTransitions.STOP ns.play2(param); ///stops the playback

 

New netStatus events

A couple of new events are also added. The netStatus event NetStream.Play.Transition is triggered when the server succeeds in switching to the new stream and starts sending messages for the new stream. If the server is unable to switch to the new stream for any reason, such as a missing file or bad oldStreamName or streamName, then a NetStream.Play.Failed message is sent.

The netStatus event object NetStatusEvent has the following properties:

  • code: NetStream.Play.Transition
  • description: Transitioned to [new stream name]
  • clientid
  • level: status
  • details: [new stream name]

 

Based on the player's client-side buffer, the actual switch would be visible in the video later. When the switch actually happens during playback, the NetStream.Play.TransitionComplete event is called as a playStatus event along with the onMetaData event. Both these playStatus events can be used to make changes to the UI to accommodate the new stream; for example, if the new stream has a different resolution from the old stream, then these events can be used to trigger the UI changes to fit the new size.

 

The time from making a switch call to getting the NetStream.Play.Transition event is the time it takes the server to switch to the new stream and start sending messages from the new stream. The time between getting the NetStream.Play.Transition event and NetStream.Play.TransitionComplete event is the time it takes for the client to play out the buffered data from the old stream and is equal to the client buffer at the time it started receiving messages from the new stream.

 

Depending on the information provided with the stream, the metaData object could have one or more of the following properties. Metadata is encoded into the file at the time of encoding. Each encoder may choose to include or exclude values. Examples of metata include the following:

  • audiodatarate
  • creationdate
  • title
  • width
  • videodatarate
  • lasttimestamp
  • framerate
  • height
  • lastkeyframetimestamp
  • rating
  • copyright
  • author
  • duration

Example

 

var stream:NetStream = new NetStream(connection); stream.addEventListener(NetStatusEvent.NET_STATUS, netStatusHandler); stream.client = new CustomClient(); var video:Video = new Video(); video.attachNetStream(stream); stream.play(videoURL); addChild(video); ... ... ... var param:NetStreamPlayOptions = new NetStreamPlayOptions(); param.oldStreamName = videoURL; param.streamName = videoLowBitRateURL; param.transition = NetStreamPlayTransitions.SWITCH; stream.play2(param); class CustomClient { function metaHandler(eventObj:Object):void { var metaDataText = "MetaData: " + "\n"; for (i in eventObj) { metaDataText += i + ": " + eventObj[i] + "\n"; } } function onPlayStatusHandler(info:Object):void { if(info.code == "NetStream.Play.TransitionComplete") { trace("switched playback to new stream: "+info.details); } else if(info.code == "NetStream.Play.Complete") { } } function nsHandler(event:NetStatusEvent):void { if(event.info.code == "NetStream.Play.Transition") { trace("switch succeeding – getting new stream data now: "+event.info.details); } if (event.info.code == "NetStream.Play.Stop") { } if(event.info.code == "NetStream.Play.Start") { } if (event.info.code == "NetStream.Buffer.Full") { } if (event.info.code == "NetStream.Buffer.Empty") { } }

QoS statistics

 

Flash Player 10 introduces some QoS properties that provide useful information about the stream/video performance and client network capabilities. A new class, NetStreamInfo, encapsulates the new properties and can be retrieved by accessing the info property of a NetStream object.

The info property, when accessed, takes a snapshot of the current state of the stream and the internal buffers, and can be used to check the performance of various attributes and switch to a different stream if necessary. The following properties track the total data rate:

  • currentBytesPerSecond: NetStream buffer fill rate. This specifies the rate at which the NetStream buffer is being filled or is publishing in bytes per sec. It is calculated as a moving average of the data received by the stream or publishing over a period of a second. It is valid for both progressive and streaming cases of NetStream.
  • maxBytesPerSecond: NetStream buffer capacity. Valid only for receiving streams, it specifies the max capacity of the stream. This is the maximum rate at which the stream buffer can get data currently. The FMS server sends as much data as is needed by the player to play the stream, which is measured by the currentBytesPerSecond property above. However, the server could send more data if needed up to a maximum value, which is limited by the client's network connection. This property provides this maximum limit at a given point in time. This value would change dynamically along with any changes in the network speeds or capacity of the player's network environment. This property can be effectively used to calculate the best bit rate that a client could play for its network. While all the other properties could be used with any version of the FMS server, maxBytesPerSecond requires FMS 3.5 or later; otherwise, it will provide false low dips when the FMS server throttles the stream.
  • byteCount: NetStream buffer fill size. This specifies the total bytes which have arrived into the queue from the start of the session, regardless of how many have played or flushed. Also, it can be used to access the amount of data published by the stream. This value is useful in calculating the data rate on a different granularity of time than calculated by currentBytesPerSecond above. The application could set up a timer and calculate the difference in current value with a previous value and get the data rate during the time interval.

 

More specific information can be accessed for each type of data (video, audio, and data) independently through the following API properties (see also the complete list of NetStreamInfo properties in the ActionScript 3.0 Language and Components Reference):

  • audioByteCount: NetStream audio buffer fill size. This specifies the total audio bytes which have arrived into the queue from the start of the session. Also, it can be used to access the amount of audio data published by the stream.
  • audioBytesPerSecond: NetStream audio buffer fill size. This specifies the total audio bytes which have arrived into the queue from the start of the session. Also, it can be used to access the amount of audio data published by the stream.
  • videoByteCount: NetStream video buffer fill size. This specifies the total video bytes which have arrived into the queue from the start of the session. Also, it can be used to access the amount of video data published by the stream.
  • videoBytesPerSecond: NetStream video buffer fill rate. This specifies the rate at which the NetStream video buffer is being filled or is publishing in bytes per sec. It is calculated as a moving average of the video data received by the stream or publishing over a period of a second.
  • dataByteCount: NetStream data buffer fill size. This specifies the total data bytes which have arrived into the queue from the start of the session. Also, it can be used to access the amount of data messages published by the stream.
  • dataBytesPerSecond: NetStream data messages buffer fill rate. This specifies the rate at which the NetStream data message buffer is being filled or is publishing in bytes per sec. It is calculated as a moving average of the data messages received by the stream or publishing over a period of a second.
  • droppedFrames: Dropped frame count—the absolute number of frames dropped. In recorded streaming or progressive cases, if the video is a high quality or a high resolution, high bit-rate video, the decoder may lag behind in decoding the required number of frames per second if it does not have adequate system resources (CPU). In the live streaming case, it is possible that queue has to drop frames to catch up if the latency is getting high. The droppedFrames property would provide useful information in such cases, specifying the number of frames which were dropped and not presented to the video object. Hence this property is very useful in determining whether the client has enough CPU to display a video of a particular bit rate or resolution, or otherwise switch to a lower quality or resolution stream.

    This property could also be used during the video encoding stage to test the compression or resolution values by tracking the dropped frame count or rate at different target platforms and make sure the numbers stay low, or otherwise encode the video at lower resolutions. For a publishing stream, droppedFrames would provide the frames dropped from publishing due to limited CPU or network resources.

    The above properties are computed both for receiving and publishing NetStream instances as well as publishing Camera and Microphone data, unless otherwise noted.

  • playbackBytesPerSecond: The current NetStream playback rate in bytes per second. The playback buffer could have various playlists buffered in it. With multi-bit-rate switching, this would happen more frequently now. This property would provide the bit rate at which the current video is playing.
  • audioBufferByteLength, videoBufferByteLength, dataBufferByteLength: The NetStream buffer size in bytes. Similar to NetStream.bytesLoaded that is used in progressive cases, these properties specify the size in bytes in the buffer for audio, video, and data, respectively. This provides raw buffer size information for streaming both live and recorded streams.
  • audioBufferLength, videoBufferLength, dataBufferLength: The NetStream buffer size in units of time. These calls extend the bufferLength property, which exists in NetStream already, to provide the buffer length in time values for audio, video, and data, respectively.
  • SRTT: Session Round Trip Time. Provides a smoothened value of the round trip time of the NetStream session (averaged over a number of round trips between the client and server or two clients). This returns a valid value only for RTMFP streams; it returns 0 for RTMP streams.

 

Using ActionScript 2.0 for dynamic streaming

 

Dynamic streaming can also be implemented in ActionScript 2.0. Instead of the NetStreamPlayOptions object, the NetStream.play2 method takes an object of type Object with the same properties as the NetStreamPlayOptions class. For example:

 

var obj:Object = new Object(); obj.oldStreamName = "video1.flv"; obj.streamName = "video1_lowerbitrate.flv"; obj.start = 0; obj.length = \-1; obj.transition = "switch"; stream_ns.play2(obj);

QoS properties are accessed with the NetStream.info() method, which returns an object of type Object with the same property names as the NetStreamInfo object as described above:

bufferInfoObj:Object = netstreamObj.getInfo(); var maxBPS:Number = bufferInfoObj.maxBytesPerSecond;

 

Live dynamic streaming

 

From a subscriber's perspective, live (broadcast) dynamic streaming works the same way as dynamic streaming for recorded video on demand. The client specifies the new stream name for switching. The FMS server identifies the right frame at which to make the switch, keeping the switch smooth without introducing a gap in the live stream. The amount of delay the server will introduce is based on the size of the client's playback buffer. For the best experience, the client-side buffer size (in units of time) needs to be more than the keyframe interval of the live stream.

 

Frame-accurate live switching will depend on having a publisher that can produce multiple streams with timestamps that are synchronized closely enough that the server can make accurate transition-point selections. Flash Media Live Encoder 3.0 provides this capability.

 

For more details about live dynamic streaming, please read my article, Live dynamic streaming with Flash Media Server 3.5, which talks about live scenarios in greater depth and discusses various recommendations and best practices.

 

Where to go from here

 

You can find an exhaustive list of recommendations and best practices for stream encodings and client application development in my article, Dynamic streaming on demand with Flash Media Server 3.5.

 

Read about a sample client-side ActionScript class that implements dynamic streaming, and is recommended for application development using dynamic streaming, in the DynamicStream class section David Hassoun's three-part series about dynamic streaming in Flash Media Server 3.5:

 

 

Finally, be sure to check out the new Flash Media Server tools page.