Jozsef Vass

Adobe

Adobe Flash Player 10 and Adobe AIR 1.5 introduce a new communications protocol, Real-Time Media Flow Protocol (RTMFP), whose low latency, end-to-end peering capability, security, and scalability make it especially well suited for developing real-time collaboration applications by not only providing superior user experience but also reducing operators' costs.

Earlier versions of Flash Player use Real-Time Messaging Protocol (RTMP) and require Adobe Flash Media Server (FMS) for interactive collaboration applications (such as Adobe Acrobat Connect Pro) or audio/video streaming. While RTMP is an excellent choice for streaming media, shared objects, or remoting, it has limited ability of meeting real-time requirements of interactive audio and video communications.

In order to use RTMFP, Flash Player endpoints must connect to an RTMFP-capable server, such as the Adobe Stratus beta service or a future version of FMS. Stratus is a hosted rendezvous service that aids establishing communications between Flash Player endpoints. Unlike FMS, Stratus does not support media relay, shared objects, scripting, etc. So by using Stratus, you can develop applications only where Flash Player endpoints are directly communicating with one another.

Flash Player is already the market leader in online video distribution over the web. With the introduction of RTMFP and advanced media compression technologies, Flash Player 10 is well positioned as the leader in real-time communications as well.

In this article, I first highlight the benefits of using RTMFP in real-time communications applications. Second, I describe the new ActionScript 3.0 API for managing direct end-to-end RTMFP connections. Finally, I present our VideoPhone sample application.

Requirements

In order to make the most of this article, you need the following software and files:

Flex Builder 3

• Try
• Buy

Note: Please follow the instructions in Introducing Flex SDK 3.2 and Flex Builder 3.0.2 to install Flex Builder 3.0.2. This also includes Flex SDK 3.2, which is required to build applications targeting Flash Player 10. Although you may also use Adobe Flash Professional CS4 for development, this article instructs how to build a sample application using Flex Builder 3.

Flash Player 10

• Download

Sample files:

• stratus_article_assets.zip (ZIP, 13K)

Prerequisite knowledge

Familiarity with ActionScript 3.0 and Flex Builder is required.

Benefits of RTMFP

Real-Time Media Flow Protocol (RTMFP) is a new communications protocol introduced in Flash Player 10 and also available in AIR 1.5. One of its major differentiators from Real-Time Messaging Protocol (RTMP), which is based on the Transmission Control Protocol (TCP) and exclusively used in previous versions of Flash Player, is that RTMFP is built on User Datagram Protocol (UDP).

While TCP provides reliable data delivery (well applicable for file transfer, e-mail, etc.), it does not provide any end-to-end delay guarantees. Reliable data transmission in TCP is achieved by re-transmission of lost data, which introduces latency. Because minimizing end-to-end delay is one of the most important goals in real-time communications (a few hundred milliseconds' delay may render a conversation unusable), TCP is not well suited for this purpose. Transmission error resilience and recovery form an integral part of most advanced audio and video compression techniques—such as Speex audio and H.264 video codec, both available in Flash Player 10. Reliable delivery provided by TCP is therefore not needed. As a result, UDP, which provides an efficient and rapid data delivery, is popularly used in real-time collaboration applications where minimizing end-to-end delay is of paramount importance. Another advantage of UDP over TCP that it enables end-to-end peering—that is, direct data transmission between two clients located behind network address translators (NATs).

When compared to RTMP, RTMFP provides the following advantages for real-time communications:

• Low latency: Since RTMFP is built on top of UDP, it provides minimal latency for real-time communications. It is important to note that RTMFP provides both reliable and unreliable service. When sending data between two Flash Player instances (for example, using the NetStream.send() method), reliable data transmission is used. When sending Speex audio between two Flash Player instances, unreliable delivery is used, providing the smallest possible latency.
• End-to-end media delivery: Media is sent directly between two Flash Player instances without routing through a central relay server. When compared to RTMP, where all data is sent through Flash Media Server, RTMFP not only further reduces end-to-end delay, but also eliminates costs associated with a central data relay, thus lending itself to extremely scalable deployments.
• Data prioritization: Audio is transmitted with higher priority than video and non-time critical data (such as instant message, etc.). This can significantly enhance user experience over a bandwidth constrained communications channel.

All of these features represent tremendous benefits for real-time communications, providing a significantly greater user experience than is achievable with earlier versions of Flash Player.

Firewall traversal

RTMFP is built on top of UDP, which enables direct connection between clients even if they are located behind NATs or firewalls. In order for RTMFP to work, your firewall must be configured to allow outgoing UDP traffic. While this is the case with most consumer or small office/home office (SOHO) firewalls, many corporate firewalls block UDP traffic altogether.

One solution is to configure Flash Player to use a TURN proxy (Traversal Using Relays around NAT). Flash Player supports IETF Internet Draft draft-ietf-behave-turn-08 without authentication. If the network administrator configures a TURN proxy that allows outgoing UDP, Flash Player can be configured by adding the following line in mms.cfg (for more information on Flash Player configuration and the location of mms.cfg, please read the Adobe Flash Player Administration Guide for Flash Player 10):

RTMFPTURNProxy=ip_address_or_hostname_of_TURN_proxy

Direct UDP traffic is always attempted and the TURN proxy is only used as a backup: it is used for UDP traffic that cannot flow between Flash Player and Stratus (in case of UDP blocking firewall) or between Flash Player endpoints.

Even if your firewall enables outgoing UDP traffic, it is possible that end-to-end peering cannot be established due to a combination of firewalls. When one endpoint is located behind a so-called "symmetric firewall," end-to-end communications may not be possible. (For a classification of firewalls, please see the Network address translation entry on Wikipedia.) In this situation, you may use a TURN proxy to aid firewall traversal.

Stratus service

Flash Player instances must connect to the Adobe Stratus service (using rtmfp://stratus.adobe.com) in order to communicate with one another. Stratus is a hosted rendezvous service that helps Flash Player instances contact one another even if they are located behind NATs. Although connecting to Stratus service is very similar to connecting to Flash Media Server, Stratus does not provide any of the typical Flash Media Server features (media relay, shared objects, remoting, etc.). Flash Player endpoints must stay connected to Adobe Stratus during the entire time of communication. In order to access Stratus, you will need a developer key that is generated when you create your Adobe Developer ID.

RTMFP support is being planned for future version of Flash Media Server (no release date). With Flash Media Server, it will be possible to enable communications between Flash Player 9 or earlier clients (using RTMP) and Flash Player 10 clients (using RTMFP).

Security

RTMFP provides secure communications between endpoints. It uses a 128-bit AES with the key negotiated using the Diffie-Hellmann key exchange method. However, it does not provide strong endpoint authentication such as SSL or RTMPS. To aid endpoint authentication, RTMFP and ActionScript expose secure nonces to application developers. These nonces are available at both communicating Flash Player endpoints and are guaranteed to match. By verifying these nonces, end users can ensure that there is no man-in-the-middle attack. These nonces can also be used to develop key continuity mechanism.

It is important to note that Flash Player only enables sending media from your microphone and webcam devices to other Flash Player endpoints that subscribe to your media streams. Flash Player does not relay data on behalf of any other Flash Player endpoints (such as in a multicast scenario).

For more information on RTMFP, please read the FAQ on Adobe Labs:

• Real‐Time Media Flow Protocol FAQ (PDF, 166K)

ActionScript 3.0 API supporting RTMFP

There is a new ActionScript 3.0 API in Flash Player 10 to support RTMFP. Connecting to the Stratus service and creating end-to-end media streams are analogous to working with Flash Media Server. Please note that you must use ActionScript 3.0 with either Flash Professional CS4 or Flex Builder 3 targeting Flash Player 10 or AIR 1.5.

As I mentioned before, first you must connect to the Adobe Stratus service:

private const StratusAddress:String = "rtmfp://stratus.adobe.com";
private const DeveloperKey:String = "your-developer-key";
private var netConnection:NetConnection;
 
netConnection = new NetConnection();
netConnection.addEventListener(NetStatusEvent.NET_STATUS,
netConnectionHandler);
netConnection.connect(StratusAddress + "/" + DeveloperKey);

The developer key is issued when you sign up for an Adobe Developer Connection account and is available on the Adobe Stratus beta service site.

Upon successful connection to Stratus, you get a NetConnection.Connect.Success event. There could be several reason for connection failure. If you provide an invalid developer key or incorrectly specify Stratus address, you'll receive NetConnection.Connect.Failed. If your firewall blocks outgoing UDP traffic, you'll receive the NetConnection.Connect.Failed event after a 90-second timeout.

After successfully establishing a connection to the Stratus service, you are assigned a unique 256-bit peer ID (NetConnection.nearID). Other Flash Player endpoints must know this peer ID in order to receive your published audio/video streams. It is out of the scope of Flash Player or the Stratus service how these peer IDs are exchanged among Flash Player endpoints. For exchanging peer IDs, you may use an XMPP service or a simple web service, as the Video Phone sample application does.

Direct communications between Flash Player instances is conducted using unidirectional NetStream channels. That is, if you want two-way voice conversation, each Flash Player endpoint must create a sending NetStream and a receiving NetStream.

First, create a sending NetStream:

private var sendStream:NetStream;
 
sendStream = new NetStream(netConnection, NetStream.DIRECT_CONNECTIONS);
sendStream.addEventListener(NetStatusEvent.NET_STATUS,
netStreamHandler);
sendStream.publish("media");
sendStream.attachAudio(Microphone.getMicrophone());
sendStream.attachCamera(Camera.getCamera());

This means that media is published as an end-to-end stream. Since Stratus cannot relay media, you can publish only end-to-end streams. This stream will include both audio and video from your local default devices chosen by the Settings Manager.

Note: Audio/video is not sent out until another Flash Player endpoint subscribes to your media stream.

Now, create the receiving NetStream:

private var recvStream:NetStream;
 
recvStream = new NetStream(netConnection, id_of_publishing_client);
recvStream.addEventListener(NetStatusEvent.NET_STATUS, netStreamHandler);
recvStream.play("media");

At this point, you hear audio and you can create a Video object to display video. In order to create the receiving NetStream, you must know the 256-bit peer ID of the publisher (id_of_publishing_client). In order to receive audio/video, you must know the name of the stream being published.

Advanced topics

The publisher has fine control over which endpoint can receive its published stream. When a subscriber attempts to receive a published stream, the onPeerConnect() method is invoked (default implementation simply returns true) on the published NetStream. The publisher could disallow certain Flash Player endpoints to receive its media:

var o:Object = new Object();
o.onPeerConnect = function(subscriberStream:NetStream):Boolean
{
   if (accept) 
   {
      return true; 
   }
   else
   {
      return false; 
   }
}
sendStream.client = o; 

On the publisher side, the NetStream.peerStreams property holds all the subscribing instances of the publishing NetStream. For example, using sendStream.send() will send the same data to all subscribers. You can use the following to send information to a specific subscriber:

sendStream.peerStreams[i].send()

The NetConnection.maxPeerConnections property specifies the number of peer streams that are allowed to connect to the publisher. The default value is set to 8 but, in practice, depending on your application, you must consider that most ISPs provide asymmetric Internet access. Figure 1 illustrates the direct communication among three instances of Flash Player. Each Flash Player endpoint sends and receives two streams, creating a fully connected mesh. Since Internet download capacity is generally much higher than upload capacity, you must be extra careful not to overload the end-user's uplink.

Figure 1. End-to-end connections using the Stratus service

The NetConnection.unconnectedPeerStreams property is an array of NetStreams that are not associated with a publishing NetStream yet. When a publishing stream matches a subscribing stream name, the subscribing NetStream is moved from this array to the publishing NetStream.peerStreams array.

Exploring the Video Phone sample application

We have developed a sample video phone application for illustrating how to use end-to-end capabilities of Flash Player 10. It is also available as part of this article.

The Video Phone sample application relies on a simple HTTP service to exchange the Flash Player peer ID. The script is provided as part of the package (reg.cgi). This web service does not provide any user authentication. After Flash Player successfully connects to Stratus, it registers its peer ID with the web service. When making a call, the Video Phone caller uses this web service to look up recipient's peer ID.

Adobe runs this web service exclusively for the hosted Video Phone sample. When you build your own Video Phone sample, you must run your own web service and specify WebServiceUrl in VideoPhoneLabs.mxml. You should override the AbstractIdManager class to implement your own peer ID exchange mechanism—using, for example, XMPP, Google Apps, or the Facebook framework.

The following steps are necessary to build a Video Phone sample application (for more details, please see ReadMe.txt included in the package):

1. Host a web service for the peer ID exchange using the provided reg.cgi Python script.
2. Update to Flex Builder 3.0.2 to target Flash Player 10 or AIR 1.5.
3. Create a new Flex project.
4. Add the source files from the package (VideoPhoneLabs.mxml, AbstractIdManager.as, HttpIdManager.as, IdManagerError.as, and IdManagerEvent.as) to the project src folder.
5. Configure your project with Flex SDK 3.2 and target Flash Player 10 or AIR 1.5.
6. Specify your Stratus developer key in DeveloperKey in VideoPhoneLabs.mxml.
7. Specify the URL for the web service in WebServiceUrl in VideoPhoneLabs.mxml.

The Video Phone sample application uses the phone model. The call establishment procedure is implemented using end-to-end NetStream.send() messages. Since you can use the NetStream.send() method only on an established NetStream, Video Phone publishes a so-called "listener stream" (with a fixed name) to which other Flash Player endpoints can connect. When client A (the caller) wishes to communicate with client B (A calls B), he or she subscribes to client B's listener stream. At this point, client B is notified of the peer ID of the caller (using the onPeerConnect() method) and subscribes to client A's media stream. Through this media stream, client A notifies client B about his or her user-friendly name (using the NetStream.send() method), which is presented to the user to either accept or reject the call. If the call is accepted, client B publishes the media stream and two-way communications is established.

Where to go from here

In this article, I presented some of the most exciting features of the new RTMFP protocol along with an overview of the new ActionScript 3.0 API. After reading this article, you should have a good understanding of how to use this revolutionary protocol. I hope you will be developing blockbuster applications using the end-to-end and advanced media capabilities of Flash Player 10 and AIR 1.5.

About the author