Measuring video consumption in Flash
Created
23 May 2011
Requirements
Prerequisite knowledge
Requires advanced knowledge of video on the Adobe Flash Platform runtimes, ActionScript, and Adobe Flash Builder or Adobe Flash Professional.
User level
Advanced
Required products
- Flash Builder (Download trial)
Sample files
One of the most important features of Flash is web video playback. In addition to advanced video compression technologies and efficient playback mechanisms, Flash also enables rich viewing experiences, such as user interaction, social media integration, targeted advertisement, and more.
In order to make most of their content, both website operators and content owners would like to know how video is consumed by users, and then take informed decisions based on actual data. Some of the questions to be answered include how much video has been watched or skipped, whether the user paused the video, viewed content as full-screen, clicked to an advertisement, or performed other actions. Understanding user interaction is invaluable information for video monetization. As such, video measurement is a very active area with several companies offering solutions, such as Omniture (owned by Adobe), comScore, Nielsen, and Google Analytics.
Certain video consumption data has always been present in Flash. Basic information, such as playback time and seek events, is readily available; however, some of the information may not be consistent among different video delivery mechanisms. Furthermore, different video technologies—such as Open Source Media Player (OSMF), Brightcove, JW Player, and others—provide their own APIs to obtain some sort of video consumption information, thereby complicating development of media measurement applications.
Adobe Flash Player 10.3 and Adobe AIR 2.7 improve availability of video consumption information. Adobe's goal is to provide consistent information among different video delivery mechanisms independent of the actual video player used. This would tremendously simplify video measurement for website operators, content owners, and measurement providers. Video consumption now can be derived directly from the Adobe Flash Platform runtimes instead of relying on the actual video player.
This article introduces you to measuring video in Flash. First, I explain three different video delivery mechanisms available in Flash. I also present the new ActionScript API that enables accurate video consumption measurement. Then I describe the basic principles of a video consumption measurement system. Next, I cover security implications. Finally, I present a video player–agnostic measurement application sample that works with both OSMF and JW Player.
Video delivery architectures
Flash provides three fundamentally different video delivery mechanisms: progressive download, streaming, and HTTP streaming. I'll briefly explain all three delivery mechanisms along with small code snippets. If you use a video player application, these underlying details are not visible.
Progressive download
Progressive download is the simplest, and hence probably the most widely used, video playback mechanism offered by Flash. Video files are stored on an HTTP server; all you need to do is to specify the URL of the video. Flash downloads the video to the browser's cache and plays it when a sufficient amount of data is received.
The following sample ActionScript code demonstrates progressive video playback:
private function playProgressive():void
{
var nc:NetConnection = new NetConnection();
nc.connect(null);
var ns:NetStream = new NetStream(nc);
ns.play("http://wwww.example.com/video.flv");
}
The advantage of the scheme is its simplicity (there is no need for a special streaming server) and the automatic use of HTTP caching infrastructure. The disadvantage of the scheme is that you have no control of the video download, limited seeking ability (the user cannot seek to video that has not been downloaded), waste of bandwidth (the entire video is downloaded as fast as network connection can handle, even if it is not viewed), among other drawbacks. As such, progressive download is only applicable to the simplest video applications. Since video is transferred over HTTP, a content-protection mechanism may be required.
YouTube uses progressive video download along with special server-side components to overcome the above-mentioned limitations.
Streaming
Streaming refers to a more advanced video playback mechanism wherein the Flash Platform client obtains video data from a special streaming server, such as Adobe Flash Media Server. Streaming is implemented using Real-Time Messaging Protocol (RTMP), which is an open protocol developed by Adobe.
Client-side ActionScript code for streaming is very similar to that for progressive download:
private function playStreaming():void
{
var nc:NetConnection = new NetConnection();
nc.connect("rtmp://example.com/video");
var ns:NetStream = new NetStream(nc);
ns.play("flv:video");
}
The advantage of RTMP streaming is comprehensive control over buffering and download for optimal performance. Fast start, seeking, trick modes, and multi-bitrate streaming are readily supported. There is no wasted bandwidth; only the necessary video data is downloaded. Video data is not written to disk (browser cache, or any other location) but processed by Flash as soon as it is received.
There are several flavors of RTMP: it can be tunneled over HTTP (RTMPT) or SSL (RTMPS). Flash Media Server also offers an encrypted format (RTMPE), used by many video sites.
HTTP streaming
HTTP streaming was recently introduced in Flash Player 10.1. It combines the advanced features of streaming with the availability of HTTP caching infrastructure. Video files are stored on a web server. In the current implementation, complexity is pushed to the video player application: video download, buffer control, seeking, and many other functions must be implemented by the application developer in ActionScript. ActionScript is responsible for providing video messages as a ByteArray to Flash for parsing, decoding, and presentation. The parser only understands FLV files. If you want to play video encoded in MPEG-4 format, you must parse the MPEG-4 container in ActionScript, which could be quite challenging. Alternatively, as part of Flash Media Server, Adobe offers an HTTP plug-in that is capable of parsing an MPEG-4 container and can be easily used for HTTP streaming.
The following code snippet provides sample code to play an FLV movie:
private var httpStream:URLStream = null;
private function playHttpStreaming():void
{
var nc:NetConnection = new NetConnection();
nc.connect(null);
var ns:NetStream = new NetStream(nc);
ns.play(null);
var request:URLRequest = new URLRequest("http://wwww.example.com/video.flv");
httpStream = new URLStream();
httpStream.addEventListener(ProgressEvent.PROGRESS, onData);
httpStream.load(request);
}
private function onData(e:ProgressEvent):void
{
var b:ByteArray = new ByteArray();
httpStream.readBytes(b, 0, httpStream.bytesAvailable);
ns.appendBytes(b);
}
It is the responsibility of the video player to implement seeking, playlist, multi-bitrate switching, and more (which is not trivial). To implement seeking, for example, the video player needs to buffer video messages, keep track of seekable points, and provide the proper FLV segment to Flash when a seek occurs. The best way to take advantage of all the advanced features offered by HTTP streaming is to use the OSMF video player and Flash Media Server HTTP streaming plug-in.
The advantage of HTTP streaming is the full-application ActionScript control over buffering and use of the HTTP caching infrastructure. Trick modes and multi-bitrate streaming can be readily implemented at the application layer. As with progressive playback, additional content protection may be required.
Media Measurement API
Flash Player 10.3 and AIR 2.7 introduce a new ActionScript API to measure video consumption better. The new API requires SWF version 12 or later. Make sure that you publish your file to SWF 12 and configure your authoring environment with the new playerglobal.swc. The changes can be grouped into three categories:
- Enhanced video playback events
- NetStream monitoring
- Access to page URL
In the following subsections, I'll explain each of these areas in detail.
Enhanced video playback events
One of Adobe's goals is to be able to derive the same information regardless of which method (progressive, streaming, or HTTP streaming) is used for video delivery. To achieve this, we have made the following changes (Table 1 shows the new events highlighted):
- We have added a new
NetStream.SeekStart.Notifyevent for progressive, streaming, and HTTP streaming. This event is always dispatched when a seek is started. It contains the seek start time. Note that you may not get aNetStream.Seek.Notifyevent in progressive mode if the data has not been downloaded yet. NetStream.Play.Complete,NetStream.Pause.Notify, andNetStream.Unpause.Notifyare triggered for progressive playback.
Table 1. NetStream events available for video measurement (new events are in boldface)
| Event | Description | Progressive Download | Streaming | HTTP streaming |
|---|---|---|---|---|
NetStream.Play.Start |
Playback has started | Triggered | Triggered | Not applicable |
NetStream.Play.Stop |
Playback has stopped | Triggered | Triggered | Not applicable |
NetStream.Play.Complete |
The video has been played completely; callback on NetStream.client | Triggered | Triggered | Not applicable |
NetStream.SeekStart.Notify |
Seek has started | Triggered | Triggered | Triggered |
NetStream.Seek.Notify |
The seek operation is complete | Triggered | Triggered | Triggered |
NetStream.Pause.Notify |
The stream is paused | Triggered | Triggered | Triggered |
NetStream.Unpause.Notify |
The stream is resumed | Triggered | Triggered | Triggered |
NetStream.Play.Transition |
The stream transitions to another as a result of bitrate stream switching | Not applicable | Triggered | Not applicable |
NetStream.Play.TransitionComplete |
The stream has changed from one bitrate to another in a multi-bitrate scenario; callback on NetStream.client | Not applicable | Triggered | Not applicable |
NetStream.Buffer.Full |
The buffer is full and the stream will begin playing | Triggered | Triggered | Triggered |
NetStream.Buffer.Flush |
Data has finished streaming and the remaining buffer will be emptied | Triggered | Triggered | Triggered |
NetStream.Buffer.Flush |
Data is not being received quickly enough to fill the buffer | Triggered | Triggered | Triggered |
For HTTP streaming, NetStream.Play.Start and NetStream.Play.Stop cannot be triggered by Flash, since the platform does not know when video is ended. Similarly, multi-bitrate switching must be implemented by the application, and therefore NetStream.Play.Transition and NetStream.Play.TransitionComplete are also not applicable. For progressive download, NetStream.Play.Transition and NetStream.Play.TransitionComplete are not applicable because multi-bitrate is not supported.
For HTTP streaming, it is the responsibility of the video player to dispatch these events; for a code example, please see the following:
package org.osmf.net.httpstreaming
{
public class HTTPNetStream extends NetStream
{
private function signalPlayStart():void
{
dispatchEvent(new NetStatusEvent(NetStatusEvent.NET_STATUS, false, false,
{code:"NetStream.Play.Start", level:"status"}));
}
}
NetStream monitoring
Flash Player 10.3 and AIR 2.7 introduce the NetMonitor class, which enables access to NetStream objects belonging to the same security context. The NetMonitor class has two functions: to return a list of NetStream objects and to dispatch a NetMonitor event when a new NetStream object is created.
To obtain a list of existing NetStream objects, use the following:
var monitor:NetMonitor = new NetMonitor();
var streams:Vector.<NetStream> = monitor.listStreams();
To receive notification when a new NetStream object is created, add an event listener:
monitor.addEventListener(NetMonitorEvent.NET_STREAM_CREATE, netStreamCreate);
to the NetMonitor object, with the following handler:
private function netStreamCreate(e:NetMonitorEvent):void
{
// e.netStream is the new NetStream
}
Once you have access to the NetStream object, you can attach event listeners to be notified about playback events.
We have also extended the NetStreamInfo class to include metadata, XMP data, and other information about the NetStream object. The following properties have been added:
NetStreamInfo.metaData: Last metadata object received by the NetStream object. Usually, video files start with metadata. In addition, Flash Media Server sends an updated metadata object when seeking or executing a multi-bitrate transition.NetStreamInfo.xmpData: Last XMP data object received by the NetStream object.NetStreamInfo.uri: URI used forNetConnection.connect()method; provides similar information toNetConnection.uri. You can use this information to associate a specific NetStream object to its NetConnection object. This is only used for streaming and returnsnullfor progressive and HTTP streaming.NetStreamInfo.resourceName: Resource name used in theNetStream.play()method. This information contains the full URL for progressive download andnullfor HTTP streaming.NetStreamInfo.isLive: This property is only applicable for streaming, and provides information whether the media is live or recorded. This information is communicated to Flash Player by Flash Media Server.
Finally, we have added a new event termed NetDataEvent that is dispatched by the NetStream object when a data message is processed. This is applicable for all media delivery mechanisms. Data messages are processed by the NetStream.client object as a callback. The disadvantage is clear: if you would like to have multiple listeners, you need to develop a proxy mechanism. By turning these callbacks into standard Flash events, you can easily capture these messages. By using NetDataEvent, monitoring application has access to metadata, NetStream.Play.Complete, NetStream.Transition.Complete, and more.
Access to page URL
We have extended the Security class to provide the page domain. The full page URL is not exposed for security reasons. Although you can obtain the page URL using an external interface, this is a much more reliable, convenient, and consistent way across different platforms and browsers. When JavaScript is disabled in the browser, Security.pageDomain may not be able to return the proper page domain.
The new method has the following advantages over using the Flash ExternalInterface class:
- An external interface only works when JavaScript is enabled.
- An external interface requires
allowScriptAccessto be set to a permissive value in the HTML wrapper. - The
pageDomainmethod also works when the page URL and the SWF file URL are coming from different domains; beforehand, you needed to setallowScriptAccess="always"for an external interface to work.
Measuring video consumption
In a typical scenario, video files may be hosted by the website operator, content owner, or third-party distribution companies. The video player is responsible for loading and playing these video files. The video player can either be Open Source Media Framework (OSMF), the Brightcove player, JW Player, FLVPlayback, or another SWF-based player. The video player is most commonly hosted by the website operator. Both the website operator and the content owner would like to collect video consumption data, which is achieved by using a measurement application. A typical measurement application collects video consumption data, aggregates it, and sends it to a server for further processing and presentation.
Most of the time, the measurement application is a video player plug-in. Note that the developed measurement application must be in the same security context as the video player.
Using the new APIs, you can make the measurement application directly access the NetStream object of video player; that is, you can develop a single measurement application that is agnostic of the video player and directly derives video consumption information from the NetStream object.
The typical measurement application would register an event listener to be notified when a NetStream object is created and store NetStream objects in a list. By listening to NetStatus events, the video state could be determined (playing, paused, seek, ended, etc.). If the measurement application was loaded after video playback, it can still access NetStream objects through NetMonitor and the corresponding metadata and XMP data through NetStreamInfo. Usually, measurement applications include a heartbeat timer, which can periodically record the playhead position using the NetStream.time property and transmit video consumption data to a measurement service.
Figure 1 depicts a typical usage scenario. The website operator hosts the video player and measurement application. Content is provided by third-party hosting to ensure wide availability.
Privacy
The focus of the described enhancements is to make video measurement simple to implement. There must be a trust relationship between video player and measurement application (and thus among website operator, content owner, and measurement provider). With the new APIs, you can only access video player objects that belong to the same security context as your measurememt application. For example, if your video player plays an advertisement with a plug-in that belongs to a different security context, your measurement application will not be able to obtain any information about the advertisement.
The website operator must explicitly load the measurement application (directly or indirectly), which implies that the user agrees to the terms and conditions of the website.
Measurement data needs to be communicated to a server for aggregation and presentation. It is the responsibility of the measurement application, and thus the website operator, to ensure secure transmission of video consumption data as needed— for example, by using HTTPS.
Sample application
The following sample application below demonstrates a typical usage scenario. The main application (videoPlayer.swf) loads two child SWF files: the video player application and the measurement application (videoMonitor.swf). All three applications must be hosted by the same web server and be in the same security context.
The sample uses Strobe Media Playback to play a progressive video file. We have not made any modifications to Strobe Media Playback; we have used the prebuilt SWF application. The main application loads Strobe Media Playback and passes the video URL (stored on a different server):
private static const PlayerUrl:String = "StrobeMediaPlayback.swf?src=http://osmf.org/videos/cathy2.flv"
If you want to use streaming, just update the URL to point to your FMS server:
private static const PlayerUrl:String = "StrobeMediaPlayback.swf?src=rtmp://example.com/videos/flv:cathy2"
You can easily replace Strobe Media Playback with any other video player. For example, if you want to use JW Player, download and copy it to the document root directory of your web server and update videoPlayer.as:
private static const PlayerUrl:String = "player.swf?file=http://osmf.org/videos/cathy2.flv"
The measurement application demonstrates the following capabilities:
- Recording the page domain
- Using
NetMonitorto receive notification when new a NetStream object is created - Receiving metadata data message as an event
- Receiving
NetStream.Play.Completecallback as a NetDataEvent - Periodically capturing playback time
- Capturing the NetConnection URI and resource name using NetStreamInfo
- Capturing certain playback events (seek and pause)
- Periodically sending video consumption data to a web service
Setting up the sample application is straightforward. The package contains the following three files:
- videoPlayer.as: This is the main application that loads the video player and the monitor application. Please configure the URL of the video player. To build it, create a "videoPlayer" ActionScript project in Flash Builder. You must require Flash Player version 10.3 and target SWF version 12. Copy the resulting videoPlayer.swf and supporting files to the document root directory of your web server.
- videoMonitor.as: This is the video monitor application. Upon load, it creates a NetStream monitor and a periodic timer. The application maintains a list of NetStream objects, but for simplicity, it only reports information about the first NetStream object. Please configure the URL of your web server to send video consumption information. The application uses a query string to pass parameters to the monitor service. To build it, create a "videoMonitor" ActionScript project in Flash Builder. You must require Flash Player version 10.3 and target SWF version 12. Copy the resulting videoMonitor.swf to the document root directory of your web server.
- monitor.cgi: This is a helper application to parse and log video consumption data from the video monitor application. This is a simple Perl script and must be copied to the cgi-bin executable folder of your web server. Sample output:
Page Url: http://153.32.146.208/, Action: HeartBeat, Uri: null,
resourceName: http://osmf.org/videos/cathy2.flv, position: 42.657
In addition, download a video player (OSMF or JW Player, and configure videoPlayer.as accordingly) of your choice and place it in the document root directory of your web server.
Where to go from here
This article describes the new ActionScript API of Flash Player 10.3 and AIR 2.7 that Adobe has designed to facilitate easy measurement of video consumption. The new API provides powerful tools for website operators and content owners to derive video consumption information—yet at the same time conforms to the security model of Flash Player and complies with website privacy rules. By using these new features, you will be able to build powerful media measurement applications and better understand how consumers interact with your website and media content regardless of your chosen video player.
+
This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 Unported License. Permissions beyond the scope of this license, pertaining to the examples of code included within this work are available at Adobe.
