Exporting instant, edited on-demand highlights clips during live events with Flash Media Interactive Server
Created
8 February 2010
Requirements
Prerequisite knowledge
Adobe ActionScript 3, Adobe Flex, and Adobe Flash Media Server experience
User level
Intermediate
Required products
- Flash Media Interactive Server
- Flash Builder (Download trial)
Sample files
- fmis_video_editor.zip (3288 KB)
This article demonstrates how to build an application that uses Adobe Flash Media Interactive Server (FMIS) as an editing module for live events without requiring any DVR features on a content-delivery network (CDN).
Due to the broad reach of the Adobe Flash Platform, Adobe Flash Player is the preferred way to deliver video on the web. With its ubiquitous penetration and its reach across multiple screens—including browsers, operating systems, desktops, mobile devices, and Digital Home—the Flash Platform enables video providers to maximize the reach and the user experience. It is the solution for some of the largest-scale live webcasts in history.
The nature of these events often requires capturing instant highlights and providing the user the option to access them instantly. One of the features that Flash Media Interactive Server 3.5 introduced is DVR functionality, which allows seeking back in time during a live event. (Although the feature to record a live video was available prior to FMIS 3.5, the DVR functionality specifically allows read access while the file is written to disk.)
However, simply allowing the user to rewind to random segments is often not enough. How great it would be to edit and extract segments during a live event as a broadcaster, without having to rely on advanced CDN features or having to wait until after the end of the event, when the on-demand file has finished recording!
Application architecture, DVR concepts, and DVR live recording
Before you dig into this application, you need to understand how DVR functionality works in FMIS, how the application is architected, and how to use it in a real-world live editing scenario. Here I'll outline the components of the application and its integration into existing workflows.
On-demand file recording
The FMIS feature that this application leverages is the ability to generate server-side playlists from multiple sources (such as on-demand files and live recordings while written to disk), play the playlist as a background task on the server, and record the playlist as a merged file.
Flash Media Server stitches the segments together, even if the source segments have different encoding characteristics. The output is a F4V file (derived from the MP4 container), and can be streamed through FMIS instantly. In case the preferred delivery is HTTP progressive download, Flash Media Server provides a flattening tool (Adobe F4V post processor) to make the recording compatible with progressive downloads (see FMS Tools). In this case, Adobe recommends that you have the same encoding characteristics for each segment of the playlist.
Architecture
As I mentioned in the introduction, this architecture doesn't rely on DVR functionality of a CDN. If you are hosting your own FMIS, or the CDN of your choice supports DVR functionality, it's possible to simply play segments of the recorded DVR file as highlights. For this article, I assume the preferred method is to export physical F4V files as highlights—with the ability to store them in archives or push them to a CDN.
Figure 1 describes the overall setup. A live encoder (FMLE or third party) pushes the RTMP signal to a DVR application on the Flash Media Server. The DVR application is based on the DVRCast server-side script (see FMS Tools) and enables you to record the live event during playback.
The highlight editor is a separate FMIS application with a client-side UI based on Adobe Flex as well as a server-side component (server-side ActionScript). The editor application running on the Flash Media Interactive Server instance has access to the following sources to create highlights:
- The real-time recorded live stream from the DVRCast application (separate application)
- An set of prerecorded video clips stored in the editor application—for example, to add openers or mix the extracted highlight with existing content
These are the two footage sources that this application uses, but it would be easy to add additional live DVR sources or clips from different applications on the same server, or even remote servers.
DVR live recording
The DVRCast application is part of the FMS Tools and provides a standardized API to control the DVR behavior (such as to start and stop the DVR recording). Once you've downloaded the DVRCast application, copy the dvrcast_origin folder to your FMIS application folder. The DVRCast application is compatible with Flash Media Live Encoder (FMLE) 3 and the FLVPlayback 2.5 component.
FMLE has controls as part of the interface to start/stop the recording of the live stream. In case you are working with a third-party encoder that doesn't provide this functionality, my article Live dynamic streaming and DVR for non-developers contains a DVRController application to control the DVR behavior separately.
We will use the FLVPlayback 2.5 component to actually play and rewind the stream.
Editor application
This section describes the core of the application, including the client-side UI of the editor application (written in Flex) and the required server-side ActionScript.
I recommend looking at the source code itself in parallel while reading through the section. For simplicity reasons, I highlight the most relevant parts of the source in context of the application flow but the source includes the complete UI definitions and imports.
Interface
Before we explore the source code, I will walk through the functionality of the interface (shown in Figure 2), which will better explain the flow.
The application contains the following functional areas, shown in detail in Figure 3:
- FLVPlayback 2.5–based video player with DVR optionally enabled
- A list of available on-demand clips (dynamically received from FMIS)
- Available live DVR streams (in our case, just one)
- Ability to select a source clip, set in and out points, and add the segment to a playlist
- Ability to generate the output clip
- List of exported output files (dynamically received from FMIS)
Now I'll show you how the source code achieves this.
FLVPlayback component
FLVPlayback is a Flash component primarily known from Flash authoring. Because it is available as a SWC version, it's possible to load the component into the Flex framework. Move the FLVPlayback_2.5.swc file from the FMS Tools website into the libs folder of your Flex project and use the player as a component in Flex. Once Flex is initialized (via the creationComplete event), use the following code to configure and display the player:
// Initializes the application
private function init() : void {
videoplayer = new FLVPlayback();
var ui:UIComponent = new UIComponent();
this.addElement( ui );
ui.addChild( videoplayer );
videoplayer.x=315;
videoplayer.y=0;
videoplayer.width = 1280;
videoplayer.height = 720;
videoplayer.bufferTime = 0.1;
videoplayer.skinBackgroundColor = 0x666666;
videoplayer.skin = "SkinUnderPlayStopSeekFullVol.swf";
videoplayer.scaleMode = VideoScaleMode.MAINTAIN_ASPECT_RATIO;
videoplayer.skinAutoHide = false;
}
The FLVPlayback component will play the DVR source, the on-demand files, and the exported files. To support these different types, it's possible to reconfigure the player instantly.
The following functions will play those types:
// Event listener for the on-demand source data grid
private function sourceGridClick(e:MouseEvent) : void {
videoplayer.isDVR = false;
videoplayer.isLive = false;
videoplayer.source = "rtmp://"+fmsserver+"/editor/mp4:"+sourcegrid.selectedItem;
isDVR = false;
}
// Event listener for the output data grid
private function outputGridClick(e:MouseEvent) : void {
videoplayer.isDVR = false;
videoplayer.isLive = false;
videoplayer.source = "rtmp://"+fmsserver+"/editor/mp4:"+outputgrid.selectedItem;
isDVR = false;
}
// Event listener for the live source data grid
private function sourceLiveGridClick(e:MouseEvent) : void
videoplayer.isDVR = true;
videoplayer.isLive = true;
videoplayer.source = "rtmp://"+fmsserver+"/dvrcast_origin/mp4:"+livestreamgrid.selectedItem.name + ".f4v";
isDVR = true;
}
On-demand video clips
Before you can play the available on-demand clips, you need to get a list of available clips from FMIS and populate the datagrid. The first step in this process is to open a connection to the editor application:
// Initializes the application
private function init() : void
{
[...]
nc = new NetConnection();
nc.addEventListener(NetStatusEvent.NET_STATUS,onConnect);
nc.connect("rtmp://"+fmsserver+"/editor/");
[...]
}
Once the connection is successful, you can receive a list of all available source files from the server (and also the already exported files, which uses the same functionality) by calling user-defined FMIS server-side functions:
// Load the latest list of available source and exported videos
clips from FMIS
private function onConnect(status:Object) : void {
refreshVideos();
refreshVideosOutput();
}
// Receive a list of the available source clips from the server
private function refreshVideos() : void {
nc.call("getFilesSource", new Responder(onVideoListSource));
}
In the editor application on FMIS, the following function is defined:
Client.prototype.getFilesSource = function() {
var fileList = new File("/streams/_definst_/source");
var temp = fileList.list();
var returnvalue = new Array();
for (var i=0;i<temp.length;i++) {
returnvalue.push(temp[i].name);
}
return returnvalue;
}
It will return the data, and the client-side editor application receives the data with the onVideoListSource() responder:
// Assign the returned data to the datagrid
private function onVideoListSource(thumblist:Array):void{
movielist = new ArrayCollection();
for (var foo:Object in thumblist) {
thumblist[foo] = thumblist[foo].split("/streams/_definst_/")[1];
}
movielist.source = thumblist;
sourcegrid.dataProvider = movielist;
nc.call("getLiveStreamId", new Responder(onLiveStreamReceived));
} }
The onVideoListSource() function then converts the data into an ArrayCollection and assigns it as dataProvider to the source datagrid. Since all output files are stored in the same directory in this application, a unique ID would be useful. FMIS has a unique ID per user, and the getLiveStreamId() method will return this ID, which then will be used as part of the filename for the exported file. This is primarily to allow multiple users to use the application in parallel with the same export folder structure for simplicity, without implementing user management. It's certainly an area for improvement—for example, by providing users their own individual export folders.
Live DVR streams
It's possible to use similar dynamic FMIS calls to receive a list of available DVR streams, but since there's only one DVR stream in this use case, I'm taking a shortcut and defining the available DVR live streams manually:
// Initializes the application
private function init() : void {
[...]
var item:Object = new Object();
livestreamdp = new ArrayCollection();
item.name = "livestream1";
livestreamdp.addItem(item);
livestreamgrid.dataProvider = livestreamdp;
[...]
}
Playlist
Now you want to enable the user to create a playlist based on the available on-demand and live soures, set the in and out points, and send the playlist to FMIS for extraction. First, define the playlist data provider:
// Initializes the application
private function init() : void {
[...]
playlistdp = new ArrayCollection();
playlistgrid.dataProvider = playlistdp;
[...]
}
An interval function keeps updating the current time code display during playback. When the user finds the right in and out point, the following functions allow one to set them:
// Update the current time code
private function updateTimeCode() : void {
currenttimecode.text = videoplayer.getVideoPlayer(0).playheadTime.toString();
}
// Define start time for the edit
private function setInPoint(e:MouseEvent) : void {
inpoint_txt.text = currenttimecode.text;
}
// Define end time for the edit
private function setOutPoint(e:MouseEvent) : void {
outpoint_txt.text = currenttimecode.text;
}
Once the source clip is selected, and the in and out points are defined, you can add the segments to the playlist:
// Add selected segment to edit list
private function addToPlayList(e:MouseEvent) : void {
status_txt.text = "";
var item:Object = new Object();
if (isDVR) {
item.name = livestreamgrid.selectedItem.name;
}
else {
item.name = sourcegrid.selectedItem;
}
item.starttime = inpoint_txt.text;
item.endtime = outpoint_txt.text;
item.livesource = isDVR;
playlistdp.addItem(item);
playlistgrid.dataProvider = playlistdp;
// Call the FMIS server script with the edit list
private function generateCutList(e:MouseEvent) : void {
status_txt.text = "Creating output file...";
nc.call("generateOutputFile",null,playlistgrid.dataProvider.source,"mp4:output/"+newStream + "_" + (currentExportId++) +".f4v");
}
Each playlist item contains an information object for FMIS to create the playlist and export the file. FMIS needs to know if the file is a DVR or on-demand file, and the start and end time. We also pass in the unique client ID and an export ID to ensure that the stream is unique.
Once the playlist is complete, send the list off to FMIS:
// Call the FMIS server script with the edit list
private function generateCutList(e:MouseEvent) : void {
status_txt.text = "Creating output file...";
nc.call("generateOutputFile", null,playlistgrid.dataProvider.source,"mp4:output/"+newStream + "_" + (currentExportId++) +".f4v");
}
Extraction of clips
This is the core component of the application, which uses the server-side playlist and recording feature of FMIS.
Once FMIS receives the playlist, it assembles with the playlist items defined by the user. The playlist items can be local, but also live in other applications or even FMIS servers. In our case the DVR live stream is located in another application (dvrcast_origin) on the same server.
This code implements the editor application's functionality on the FMIS server:
// Accept the user connection
application.onConnect = function(clObj) {
this.acceptConnection(clObj);
}
// Connect to the DVR stream on application start
application.onAppStart = function() {
this.dvrcast_nc = new NetConnection();
this.dvrcast_nc.onStatus = function(info){
trace("Connection to remote server status " + info.code + "\n");
};
// Use the NetConnection object to connect to a remote server.
this.dvrcast_nc.connect("rtmp://myfmsserver/dvrcast_origin/");
}
Client.prototype.getFilesSource = function() {
var fileList = new File("/streams/_definst_/source");
var temp = fileList.list();
var returnvalue = new Array();
for (var i=0;i<temp.length;i++) {
returnvalue.push(temp[i].name);
}
return returnvalue;
}
Client.prototype.getFilesOutput = function() {
var fileList = new File("/streams/_definst_/output");
var temp = fileList.list();
var returnvalue = new Array();
for (var i=0;i<temp.length;i++) {
returnvalue.push(temp[i].name);
}
return returnvalue;
}
Client.prototype.getLiveStreamId = function() {
return this.id;
}
Client.prototype.generateOutputFile = function(playlist,streamname) {
this.streamname = streamname;
this.recordstream = Stream.get(this.streamname);
this.recordstream.record();
if (this.recordstream) {
this.recordstream.onStatus = function(info) {
trace(info.code);
}
}
var totalrecordingtime = 0;
for (var i=0;i<playlist.length;i++) {
if (!playlist[i].livesource) {
trace("mp4:"+playlist[i].name,Number(playlist[i].starttime),Number(playlist[i].endtime)-Number(playlist[i].starttime));
this.recordstream.play("mp4:"+playlist[i].name,Number(playlist[i].starttime),Number(playlist[i].endtime)-Number(playlist[i].starttime),false);
}
else {
this.recordstream.play("mp4:"+playlist[i].name+".f4v",Number(playlist[i].starttime),Number(playlist[i].endtime)-Number(playlist[i].starttime),false,application.dvrcast_nc);
}
totalrecordingtime += playlist[i].endtime-playlist[i].starttime;
}
clearInterval(this.endVideoINT);
this.endVideoINT = setInterval(this.endOfRecording,totalrecordingtime*1000,this);
}
Client.prototype.endOfRecording = function(reference) {
reference.recordstream.flush();
reference.recordstream.record(false);
clearInterval(reference.endVideoINT);
reference.call("recordingComplete");
}
When you create the application, please make sure you use the correct folder structure:
[FMS_APPLICATION_FOLDER_PATH]/editor/main.asc (this section)
Move all your on-demand clips to the following location:
[FMS_APPLICATION_FOLDER_PATH]/editor/streams/_definst_/source/
The output clips will be exported here:
[FMS_APPLICATION_FOLDER_PATH]/editor/streams/_definst_/output/
The generateOutputFile() function described above actually exports the playlist. Once the playlist is programmed with subsequent NetStream.play() calls on the dynamically generated stream, FMIS starts the playback and the recording. Once the end of the playlist is reached (tracked through a timer) the recording stops and FMIS sends a complete event back to the Flex front end:
var clientResponder:Object = new Object();
clientResponder.recordingComplete = function() : void {
status_txt.text = newStream + "_" + (currentExportId-1) + ".f4v
created";
refreshVideosOutput();
}
The front end updates the output video list, allowing the user to select the newly exported clip. And that's it! You just exported an edited highlight clip in real time during a FMS-based live event.
Future improvements
This sample application is initially very basic. There's a lot of room for improvements. Here are some you could make:
- Dynamic streaming support with multiple bit rates
- Previewing the edit with a client-side playlist
- Accommodating multiple live sources
- A visual editor simulating professional editing tools
- A playlist preview function
- Multiuser management and collaboration
- Adding cue points to the recordings for ad breaks
- A time-shift function to define future out points and export the clips in parallel
I hope this article has inspired you about the power of the interactive features of FMS and will give you a lot of new ideas for how to create a better user experience for your live events.
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License
