Eliminating the single point of failure with origin redundancy in Flash Media Server

Robert A. Colvin

pixelonda.com

Today's web users are more and more demanding when it comes to viewing their favorite content, media, and applications online. Their needs for an always-on experience are pushing providers to deliver content that meets (and exceeds) their increasingly high expectations. That's why if you don't have a failover strategy, a glitch in your Flash Media Server origin could cause you serious problems.

Macromedia Flash Media Server 2 from Adobe has made great strides with edge/origin configuration, making it easier and faster for you to deploy applications that handle incredible loads. If you run it out of the box, you can easily set up an edge cluster to handle your incoming load while running your application/content from a single origin (see Figure 1). However, the origin could create havok if it should ever fail without a well-architected failover solution in place.

Figure 1. Standard edge/origin FMS environment

Note: Edges can cache content, so it may not actually be relying on the origin when serving out cacheable files.

The goal of this article is to provide some insight into a possible solution to help you better design an overall scale-up and scale-out architecture that accounts for performance, backups, and redundancy to virtually guarantee high reliability. Think of full redundancy as no discernable interruption of service, employing a duplicate to prevent failure of an entire system. If mission-critical problems caused by hardware or software failures do occur, this failover solution (coupled with other methods) can enable a production environment never to go down, at least from a user's perspective (see Figure 2).

Figure 2. Proposed fully redundant, architected edge/origin environment

Requirements

To complete this tutorial you will need to install the following software:

Flash Media Server 2

• Try
• Buy

Any enterprise-level load balancer

Any enterprise-level, high-speed NAS (network attached storage)

Sample files:

• origin_redundancy.zip (ZIP, 296 KB)

Prerequisite knowledge

Familiarity with Flash Media Server 2 and an understanding the basics of ActionScript on the server.

Configuring the NAS

I assume that you have limited access to the NAS (network attached storage), so you will have to make a request to those with full access to create an account with read/write/create privileges. Keep in mind that this account must also be an administrator on the origin server.

You will also need to create a config file. into which you'll write the two origin names (or IPs) in order—primary\secondary, as follows:

MyOrigin001\MyOrigin002

In the example cited in this article, the file is named RedundancyMainConfig.txt but it could be named anything. Just update VBS file to reflect the change.

Configuring the load balancer and edge servers

The load balancer should have been configured to add both primary and secondary origins to the pool—which should already be configured. The pool, also known as the server farm, is a reference to a load balancer configuration that retains a number of servers, across which it load-balances traffic. In round-robin mode, it should be monitoring over TCP port 1935. Figure 3 shows BIG-IP's admin console.

Note: All administration consoles are different. Consult your operations team before making any of these changes if you have the ability to make them.

Figure 3. BIG IP's admin console, demonstrating Steps 1–4 to add a monitor

The configuration steps shown in Figure 3 describe the following:

1. Click Monitors in the left column.
2. Click the Basic Associations tab.
3. After checking the respective column to add, click Apply.
4. Note that the monitor now works using TCP/IP.

The virtual IP (VIP) address of the load balancer for this pool should replace the current IP address in all the edge servers' Vhost.xml files located in <your installation directory>\conf\_defaultRoot_\_defaultVHost_\Vhost.xml, as follows:

routeEntry tag <RouteEntry>*:*; 10.269.20.06:1935</RouteEntry>

Here 10.269.20.06 should be changed to point to the VIP of the load balancer.

IP filtering and failover on origin servers

Now that you have your load balancer set up, you need to add a rule or policy that can be enabled or disabled. In Windows we will be using Windows Firewall and deny any machine to listen on TCP port 1935 to this box unless its IP address is that of the alternate origin. Then we will create a scheduled task to execute the script below.

Note: You can name this code however you please. In the attached ZIP file I have named it OriginFailover.vbs, and the script must reside on both the primary and secondary origin. You will need to point your scheduled task to run it.

Check4allvar = "\secondaryUP"

function isFailOver(aN)
   dim aL,lastString ,valid
      aL=UBound(aN)
      lastString=trim(aN(aL))
      if lastString="secondaryUP" then
          valid=true
      else
          valid = false
      end if
      isFailOver=valid
end function
function isPrimary(txtPrimary,serverName)
   if txtPrimary=serverName then
      valid=true
   else
      valid=false
   end if
   isPrimary=valid
end function

'Firewall state change function
function FirewallState(state)
   Set oFirewall = CreateObject("HNetCfg.FwMgr")
   Set oPolicy = oFirewall.LocalPolicy.CurrentProfile
   oPolicy.FirewallEnabled = state
end Function

'Firewall check state function
function checkFirewall()
   Set oFirewall = CreateObject("HNetCfg.FwMgr")
   Set oPolicy = oFirewall.LocalPolicy.CurrentProfile
   checkFirewall=oPolicy.FirewallEnabled
end Function

'read vars
function readFromfilevars(filename2)
   fullFilename = filename2
   path2file = fullFilename
   dim fs2, file2
    set fs2 = CreateObject("Scripting.FileSystemObject")
   if fs2.FileExists(path2file) then
      set file2 = fs2.OpenTextFile(path2file, 1)
      fileContents2=file2.ReadAll
      file2.Close()
      valid2 = fileContents2
    else'file does not exist approve
       valid2="null"
    end if
    readFromfilevars=valid2
end function

'WriteFile FUNCTION: write to file as soon as primary is down
function WriteFile(filename,dataIn)
  Const ForWriting   = 2
   path = filename
   dim fs, file
    set fs = CreateObject("Scripting.FileSystemObject")
    if fs.FileExists(path) then
      set file = fs.OpenTextFile(path, ForWriting)
      file.Write(dataIn)
      file.Close()
    else
      'failed
    end if
end function

'readFromfile FUNCTION: see if file exists and, if it does, find the searchWord
function readFromfile(filename2,searchWord2)
      Const File2Read   = 1
      fullFilename = filename2
      path2file=fullFilename
 dim fs2, file2
    set fs2 = CreateObject("Scripting.FileSystemObject")
  if fs2.FileExists(path2file) then
      set file2 = fs2.OpenTextFile(path2file, File2Read)
      fileContents2=file2.ReadAll
      file2.Close()
      valid2 = Validate2compare(fileContents2,searchWord2)
   else'file does not exist approve
      valid2 = false
   end if
    readFromfile=valid2
end function

function Txtcreate(aN)
   allString=""
    Total = uBound(aN)
    for i=0 to Total
      if i<>0 then
       allString=allString+"\"
       end if
       allString=allString+aN(i)
    Next
    Txtcreate=allString & Check4allvar
end Function


'retrieveData FUNCTION: grab data from a server
function retrieveData(mURL)
      Set MyConnection = CreateObject("WinHttp.WinHttpRequest.5.1")
      ' Connecting to the URL
      MyConnection.Open "GET", mURL, False
      ' Sending and getting data
   On Error Resume Next
      MyConnection.Send
   if Err.Number <> 0 Then
      retrieveData="failed"
   else
      dData = MyConnection.responseText
      Set MyConnection = Nothing
      'Set the appropriate content type
      'Response.ContentType = MyConnection.getResponseHeader("Content-Type")
   end if
      retrieveData=dData
end function
function Validate2compare(stringIn,wordIn)
   Dim Valid
   'TODO search for rejected if password is incorrect before failing over
   set regx = New RegExp
   with regx
      .Pattern = wordIn
      .IgnoreCase= true
      .Global = true
   end with
   Set expressionmatch = regx.Execute(stringIn)
   If expressionmatch.Count > 0 then
      'success found so check the other server
      Valid=true
   else
      Valid=false
   End If
   Validate2compare=Valid
end function


'VARS
function initialize()
   Dim MyConnection,Now1, FMSadminURL, Primary,Secondary, Uname, Pwd, port, Check4allvar, Fullpath,aNms,Prod
      Prod=Split("YourProdPrimeOrigin\YourProdSecondOrigin","\")
   if  CreateObject("Shell.LocalMachine").MachineName <> Prod(1) AND CreateObject("Shell.LocalMachine").MachineName <> Prod(0) then
      Fullpath="\\YourStagingNas\FMSfiles\MainOriginConfig.txt"
   else
      Fullpath="\\YourProductionNas\FMSfiles\MainOriginConfig.txt"
   end if
      aNms=Split(readFromfilevars(Fullpath),"\")
      Primary=aNms(0)
      Secondary=aNms(1)
      port="1111"
      Uname="adminUsername"
      Pswd="youradminPass"
      FMSadminURL = "http://"&Primary&":"&port&"/admin/ping?auser="&Uname&"&apswd="&Pswd
'////////////////////////////////////////
'Main procedure

      TheData1=retrieveData(FMSadminURL) 'before sending the data check the other server to see if it is up
   if Validate2compare(LCase(TheData1),"rejected") then
      run1=true
   else
      run1=Validate2compare(LCase(TheData1),"success")
   end if
      'get this server name address and compare with the primary from txt file
      if isPrimary(LCase(aNms(0)),LCase(CreateObject("Shell.LocalMachine").MachineName)) then
      'this is the Primary
      if run1 then'success 'if text file  "backup" if so skip to down
         if isFailOver(Split(readFromfilevars(Fullpath),"\")) then
            'make sure the firewall is kept on 'til text file changes are made
            if (checkFirewall()) then
            else
            FirewallState(TRUE)
            end if
         else  'make sure the firewall is off
            if (checkFirewall()) then
               FirewallState(false)
            end if
         end if
      else 'no success 'server is actually down 'turn firewall on
         if (checkFirewall()) then
         else
            FirewallState(TRUE)
         end if
      end if
      else  'Secondary Origin ...is primary up?
   if run1<>true then'no success ' O1 is down so write to file
      if isFailOver(Split(readFromfilevars(Fullpath),"\")) then
         if (checkFirewall()) then
            FirewallState(false)
         end if 'server is already stated to failover don't write again
      else
         call WriteFile(Fullpath,Txtcreate(Split(readFromfilevars(Fullpath),"\")))
         FirewallState(False)

      end if 'turn firewall off
   else '02 is down 'turn firewall on
      FirewallState(TRUE)
   end if
   end if
end Function
initialize();

'////////////////end

This script basically grabs the information from RedundancyMainConfig.txt on the NAS and uses that information to listen in on the admin's ping response. If it finds a string other than "successful," the script turns on the primary origin's firewall, which blocks the load balancer's monitor. On the secondary (standby) origin, it turns off Windows Firewall, making it the only available choice for the load balancer.

If the script is runnning on the secondary origin and the primary goes down, the VBS script file shown above writes to the NAS. This is a precautionary measure so that when the primary is restarted, it doesn't automatically switch the clients back. This allows you to analyze what went wrong before resetting to the primary by removing \secondaryUP from RedundancyMainConfig.txt.

Note: The scheduled task allows only a minimum of one minute. To get to within seconds, you will need to create a Windows service, which can either call the VBS script or embody its functionality.

Most of the logic in the above script is straightforward and can easily be ported or refactored for your needs. The main function you may need to modify for your language or needs is the following:

function FirewallState(state)
   Set oFirewall = CreateObject("HNetCfg.FwMgr")
   Set oPolicy = oFirewall.LocalPolicy.CurrentProfile
   oPolicy.FirewallEnabled = state
end Function

In Linux, assuming your kernel has iptables support built in, you can execute a command with similar results:

#-----you may need to clear rules out first...
IPTables=/sbin/iptables
$ IPTables -A INPUT -s <Insert other Origin ip here> -p tcp --source-port 1935 -j ACCEPT
$ IPTables -A OUTPUT -d < Insert other Origin ip here > -p tcp --destination-port 1935 -j ACCEPT

This code is a simple example of how to add a rule to iptables. What is not shown is how to deny access to all possible IPs and then add rules—much like the one shown above—to allow a specific server, load balancer, or IP. The main idea here is to exercise control over when you allow the load balancer VIP access on TCP port 1935 to the primary or secondary origin based on whether the primary is up or down.

If the primary origin is down, you will not want the loadbalancer to see it, so you must block it. The secondary origin will have to allow the loadbalancer because it was previously blocking it to force all traffic to the primary. You can do this by clearing iptables on the secondary origin and allowing only the loadbalancer VIP and primary origin access to the secondary origin. Remember at all times that the primary and secondary origins should allow each other access to keep state of any application.

Tip: Be careful when setting up your cron job or scheduled task (do not use a loop), because if you create millions of calls to the admin per second, the admin could fail, causing the origin to fail over. (Remember that both the primary and secondary origins will be listening in on the primary's admin ping result.)

Configuring the origin server

Find the application.xml file located in <your installation directory>\conf\_defaultRoot_\_defaultVHost_\application.xml.

Create a virtual directory that maps to a given directory in your NAS. In the example, I use myFMSfiles like the following:

<VirtualDirectory>/ myFMSfiles;\\<NASdir>\FMSfiles\</VirtualDirectory>

Look for the <FileObject> tag. Inside that tag, you will create the <VirtualDirectory> tag, which may show an example commented out.

In the StreamManager within application.xml, find the <StorageDir> tag in order to define the root where any recorded streams should be stored: <StorageDir>\\<NASdir>\FMSfiles\</StorageDir>.

Tip: If you are going through development and staging environments before you hit production, it is a good idea to use an <ApplicationObject> tag (which should exist within the <JSEngine> tag) in defining your environments, like so:

<ApplicationObject> <config> <dept_name>Tech</dept_name><server_name>Origin1</server_name>
<env_name>staging</env_name></config></ApplicationObject>

This will become more clear when we move on to examine the server-side code.

If you are on a Windows server, go to the Services panel, which you can access by selecting Start > Run and then type services.msc in the Run dialog box (see Figure 4).

Figure 4. Opening the Services panel

In the Services panel, find Flash Media Administration Server and Flash Media Server. First stop both these services. For each service, right-click the service, select Properties, select the Log On tab, and enter the username and password that you set up in the NAS configuration setup (see Figure 5).

Figure 5. Setting up properties for services

Architecting your solution to keep persistent state

The intention in the following steps is to demonstrate a solution—to push state from primary to secondary—that accounts for data which may be in a sharedObject (RSO) or variable. You need to make sure that even if Windows Firewall or iptables rules are running, the origins' IPs should always be open to each other (primary/secondary) as described earlier in section, "IP Filtering and Failover on Origin Servers." The solution also requires you to organize all calls made to the server so that it can easily and securely be called on the secondary origin.

Step 1. Create a connection

The first step is to create a connection from the primary origin, receiving the clients to the secondary (semi-dormant) origin. To do this correctly, you will have to identify whether any users are currently connected to the primary origin. If there are, you will then connect to the secondary (semi-dormant) origin. The example shown below is included in the main.asc file.

application.onConnect = function(p_client, userName, userType){
   if ((p_client.agent.substr(0,8) == "FlashCom")) {
      if (application.newClients.length>0){
         //there are clients already connected here so reject connection
         this.rejectConnection(p_client);
         trace("(W) connection denied from Origin("+userName+") connecting to "+this.OriginName)
      }else{
         //Allow connection...// there are no clients so this must be the redundant Origin
         this.acceptConnection(p_client);
         p_client.userName = userName;
         //p_client.clientID=cliente;
         p_client.userType = userType;
      }
   }else if (p_client.agent.substr(0,8) != "FlashCom" && userType!="Origin"){
         this.acceptConnection(p_client);
            if ( !application.isStatePrxyd && application.newClients.length<2 ){
               for (var i=0, tot=application.clients.length;i<tot; ++i){
                  if (application.clients[i].userType=="Origin"){//kill connection if Origin connected
                     application.disconnect(application.clients[i]);
                     break;
                  }
               }
            }
         if (!this.O_ncc.isConnected)CreateOriginconnection(this.Oindex,this.OriginName,"Origin");
   }
}

Step 2. Send the data

Once the connection is made, you will need to pass all the current data from SharedObjects or vars and keep sending the changes (deltas) to the data flowing over to the secondary origin by means of a proxy function.

Below is a function that sends all data to the secondary origin when the connection is initially made (this happens only once per connection):

function initalizePrxy(){
   var myText = application.text_so.getProperty("txt");
   var MainArray = [application,[myText]];
   application.O_ncc.call("call2server", null, "pullOriginData", MainArray);
}

Below is a function that receives data from the primary origin (happens only once per connection):

function intiFromPrimary(Params,RSOs){
   application.text_so.setProperty(RSOs[0])
   for (var i in Params){
   application[i]= Params[i]
   }
}

Step 3. Create white lists for security

To manage and easily proxy functions with added security, create white lists of methods allowed for user types, including origin (as a user type). That way, every time a function call is made that requires proxying, it can simply be sent to the connected server:

application.onAppStart = function(info){
...
   this.AllFL = {/**/your1stfunction:true,your2ndfunction:true,your3rdfunction:true}
   ///////////////////////////////////////////////////////
   this.OriginFL={your1stfunction:true,your2ndfunction:true}
...
}

The following proxy function grabs any method being called by the client and references it against the white list, mentioned earlier, to correctly channel functions that require proxying and security. If the function is not in the white list for the user type, or is returned false when referenced, the method will not execute. If the type is not that of an origin and the application is being proxied, the method will be passed along to the secondary origin:

Client.prototype.call2server = function(p_method, p_paramArray,ClientInfo){
   var valid=false;
   trace(p_method+" and params are "+p_paramArray);
   switch (this.userType){
      case "Type1":
         valid= (application.AllFL[p_method] ?true:false);
      break;
      case "Origin":
         valid=(application.OriginFL[p_method]?true:false);
      break;
   }
   if (!valid){
      trace("(Wa) WARNING -Utype ("+this.userType+")->s call ["+p_method+"("+p_paramArray+")] NOT allowed. Ignoring call.");
   }else{
      if (valid && application.isStatePrxyd && this.userType!="Origin"){
         application.O_nc.call("c2s", null, p_method, p_paramArray,this);//push the same data over to redundant origin
      }
      return eval(p_method).apply((ClientInfo!=undefined?ClientInfo:this), p_paramArray);
   }
}

Step 4. Test the code

To test the server code locally, find the following line in main.asc:

application.O_ncc.connect("rtmp://"+application.AlternateO+"/"+application.name , Oname , Otype);

Modify it so that it points to a second instance of your application locally, such as this:

application.O_ncc.connect("rtmp://localhost/Oredundancy/second" , Oname , Otype);

Connect normally using your SWF compiled from main.as and text4failover.fla. After you have made the changes, recomplile your SWF—this time making sure that the instance name is changed in text4failover.fla from inst="Primary"; to inst="second".

Where to go from here

Throughout this article, many examples reference Windows Server but the logic and functionality should not be difficult to port over to a Linux server.

Again, this is just one possible solution. One alternative is to get script access to the load balancer (which I do not address in this article) to listen in on the admin in much the same way I have done in the article, and then to reference a failover rule in the load balancer.

You should realize that if, for any reason, you need to restart the FMS service on the primary origin, you will have to stop the scheduled task or Windows service from executing the VBS script on the secondary server, which will stop the VBS script from enabling the firewall.

The examples given here should enable you to start grasping the requirements needed and possibilities for creating a fully redundant FMS environment. With Flash Media Server 2, you can provide high levels of service with the guarantee of a fully redundant system.

For more information on Linux/Windows IP filtering, check out the following resources:

• How to create a Setup project for a Windows Service in Visual Basic .NET or in Visual Basic 2005 (Microsoft Knowledge Base )
• Using Windows Firewall (Microsoft)
• Windows Firewall (Wikipedia)
• Cron (Crontab) tutorial job (webmasters-central.com)
• Crontab (Wikipedia)
• Using iptables() (LinuxGuruz)
• Iptables Tutorial 1.2.2 (by Oskar Andreasson)
• Netfilter/iptables (Wikipedia)

About the author