Eliminating the single point of failure with origin redundancy in Flash Media Server

Additional Requirements

Any enterprise-level load balancer

Any enterprise-level, high-speed NAS network attached storage

Today's web users are more and more demanding when it comes to viewing their favorite content, media, and applications online. Their needs for an always-on experience are pushing providers to deliver content that meets (and exceeds) their increasingly high expectations. That's why if you don't have a failover strategy, a glitch in your Flash Media Server origin could cause you serious problems.

Macromedia Flash Media Server 2 from Adobe has made great strides with edge/origin configuration, making it easier and faster for you to deploy applications that handle incredible loads. If you run it out of the box, you can easily set up an edge cluster to handle your incoming load while running your application/content from a single origin (see Figure 1). However, the origin could create havok if it should ever fail without a well-architected failover solution in place.

Note: Edges can cache content, so it may not actually be relying on the origin when serving out cacheable files.

The goal of this article is to provide some insight into a possible solution to help you better design an overall scale-up and scale-out architecture that accounts for performance, backups, and redundancy to virtually guarantee high reliability. Think of full redundancy as no discernable interruption of service, employing a duplicate to prevent failure of an entire system. If mission-critical problems caused by hardware or software failures do occur, this failover solution (coupled with other methods) can enable a production environment never to go down, at least from a user's perspective (see Figure 2).

I assume that you have limited access to the NAS (network attached storage), so you will have to make a request to those with full access to create an account with read/write/create privileges. Keep in mind that this account must also be an administrator on the origin server.

You will also need to create a config file. into which you'll write the two origin names (or IPs) in order—primary\secondary, as follows:

In the example cited in this article, the file is named RedundancyMainConfig.txt but it could be named anything. Just update VBS file to reflect the change.

Configuring the load balancer and edge servers

The load balancer should have been configured to add both primary and secondary origins to the pool—which should already be configured. The pool, also known as the server farm, is a reference to a load balancer configuration that retains a number of servers, across which it load-balances traffic. In round-robin mode, it should be monitoring over TCP port 1935. Figure 3 shows BIG-IP's admin console.

Note: All administration consoles are different. Consult your operations team before making any of these changes if you have the ability to make them.

The configuration steps shown in Figure 3 describe the following:

1. Click Monitors in the left column.
2. Click the Basic Associations tab.
3. After checking the respective column to add, click Apply.
4. Note that the monitor now works using TCP/IP.

The virtual IP (VIP) address of the load balancer for this pool should replace the current IP address in all the edge servers' Vhost.xml files located in <your installation directory>\conf\_defaultRoot_\_defaultVHost_\Vhost.xml, as follows:

Here 10.269.20.06 should be changed to point to the VIP of the load balancer.

Now that you have your load balancer set up, you need to add a rule or policy that can be enabled or disabled. In Windows we will be using Windows Firewall and deny any machine to listen on TCP port 1935 to this box unless its IP address is that of the alternate origin. Then we will create a scheduled task to execute the script below.

Note: You can name this code however you please. In the attached ZIP file I have named it OriginFailover.vbs, and the script must reside on both the primary and secondary origin. You will need to point your scheduled task to run it.

This script basically grabs the information from RedundancyMainConfig.txt on the NAS and uses that information to listen in on the admin's ping response. If it finds a string other than "successful," the script turns on the primary origin's firewall, which blocks the load balancer's monitor. On the secondary (standby) origin, it turns off Windows Firewall, making it the only available choice for the load balancer.

If the script is runnning on the secondary origin and the primary goes down, the VBS script file shown above writes to the NAS. This is a precautionary measure so that when the primary is restarted, it doesn't automatically switch the clients back. This allows you to analyze what went wrong before resetting to the primary by removing \secondaryUP from RedundancyMainConfig.txt.

Note: The scheduled task allows only a minimum of one minute. To get to within seconds, you will need to create a Windows service, which can either call the VBS script or embody its functionality.

Most of the logic in the above script is straightforward and can easily be ported or refactored for your needs. The main function you may need to modify for your language or needs is the following:

In Linux, assuming your kernel has iptables support built in, you can execute a command with similar results:

This code is a simple example of how to add a rule to iptables. What is not shown is how to deny access to all possible IPs and then add rules—much like the one shown above—to allow a specific server, load balancer, or IP. The main idea here is to exercise control over when you allow the load balancer VIP access on TCP port 1935 to the primary or secondary origin based on whether the primary is up or down.

If the primary origin is down, you will not want the loadbalancer to see it, so you must block it. The secondary origin will have to allow the loadbalancer because it was previously blocking it to force all traffic to the primary. You can do this by clearing iptables on the secondary origin and allowing only the loadbalancer VIP and primary origin access to the secondary origin. Remember at all times that the primary and secondary origins should allow each other access to keep state of any application.

Tip: Be careful when setting up your cron job or scheduled task (do not use a loop), because if you create millions of calls to the admin per second, the admin could fail, causing the origin to fail over. (Remember that both the primary and secondary origins will be listening in on the primary's admin ping result.)

Find the application.xml file located in <your installation directory>\conf\_defaultRoot_\_defaultVHost_\application.xml.

Create a virtual directory that maps to a given directory in your NAS. In the example, I use myFMSfiles like the following:

Look for the <FileObject> tag. Inside that tag, you will create the <VirtualDirectory> tag, which may show an example commented out.

In the StreamManager within application.xml, find the <StorageDir> tag in order to define the root where any recorded streams should be stored: <StorageDir>\\<NASdir>\FMSfiles\</StorageDir>.

Tip: If you are going through development and staging environments before you hit production, it is a good idea to use an <ApplicationObject> tag (which should exist within the <JSEngine> tag) in defining your environments, like so:

This will become more clear when we move on to examine the server-side code.

If you are on a Windows server, go to the Services panel, which you can access by selecting Start > Run and then type services.msc in the Run dialog box (see Figure 4).

In the Services panel, find Flash Media Administration Server and Flash Media Server. First stop both these services. For each service, right-click the service, select Properties, select the Log On tab, and enter the username and password that you set up in the NAS configuration setup (see Figure 5).

The intention in the following steps is to demonstrate a solution—to push state from primary to secondary—that accounts for data which may be in a sharedObject (RSO) or variable. You need to make sure that even if Windows Firewall or iptables rules are running, the origins' IPs should always be open to each other (primary/secondary) as described earlier in section, "IP Filtering and Failover on Origin Servers." The solution also requires you to organize all calls made to the server so that it can easily and securely be called on the secondary origin.

Step 1. Create a connection

The first step is to create a connection from the primary origin, receiving the clients to the secondary (semi-dormant) origin. To do this correctly, you will have to identify whether any users are currently connected to the primary origin. If there are, you will then connect to the secondary (semi-dormant) origin. The example shown below is included in the main.asc file.

Step 2. Send the data

Once the connection is made, you will need to pass all the current data from SharedObjects or vars and keep sending the changes (deltas) to the data flowing over to the secondary origin by means of a proxy function.

Below is a function that sends all data to the secondary origin when the connection is initially made (this happens only once per connection):

Below is a function that receives data from the primary origin (happens only once per connection):

Step 3. Create white lists for security

To manage and easily proxy functions with added security, create white lists of methods allowed for user types, including origin (as a user type). That way, every time a function call is made that requires proxying, it can simply be sent to the connected server:

The following proxy function grabs any method being called by the client and references it against the white list, mentioned earlier, to correctly channel functions that require proxying and security. If the function is not in the white list for the user type, or is returned false when referenced, the method will not execute. If the type is not that of an origin and the application is being proxied, the method will be passed along to the secondary origin:

Step 4. Test the code

To test the server code locally, find the following line in main.asc:

Modify it so that it points to a second instance of your application locally, such as this:

Connect normally using your SWF compiled from main.as and text4failover.fla. After you have made the changes, recomplile your SWF—this time making sure that the instance name is changed in text4failover.fla from inst="Primary"; to inst="second".

Where to go from here

Throughout this article, many examples reference Windows Server but the logic and functionality should not be difficult to port over to a Linux server.

Again, this is just one possible solution. One alternative is to get script access to the load balancer (which I do not address in this article) to listen in on the admin in much the same way I have done in the article, and then to reference a failover rule in the load balancer.

You should realize that if, for any reason, you need to restart the FMS service on the primary origin, you will have to stop the scheduled task or Windows service from executing the VBS script on the secondary server, which will stop the VBS script from enabling the firewall.

The examples given here should enable you to start grasping the requirements needed and possibilities for creating a fully redundant FMS environment. With Flash Media Server 2, you can provide high levels of service with the guarantee of a fully redundant system.

For more information on Linux/Windows IP filtering, check out the following resources:

• How to create a Setup project for a Windows Service in Visual Basic .NET or in Visual Basic 2005 (Microsoft Knowledge Base )
• Using Windows Firewall (Microsoft)
• Windows Firewall (Wikipedia)
• Cron (Crontab) tutorial job (webmasters-central.com)
• Crontab (Wikipedia)
• Using iptables() (LinuxGuruz)
• Iptables Tutorial 1.2.2 (by Oskar Andreasson)
• Netfilter/iptables (Wikipedia)