Performance-tuning Flash Media Server 2 for live webcasts using Linux
Created
26 February 2007
Requirements
Prerequisite knowledge
This article assumes you are familiar with using Flash Media Server 2, Apache Web Server 2.0, the Linux file system, and editing files on Linux.
User level
Advanced
Required products
- Flash Media Server 2 (Download trial)
Additional Requirements
Apache Web Server 2.0
SUSE Linux Enterprise Server
Note: In this article we specifically use and discuss SUSE Linux Enterprise Server. Other Linux distributions should work in similar fashion. However, be aware that other distributions may store their files in different locations that are not mentioned in this article.
Novell, Inc. has been using Macromedia Flash Media Server since its initial release and now relies on it for the delivery of high-profile webcasts such as earnings calls, employee communications, and training. Flash Media Server has been Novell's preferred choice for streaming media broadcasts due to its client-side interoperability and its platform ubiquity.
While Flash Media Server can be used right out of the box in its default configuration, Novell wanted to squeeze better performance out of the server for specific use cases so that each server could handle more load. This article identifies the challenges we faced and the changes we made that have helped us gain optimal performance.
Encountering roadblocks
Using the default configurations for Flash Media Server, Apache, and Linux OS, Novell found that the number of users that a single server could handle was less then they desired. As we approached 500 user connections, Novell's live webcasts experienced severe audio skips and pauses, as well as frequent user disconnects. As you can imagine, it didn't take long before users and stakeholders started asking questions about how they could handle more load on a server.
With this scenario in mind, Novell worked with Huddle, its web conferencing solutions provider, to diagnose the problems and come up with a solution. After spending a week in the "Superlab," Novell's state-of-the-art, onsite testing facility, we were able to make Novell's servers running Flash Media Server perform better than ever. Since our efforts resolved the issues and worked out well, we felt it would be helpful to share our experience with the community.
Results and findings
By applying a few (relatively) simple performance tuning parameters to Flash Media Server, Apache Web Server, and SUSE Linux Enterprise Server Novell's webcasts went from supporting 500 concurrent users per server to well over 1500, while increasing broadcast quality at the same time. The tuning guidelines discussed in this article will help administrators get the most out of their servers and help improve broadcast quality under heavy loads. The strategies discussed here should apply equally to other GNU and BSD Linux systems as well.
Hardware used in testing
Our particular server situation at the time of testing included the following:
- Make and model: HP ProLiant DL380 G4 Server
- CPUs: 2 x Intel Xeon 3.4 GHz/800MHz
- Server memory: 6 GB
- Operating system: SUSE Linux Enterprise Server 9
- Disk subsystem: 6 x 146 GB drives @ 10K RPM in RAID 5 configuration
- Network card: NC7782 Dual Port Gigabit NIC (only single port used)
Note: The same hardware was used to test results before and after implementing performance tuning.
Performance before and after
The following tables list the performance statistics measured before and after the tuning parameters were applied. Table 1 shows performance test results prior to server tuning. Note the CPU-bottlenecking at the 500-user level.
| Users | Audio sampling | CPU utilization | Bandwidth |
|---|---|---|---|
| 500 | 11 kHz | 95–100% | 13 Mbps |
| 1000 | N/A | N/A | N/A |
| 1500 | N/A | N/A | N/A |
| 2000 | N/A | N/A | N/A |
Table 2 shows performance test results after server tuning. Note a dramatic increase in CPU performance at the 500-user level and also the statistics for user connections exceeding 500.
| Users | Audio sampling | CPU utilization | Bandwidth |
|---|---|---|---|
| 500 | 11 kHz | 40% | 13 Mbps |
| 1000 | 11 kHz | 75% | 26 Mbps |
| 1500 | 11 kHz | 90% | 40 Mbps |
| 2000 | 11 kHz | 100% | 55 Mbps |
Tuning Flash Media Server
The first modification we made to Flash Media Server involved changing the audio resampling rate from four audio samples combined into one message to 16 audio samples. This increases latency slightly but also increases the number of clients that can reliably connect.
Make this modification in the Application.xml configuration file, or for the specific virtual host that you plan on using, as follows:
<!-- Combine N audio samples into one message. -->
<MaxSamples>16</MaxSamples>
The second tuning change we made to Flash Media Server was setting the publishing rate to 11 kHz. After some experimentation, we found that the published audio rate is a huge factor in performance. Our tests indicate that CPU cycles are consumed much faster when the publishing streams are set at 8 kHz audio, which limits the number of users that can connect reliably. This occurs because Flash Media Server is unable to combine 8 kHz audio samples. The other audio rate settings (5, 11, and 22) can be combined, so using one of those settings helps eliminate excessive CPU consumption.
Other than these two changes, we left the Flash Media Server configuration files exactly as they were shipped.
Tuning file descriptors and number of processes on Linux
After making the changes described previously, we adjusted the maximum number of file descriptors and processes allocated to the Flash Media Server and Apache Web Server processes. Linux processes use file descriptors to open files and network sockets. Ensuring that Flash Media Server and Apache do not run out of these resources enables you to scale your Flash Media Server implementation.
Although determining values for these parameters is beyond the scope of this article, the values we provide should be sufficient for most Flash Media Server implementations. If you are interested in learning more, we encourage you to check out these resources for additional information on Linux performance tuning:
- Tuning SUSE Linux Enterprise Server on IBM @server xSeries Servers (IBM Redbooks)
- How Many Open Files? (NetAdminTools.com forum)
- Linux info pages for
ulimit
Add the following four entries to the bottom of /etc/security/limits.conf file:
* soft nofile 8192
* hard nofile 8192
* soft nproc 8192
* hard nproc 8192
The first two lines (containing nofile) set the maximum number of files, including sockets, that a process is allowed to open. If a process attempts to open more files than specified, the attempt will fail.
The second two lines (containing nproc) set the maximum number of server processes that can be created with the same user ID. If the limit is reached for a particular user, no more processes will be created.
More information on setting Linux system limits can be found in the comments section of the limits.conf file. You must reboot the server after making the changes above to apply these parameters.
Tuning Apache Web Server 2.0
Novell uses the Apache Web Server included with SUSE Linux Enterprise Server 9 for handling content delivery and user authentication to the web streams player. You can tune Apache for high-traffic environments to handle many simultaneous user connections efficiently. A typical Apache connection profile during a webcast may look similar to Figure 1.
Figure 1. Representative look at the number of connections made prior and immediately after a webcast begins. (Graph generated at National Center for Education Statistics' Create a Graph.)
It is important to configure Apache to handle the peak user load as efficiently as possible—to ensure that users are able to connect to your webcasts. If you run Apache on a different server than Flash Media Server, you'll also want to apply the "file descriptors and number of processes" change as described above.
To tune Apache Web Server 2.0 when running the "prefork"—this is the default—MPM (multiprocessing module) edit the /etc/apache2/server-tuning.conf file as follows:
<IfModule prefork.c>
# number of server processes to start
# StartServers 5
StartServers 30
# minimum number of server processes which are kept spare
# MinSpareServers 5
MinSpareServers 10
# maximum number of server processes which are kept spare
# MaxSpareServers 10
MaxSpareServers 30
# highest possible MaxClients setting for the lifetime of the Apache process.
# ServerLimit 150
ServerLimit 300
# maximum number of server processes allowed to start
# MaxClients 150
MaxClients 300
# maximum number of requests a server process serves
MaxRequestsPerChild 1000
</IfModule>
These values should be sufficient for all but the most extreme implementations. If you find that Apache is running out of connections, or cannot keep up with connection loads, there are many online resources and websites filled with tips and tricks for performance tuning the Apache Web Server. Begin your research by using the search term "performance tuning apache web server".
Tuning TCP/IP kernel parameters
The tuning changes we made up to this point—adjusting the maximum number of file descriptors and open files, as well as updating configuration settings for Flash Media Server and Apache Web Server as described—allowed Flash Media Server to sustain over 1300 concurrent users with no audio degradation in our testing. This round of tests indicated that Novell's webcasts were improved by allowing almost three times the 500 users we could initially support.
However, there was still one more group of changes to the TCP/IP kernel settings in Linux that were required to complete our server tuning project.
A brief side step needs to be taken at this point to warn you about changing Linux kernel parameters. Use extreme caution when tuning TCP/IP parameters, as this is an advanced server administration technique. Any changes made to kernel tuning parameters should be tested well before using the configuration in a production environment. We recommend that users back up their current configuration by issuing the following command sequence before applying any changes (remember to verify the content in the file before proceeding):
sysctl -A > /tmp/sysctl.bak
Now, with our side step completed—assuming you've heeded our warning and have in your possession a backup file with all of your current settings—we can confidently proceed by adding the following lines anywhere in the /etc/sysctl.conf file:
# Disable response to broadcasts.
# You don't want yourself becoming a Smurf amplifier.
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Filter packets not meant for this network.
net.ipv4.conf.eth0.rp_filter=1
net.ipv4.conf.lo.rp_filter=1
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
net.ipv4.tcp_rmem = 4096 10000000 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# Disabling the following parameters will prevent a hacker
# from using a spoofing attack against the IP address of the server.
net.ipv4.conf.eth0.accept_source_route=0
net.ipv4.conf.lo.accept_source_route=0
net.ipv4.conf.default.accept_source_route=0
net.ipv4.conf.all.accept_source_route=0
# These commands configure the server to ignore redirects from
# machines that are listed as gateways. Redirects can be used to
# perform attacks, so we only want to allow them from trusted sources.
net.ipv4.conf.eth0.secure_redirects=1
net.ipv4.conf.lo.secure_redirects=1
net.ipv4.conf.default.secure_redirects=1
net.ipv4.conf.all.secure_redirects=1
# Don't allow ICMP redirects
net.ipv4.conf.eth0.accept_redirects=0
net.ipv4.conf.lo.accept_redirects=0
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.all.accept_redirects=0
# If the server does not act as a router, it does not need to
# send redirects.
net.ipv4.conf.eth0.send_redirects=0
net.ipv4.conf.lo.send_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.all.send_redirects=0
# For servers that receive many connections at the same time, the
# TIME-WAIT sockets for new connections can be reused. This is useful
# in Web servers, seems to be good for Flash servers as well.
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_fin_timeout=30
# Move keepalive from 2hrs to 30 min. You may want to tune this up or down depending on your implementation
net.ipv4.tcp_keepalive_time=1800
# Help protect from denial-of-service (syn-flood) attack.
net.ipv4.tcp_max_syn_backlog=4096
You should add these changes to both Apache Web Server and Flash Media Server—if they are running on separate servers. To apply these changes, no restart is necessary. Simply type sysctl -p at the command prompt. These changes will now be applied automatically every time the server is restarted.
Where to go from here
The tuning changes discussed in this article are specific to our Apache Web Server and Flash Media Server implementation and should not be used in your production environment without proper testing. Individual implementations could benefit by further adjusting tuning parameters based on your actual performance data. We recommend starting with the information provided here to obtain a performance data baseline. Once baseline performance is obtained, you can judge each change to the server and the corresponding results with respect to the baseline measurements. By systematically testing and subsequently evaluating the data, you can achieve the best performance results.
Where should you begin? We believe that adjusting the maximum number of file descriptors and open files—as well as updating Flash Media Server and Apache Web Server configuration files—will help improve performance for the majority of high-traffic implementations. If you would like to pursue further performance gains or learn more about how to tune your individual environments, we recommend starting with the Tuning TCP/IP kernel parameters section of the article and reviewing the great reference article we've mentioned earlier in this article, IBM's Redpaper series: Tuning SUSE Linux Enterprise Server on IBM @server xSeries Servers.
More Like This
- Calculating bandwidth needs for Flash Media Server 3
- Learning Flash Media Server excerpt: Server-side streams
- H.264 for the rest of us
- Beginning Flash Media Server 4.5 – Part 1: Installation
- Using plug-ins to configure and optimize Flash Media Server 4.5
- New buffering strategies in Flash Player 9 and Flash Media Server 3
- Implementing a dual-threshold buffering strategy in Flash Media Server
- Eliminating the single point of failure with origin redundancy in Flash Media Server
Tutorials & Samples |
