Exploring Flash Player support for high-definition H.264 video and AAC audio

The next major step in the video revolution is here. Adobe Flash Player 9 Update 3 is taking a step into the high-definition (HD) video realm in a major way by adding MPEG-4 video to its already impressive ranks of video support. MPEG-4 utilizes crisp, powerful H.264 encoding and is an industry standard for video, which includes high-definition (HD) delivery. It is also the standard for HD content online and on devices such as your home television.

To pair with the excellent visual power of H.264 encoded video, Flash Player 9 Update also supports HE-AAC audio, which is the higher quality successor to MP3. Advanced Audio Coding (AAC) is a high-efficiency (HE) and high-fidelity (HiFi), low-bandwidth audio codec that can be used with or without video.

These codecs allow users to stream high-quality video at low bit rates. In addition, this update to Flash Player makes it possible to leverage tools that are standard across the industry, such as Adobe Premiere Pro, Adobe After Effects, Adobe Soundbooth, and Adobe Audition—and even such non-Adobe technology as Apple Final Cut Pro—to create and deliver compelling video content. Future updates are even being planned for Adobe Creative Suite 3 (CS3) to enhance end-to-end H.264 content development and distribution. The future is looking very bright indeed.

The new video and audio capabilities are available in Adobe Flash Player 9 Update 3 (version 9,0,115,0) and will be included in the upcoming release of Adobe AIR. Both platforms will have the ability to stream from the next generation of Flash Media Server, due to release early next year. The new Adobe Media Player, a powerful new AIR application, will also support H.264, HE-AAC, and encrypted video content using the new Flash Media Rights Management Server from Adobe.

So does this new Flash Player support for MPEG-4 and H.264 mean that it will replace the On2 VP6 codec? Absolutely not. The addition of H.264 gives developers greater choice to select the technology that best meets their needs. The current implementation of H.264 does have some limitations, such as lack of support for alpha channel and the inability to embed video into a SWF file. On2 VP6 is a solid, high-quality choice for Flash-based video projects. The On2 VP6 codec is also clear of any licensing issues that may arise with MPEG-LA. (Licensing information can be found on the MPEG LA and Via Licensing sites.) The On2 VP6 codec will remain a consistent and viable option for media delivery. The added support for H.264 simply means that there are now more options and wider spread compatibility for high quality and HD video.

This article takes a look at the new codecs and file types supported in Flash Player "Moviestar." In the article I'll show you how to import both H.264 video and AAC audio into Flash, examine how to use hardware acceleration with the new Flash Player Update, and provide instructions on how to import the metadata included with MPEG-4 files.

This section provides information on the H.264 video spec guidelines and AAC/AAC+ compression. You will learn what is technically supported in H.264, what standard HD video dimensions are, how H.264 is a container format—and what that means—some basic information on what AAC/AAC+ has to offer, as well as some basic encoding resources.

MPEG-4 H.264

MPEG-4 (Motion Pictures Expert Group) H.264 is an international standard for video compression recognized by the International Organization for Standardization (ISO). H.264 is also known as "MPEG-4 Part 10" or AVC (Advanced Video Coding).

What does that mean? First, it means that when you encode your video for H.264, you're not just encoding it for Flash Player; video can be played back using other software like Apple iTunes or your iPod, or on Sony PlayStation Portable (PSP). With the new hardware scaling and multithreading support of Flash Player, you can now play back video at any resolution and bit rate, including the ultimate high-definition 1080p—as long as the system supports it (see Figure 1).

MPEG-4 containers

The MPEG-4 standard also describes a container format, which means that one file can contain several different types of data, stored as tracks. The overall file synchronizes and interleaves the data. So the video or audio in an MPEG-4 container can also be accompanied by metadata, cover art, subtitles, and other textual or visual data that can potentially be extracted by Flash Player. The container can also include multiple video and audio tracks too, but Flash Player will only play back one of each right now and ignores the rest.

AAC/AAC+

Audio files encoded in an MPEG-4 container can now be used in Flash Player if they use the AAC (Advanced Audio Coding) codec. The AAC codec is a lossy compression scheme for audio that has been used since MPEG-2, but has been updated for MPEG-4. It should be familiar to most people, since all music sold from iTunes uses AAC. For you technical folks, it is much higher quality because it can support capture up to 96kHz and can support up to 48 channels and backwards prediction. AAC achieves higher audio quality than MP3 files and maintains similar or lower file sizes. AAC+ or HE-AAC (High Efficiency AAC) is the AAC codec with the addition of parametric stereo and spectral band replication that has been optimized to provide high-quality audio at a low bit rates for applications such as streaming audio. Now you can take advantage of this file format through Flash Player by bringing in MPEG-4 files with AAC encoded audio.

Video encoding

With the H.264 update to Flash Player, video encoding has now been standardized across the entire video ecosystem, including the Adobe Creative Suite 3 family. Tools such as Adobe Premiere and After Effects can now be used to deliver content without using additional plug-ins.

Part of adopting H.264 is leveraging the massive number of solutions on the market. The following is a list of some of the companies that provide industry standard tools for encoding video in the supported format for the new Flash Player 9 Update 3: 

• Anystream
• Digital Rapids
• Kulabyte
• On2 Technologies
• Rhozet
• Sorenson Media
• Telestream
• x264

Supported file formats

Flash Player 9 Update 3 plays files derived from the standard MPEG-4 container format that contain H.264 video and/or HE-AAC audio, such as F4V, MP4, M4A, MOV, MP4V, 3GP, and 3G2. One thing to note is that protected MP4 files, such as those downloaded from iTunes or digitally encrypted by FairPlay, are not supported. For a full list of codecs supported by Flash Player, read the TechNote about it.

To get more information about Flash Player 9 Update 3, visit the Flash Player product page.

To discuss Flash Player 9 Update 3 with the online developer community, visit the main Flash support forums.

Important considerations when streaming H.264 content

One important thing about playing an H.264 video file as progressive download is that the moov atom needs to be located at the beginning of the file, or else the entire file will have to be downloaded before it begins playing. The moov atom is a part of the file that holds index information for the whole file. Unfortunately, tools such as Adobe Premiere and After Effects place this information at the end of the file, but Adobe is working to fix this in a future update to the CS3 video production tools. This isn't an issue for streaming the H.264 video files, however, so Flash Media Server users can breathe easy.

Two open-source solutions to adjusting the moov atom to allow progressive streaming are:

• QTIndexSwapper (Adobe AIR app by Renaun Erickson)
• qt-faststart.c (command-line app by Mike Melanson)

This section gives a hands-on approach to integrating H.264 video and AAC/AAC+ audio using the power of ActionScript. For these examples you will use an AAC/AAC+ encoded M4A file. You will gain an understanding of how to play audio and video content using the power of the NetConnection, NetStream, and Video classes.

Playing H.264 content using NetStream

MPEG-4 content must be brought in through a NetStream object, but what it takes to play H.264 content in your Flash movie doesn't change much—although you'll need to watch out for a few more unsupported content messages. The new status messages are NetStream.Play.FileStructureInvalid and NetStream.Play.NoSupportedTrackFound. The former is sent when the MPEG-4 file structure is invalid, and the latter is sent when none of the audio or video tracks in the MPEG-4 file is supported.

You can also trick the video components in Flash into playing H.264 video files by changing their file extension to .flv. The video components currently look for the .flv extension for files they are playing, although Adobe is currently working on changing the specification and in a future update this won't be necessary.

Here is the more orthodox way to play a video using progressive download:

1. Create a new ActionScript 3.0 Flash file. Save the file and name it H264_Video_AS3.fla.
2. In the Timeline, rename Layer 1 as ~AS:Main (see Figure 2).

3. Select Frame 1 in the Timeline and open the Actions panel (on Windows: F9, on Mac: Option+F9).
4. Declare a video variable at the top of the code:

5. Create a new NetConnection object and pass in null as the only parameter for the connect method:

6. Create a new NetStream object, passing your NetConnection object as a parameter to the constructor:

7. In ActionScript 3.0 you'll need to specify a client object for the NetStream to handle stream events, otherwise you will receive a compile error. You don't actually have to create the handler methods, but you must have the client object. Let's set the client object equal to this so the Timeline can handle the NetStream's events. Add this code before you attach the NetStream video to the video object:

8. Now to create the event listeners. Create the handler function for the NetStatusEvent as shown below:

9. Inside the netStatusHandler function, create an if conditional statement. The conditional will check if the code property of the event object is equal to the string value of NetStream.Play.FileStructureInvalid:

10. Inside the if conditional, use the trace() method to output a string that states the file structure is invalid:

11. After the if condition, create an else condition. The else condition will check if the code value of the event object is equal to the string NetStream.Play.NoSupportedTrackFound:

12. Inside the else condition, use the trace() method to output a string that states the MP4 doesn't contain any supported tracks:

The completed netStatusHandler() function should look like this:

13. Now let's attach this event handler to the NetStream object:

14. Under the line where you added the event handler, let's create your video dynamically:

15. Use the attachNetStream method of the Video class rather than the old attachVideo method used in ActionScript 2.0:

16. Pass the URL or path of your H.264 video to the play method of the NetStream object:

Using ActionScript to play AAC files

Playing AAC (M4A) files is the same in both ActionScript 2.0 and ActionScript 3.0. Just use the NetStream object and play the file:

1. Create a new ActionScript 3.0 Flash file. Save the file and name it AAC_Audio_AS3.fla.
2. In the Timeline, rename Layer 1: ~AS:Main (see Figure 3).

3. Select the Frame 1 in the Timeline and open the Actions panel (on Windows: F9, on Mac: Option+F9).
4. Create a new NetConnection object named connect_nc, and then call the connect() method, passing in null as the only parameter:

5. Create a new NetStream object, passing your NetConnection object as a parameter to the constructor:

6. Pass the URL or path of your AAC (M4A) file to the play method of the NetStream object:

The completed code should look like this:

7. Test your movie in the browser. The audio should begin to play.

Using a Sound object to control an AAC file

The previous section reviewed how to use ActionScript to play AAC files. Controlling your M4A audio with a Sound object is a little trickier. The Sound object can't load in M4A files the way it can load MP3 files, so for this part you'll need to bring files in using the NetStream and then work with them from there:

1. Open the same Flash file that you worked with in the first audio example (AAC_Audio_AS3.fla) or open the AAC_Audio_AS3_Sound_complete.fla file in the sample files folder.
2. On the ~AS:Main layer in the Timeline, select Frame 1 and open the Actions panel (on Windows: F9, on Mac: Option+F9).
3. Declare the SoundTransform variable at the beginning of the code:

1. You can't directly access the properties of the NetStream's SoundTransform object, but you can reference the object itself. So after you start playing the video in the NetStream, set the mySound variable equal to the NetStream's soundTransform property, like this:

2. Set the volume for mySound, using a number between 0 and 1:

3. Overwrite the NetStream's soundTransform property with mySound so the changes to the volume will be applied:

The completed ActionScript should look like this:

4. Test your movie in the browser. The audio should begin to play, with the volume level at about half of what it was originally.

Using a SoundTransform object to control an H.264 file

With ActionScript 3.0, you cannot use the Sound object to control the video's audio, because the Sound object has been altered so you can no longer attach sounds. Instead, you'll need to use the SoundTransform object attached to the NetStream. Here are the steps:

1. Open the same file that you worked with in the video example (H264_Video_AS3.fla) or open the H264_Video_AS3_Audio.fla file in the sample files folder.
2. On the ~AS:Main layer in the Timeline, select Frame 1 and open the Actions panel (on Windows: F9, on Mac: Option+F9).
3. Declare the SoundTransform variable at the beginning of the code:

4. This is where things get a bit odd. You can't directly access the properties of the NetStream's SoundTransform object, but you can reference the object itself. So after you start playing the video in the NetStream, set your mySound variable equal to the NetStream's soundTransform property, like this:

5. Set the volume for mySound, using a number between 0 and 1:

6. Overwrite the NetStream's soundTransform property with mySound so the changes to the volume will be applied:

Using ActionScript 3.0 and Flex to play H.264 content

H.264 content is supported in Flex as well. Here are the steps:

1. Create a custom class that extends the UIComponent class. Call yours H264Video and store it in a subfolder called video.
2. Before the constructor, declare some variables, like this:

3. In the constructor, call both the super() and an init() methods:

4. Write the init method as shown below to create Video, NetConnection, and NetStream objects:

5. Under the _stream_ns = NetStream(_connection_nc); line, add in a line of code to set an object as the client for your NetStream. The handling isn't necessary, but you must construct the client object:

6. Also add in a line of code to handle status events from the NetStream:

7. In the init() method, attach the NetStream to the video, add the video to the display list, and start playing the video file if the _filePath exists. The completed method should look like this:

8. Create the function to handle status events. In particular, handle those two new status events that you can track in H.264 video:

9. Create getter/setter methods for the _filePath property. On setting the filePath, call the play() method on the NetStream instance and pass it the filePath's value. Also add a [Bindable] metadata tag above the getter, like this:

10. Save your file.

You can then bring your video into the Flex application either solely by ActionScript or by using MXML. In your MXML file, you can create a tag on the Stage for your H264Video, and then specify an id, x, y values, etc. If you are using the Adobe AutoComplete Input Flex component to get code hints, Flex will create the namespace for you to use this tag. Also, set the filePath property on the MXML tag, pointing to your video file:

In the past, when you worked with Flash-based video, you could inject additional data into the video file for use during playback. The onMetaData event was commonly used to retrieve this data during runtime. The data would contain simple pertinent information such as track length and video dimensions. That same type of information is also available with the new supported video formats.

Since MPEG is a container format that can contain many different items of information grouped together into one file, it is useful to access different bits of information inside that container. Using the onMetaData event of the NetStream, you can extract a variety of information such as dimensions, length, codec, seek points, and potentially cover art, subtitles, audio book chapters, and other text and images from both H.264 and M4A files.

It's important to note that if the seek points aren't included in the metadata, you won't be able to seek in the file at all. There are also new onImageData and onTextData events that respond to this data for both audio and video files. The onImageData returns GIF, PNG, or JPEG data as a ByteArray in ActionScript 3.0 only. The onTextData can return text data such as subtitles. Cover art may also be encoded in your audio or video files; they are most often accessed through the onMetaData event. I will explore this in the following walkthrough.

Here are the steps for extracting metadata from video files:

1. Open the same file that you worked with in the video example (H264_Video_AS3.fla) or open the H264_Video_AS3_MetaData.fla file in the sample files folder.
2. Create a new layer in the Timeline under the ~AS:Main layer and name it output (see Figure 4).

3. While Frame 1 of the output layer is selected, create a dynamic text field on the Stage.
4. With the text field selected, open the Property inspector (on Windows: Control+F3, on Mac: Command+F3).
5. Set the x property of the text field to 5.
6. Set the y property of the text field to 5.
7. Set the width property of the text field to 175.
8. Set the height property of the text field to 300.
9. Give the text field the instance name of output_txt.
10. On the layer ~AS:Main select Frame 1 and open the Actions panel (on Windows: F9, on Mac: Option+F9).
11. In ActionScript 3.0, you have to handle the onMetaData event through the client object of the NetStream. So first write the function that handles the metadata and have it receive the event object as its only parameter. Write this code below the netStatusHandler function:

12. Inside the onMetaData event handler, create a for loop that will loop through and display the properties of the p_info object to the dynamic text field. The code should look similar to this:

13. Use the metadata to set the size of the video object. After the for loop in the onMetaData event handler, set the width property of the video object to the value of p_info.width, like this:

14. Set the height property of the video object to the value of p_info.height:

Note: The video file in the sample files is 1280 x 720, so you'll want to size your Flash project to accommodate that size. Use 1465 x 720 as the width and height dimensions to size your sample files.

The completed onMetaData handler should now look like this:

15. Test your movie in the browser. You may not see your output text field because the video is getting loaded on top of it. To solve that problem, you'll move the video over. Right before the addChild(video); line, set the x value of the video to 185 and the y value of the video to 5:

16. Save the file and test your movie in the browser again. The metadata should be displayed in the visible output text field and the Video object should resize to match the size of the source video.

Here are the steps for extracting metadata from audio files:

1. Open the same audio file you've been working with (AAC_Audio_AS3.fla) or open the AAC_Audio_AS3_MetaData.fla file from the sample files folder.
2. Create a new layer in the Timeline under the existing ~AS:Main layer and name it output (see Figure 5).

3. While Frame 1 of the output layer is selected, create two dynamic text fields on the Stage and give them the instance names length_txt and copy_txt.
4. Select the text field with the instance name length_txt and set its x position to 10 and its y position to 5. Set the width of the text field to 240 and its height to 21. Also click the Show Border Around Text button in the Property inspector to add a border around the text field. Your Property inspector should look similar to Figure 6.

5. Select the text field with the instance name copy_txt TextField and set its x position to 10 and its y position to 31. Also set its width to 240 and its height to 21 and click the Show Border Around Text button in the Property inspector. After making these changes, the Stage should now roughly appear like the example in Figure 7.

6. Select Frame 1 of the ~AS:Main layer in the Timeline and open the Actions panel (on Windows: F9, on Mac: Option+F9).
7. After declaring the NetStream object, set the client property of the NetStream object equal to the reference value this, so that the root Timeline will handle events from the NetStream. In ActionScript 3.0, you have to handle the onMetaData event through the client object of the NetStream:

8. Create an event handler for the onMetaData event. The function should receive one parameter named p_info that is typed as an Object:

9. Inside the onMetaData event handler, set the text property of the length_txt text field to display the length of the audio source file. The length of the audio is stored as seconds in the duration property of the event object, so you can multiply that number by 1000 to get the duration of the media in milliseconds:

10. Set the text of the copy_txt TextField to display the codec used to encode the audio source file. This codec information is stored in the audiocodecid property of the p_info object received by the onMetaData event handler:

The completed code should look like this:

11. Test the movie in the browser and you should see some basic information extracted from the file's metadata. That's cool, but you can make things more interesting.
12. Close the browser window and return to Flash. Open the Action panel.
13. Write a function that will display a cover image. The cover image data is received in the onMetaData event handler as an array of ByteArrays. Under the onMetaData event handler, create a function named handleCoverImage and have it accept an Array named p_data as its only parameter:

Note: The cover data you'll be bringing in is 320 x 240, so you should resize your Flash movie to match those dimensions.

14. Inside the handleCoverImage() function, create a new Loader variable and set it equal to a new Loader:

15. After creating the new Loader, call its loadBytes method to load in the data at the first index of the array, casting that value as a ByteArray. You can also retrieve this information for video, as well as audio, if it has the cover art metadata attached:

16. In the last line of the function, you'll need to add the Loader to the display list at index 0 so that the image you load will be visible. Do this by calling the addChildAt() method and passing it loader as the first parameter and 0 as the second parameter:

The completed handleCoverImage() function should look like this:

17. Above the existing code in the onMetaData event handler, create an if conditional statement. The if condition should check if the cover data, stored as covr, in the tags object exists:

18. Move the code where you set the text property for the length_txt TextField above the if condition, like this:

19. Wrap the code where you set the text properties for copy_txt in an else statement:

20. Inside the if condition, set the htmlText property of the length_txt TextField to display the copyright information for the audio as a hyperlink to the URL associated with the media. This information is stored as the ©cpy and ©url attributes of your sample media files:

21. Declare a covr variable and set it equal to the covr metadata and cast this value as an Array. Theoretically, you can have multiple images associated with the media as cover data, so the covr property should be treated as an Array:

22. Pass the covr variable as a parameter to the handleCoverImage function you wrote earlier:

The completed onMetaData handler should look like this:

23. Test the movie in the browser. You should now see the cover art included with the audio duration and the copyright information about the media (see Figure 8).

Another very exciting new feature in Flash Player 9 Update 3 is hardware scaling support for Flash Player—which lends itself perfectly to enhancing the video playback experience of HD video in full screen—especially when you consider the size of HD 1080p video (1920 x 1080). The new hardware acceleration was not built solely for the new H.264 video capabilities. It also helps with larger On2 VP6 video files and the display of SWF content in general.

Right-click (Windows) or Control-click (Mac) inside the Flash Player content in your browser window to display the Flash Player context menu and choose the Settings option to access the Adobe Flash Player Settings panel. The first tab shown is the Display tab (see Figure 9).

The only item on the Display tab is a checkbox asking whether or not you'd like to enable hardware acceleration. If the box isn't already checked, check it to enable hardware acceleration—which will be in use if the content triggers it using the new ActionScript API. Unchecking or disabling this option forces software acceleration for the case when the scaling API is used. If hardware acceleration is enabled and the API is used, then Flash Player uses the hardware. The key area where you'll see hardware acceleration in effect is in full-screen mode, whether you're scaling a portion of the movie to full screen or the entire movie.

Now let's put the new hardware acceleration to use by scaling just a portion of your SWF to full screen. For overall advice on the particulars of making your movie go full screen, see Tracy Stampfli's excellent article, Exploring full-screen mode in Flash Player 9. If you would like to see an example of the visual effectiveness of full-screen video, check out this Adobe Labs demo.

Here are the steps to scale a portion of the movie to full-screen video:

1. Open the same file that you worked with in the video metadata example (H264_Video_AS3.fla) or open the H264_Video_AS3_FullScreen.fla file in the sample files folder.
2. Select Frame 1 on the layer ~AS:Main and open the Actions panel (on Windows: F9, on Mac: Option+F9).
3. Import the Rectangle and Stage classes at the beginning of your ActionScript:

4. Any attempts to make the Stage go full screen have to be in response to a user interaction, such as an onRelease or onKeyDown event. Right now, however, you can write the function to make the movie go full screen. Your goFullScreen() function will take an event object as its sole parameter:

5. Inside the function, create a Rectangle that has the same dimensions as the video and place it at the same x and y coordinates:

6. Set the fullScreenSourceRect property of the Stage to the Rectangle you just made. Because that property is not yet officially in the class, you must use square bracket notation to do this:

7. After that line of code, create an if conditional statement that checks to see if the display state of the Stage is normal or not, by checking it against the Stage class's NORMAL property, like this:

8. Inside the conditional statement, change the display state of the Stage to full screen using the FULL_SCREEN property:

9. Extend the if conditional statement by adding an else statement that sets the display state of the Stage to normal again:

The completed goFullScreen() function looks like this:

10. Return to the Timeline and create a new layer above the output layer. Name the new layer control (see Figure 10).

11. While Frame 1 of the control layer is selected, draw a roughly 30 x 30 rectangle on the Stage. Select the rectangle and invoke the Convert to Symbol dialog box (press F8). Select the type of movie clip to have the rectangle act as a full-screen button. Name the new symbol Button click OK (see Figure 11).

12. Using the Property inspector, position the button with an x value of 40 and a y value of 320.
13. Give the button the instance name of fullScreen_mc.
14. Select Frame 1 of the layer ~AS:Main and open the Actions panel (on Windows: F9, on Mac: Option+F9).
15. The fullScreen_mc button will have to use the addEventListener method to call the goFullScreen() function. Add this line after the goFullScreen() function:

16. Select File > Publish Settings to access the Publish Settings dialog box. Select the HTML tab. In the Template drop-down menu, select Flash Only – Allow Full Screen (see Figure 12) and click OK.

17. Test the movie in the browser. Click the fullScreen_mc button. You should see your video go to full-screen display mode.

Where to go from here

To learn more about working with video in Flash Player 9 Update 3, go visit Tinic Uro's blog. To get more details about the high-definition standards and codecs, see the Wikipedia articles on H.264 and AAC.

The addition of H.264 and AAC support in Flash Player 9 Update 3 allows you to easily use high-definition, industry standard video and audio. With this update, Flash Player raises the bar further for video on the web and offers display possibilities across a wide range of devices. Pairing the new video capabilities of Flash Player and future updates to the Adobe Creative Suite workflow—in addition to new products like Adobe AIR and Adobe Media Player—it is safe to say that very exciting times are ahead.