Requirements

Prerequisite knowledge

Familiarity with Stage3D and VideoTexture is required.

User level

{adcConfig::_[skillLevel]}

Overview

Spherical videos have a specific metadata attached to them. To provide a 360 degree experience, the associated metadata is manipulated at runtime. With the introduction of Spherical video support in Flash Player 24 and AIR 24, it is easy to retrieve the attached metadata at runtime. In the earlier versions of Flash Player and AIR, the metadata information attached to the spherical video had to be hard-coded.

The support for spherical video is available on desktop and Android platforms.

Global metadata sample

Here is a sample metadata information that you can find attached to video streams.

<rdf:SphericalVideo xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:GSpherical="http://ns.google.com/videos/1.0/spherical/"> <GSpherical:Spherical>true</GSpherical:Spherical> <GSpherical:Stitched>true</GSpherical:Stitched> <GSpherical:StitchingSoftware>OpenCV for Windows v2.4.9</GSpherical:StitchingSoftware> <GSpherical:ProjectionType>equirectangular</GSpherical:ProjectionType> <GSpherical:SourceCount>6</GSpherical:SourceCount> <GSpherical:InitialViewHeadingDegrees>90</GSpherical:InitialViewHeadingDegrees> <GSpherical:InitialViewPitchDegrees>0</GSpherical:InitialViewPitchDegrees> <GSpherical:InitialViewRollDegrees>0</GSpherical:InitialViewRollDegrees> <GSpherical:Timestamp>1400454971</GSpherical:Timestamp> <GSpherical:CroppedAreaImageWidthPixels>1920</GSpherical:CroppedAreaImageWidthPixels> <GSpherical:CroppedAreaImageHeightPixels>1080</GSpherical:CroppedAreaImageHeightPixels> <GSpherical:FullPanoWidthPixels>1900</GSpherical:FullPanoWidthPixels> <GSpherical:FullPanoHeightPixels>960</GSpherical:FullPanoHeightPixels> <GSpherical:CroppedAreaLeftPixels>15</GSpherical:CroppedAreaLeftPixels> <GSpherical:CroppedAreaTopPixels>60</GSpherical:CroppedAreaTopPixels> </rdf:SphericalVideo>

Formats supported

Flash Player 24 and AIR 24 support spherical videos that conform to the ISO/IEC 14496-12:2008 specifications and have the metadata attached to them.

To know more about the supported format, see Spherical Video RFC.

Projection types

Projection type of a spherical video specifies the way the video is stitched. Spherical videos can be classified into two types on the basis of the projection type — Equirectangular and Cubemap.

Equirectangluar

Equirectangular projection is commonly used by many streaming sites such as Youtube. Vertices in Equirectangular projection build a mesh shaped sphere having 'n' columns and 'm' rows. The default columns and rows can change depending on the use cases, as columns and rows used for generating vertex affect the overall performance and output quality of the video.

Cubemap

Cubemap projection is used in specific cases because cubemap usually shows better efficiency in performance and encoding. For those who want to use cubemap projection, it can be easily archived by skybox scheme. In Flash Player, VideoTexture object used for SphericalVideo is a simple 2D texture. It is rendered as a skybox. But it is possible to exhibit artifacts on edge of skybox depending on the wrapping mode of VideoTexture.

To remove the artifacts, wrapping mode for edge needs to be set as clamp and antialiasing factor set to 0, when calling configureBackBuffer.

Sample spherical video player

You can download the sample project SphericalVideoSamplePlayer.zip provided at the top of the page. Here is a brief description of the sample player.

The sample player includes:

  • Functionality for handling spherical video related video metadata
  • Use of VideoTexture for spherical video rendering
  • Vertex and index generating code used for spherical video rendering
  • Helper code for handling camera view for rendering

Here is an overview for ActionScript classes that are provided to the developers:

SampleSphericalVideoPlayer class
This class provides some functionalities for basic player for spherical video.

SphericalVideo class
This class is similar to that of the StageVideo class, you may use this class to play spherical video.

Projection classes
These classes are used for generating vertices and indices corresponding to the given projection type.

Rendering spherical video

Create VideoTexture

The rendering of Spherical video begins with the creation of an object of class SphericalVideo.

sphericalVideo = new SphericalVideo(stage, 0, 0, 800, 500);

SphericalVideo class creates an instance of context3D to render the video using VideoTexture. Remember that the use of this context3D instance for any other purpose should be avoided. If required, it should be appropriately modified or a separate instance of Context3D should be used.

Attach video stream

On successful creation of the sphericalVideo object, a SphericalVideo.AVAILABLE event is generated. Then a call is made to onSphericalVideoAvailable(), which handles the NetStream that is attached to sphericalVideo

private function onSphericalVideoAvailable(e:Event):void { nc = new NetConnection(); nc.connect(null); ns = new NetStream(nc); ns.addEventListener(NetStatusEvent.NET_STATUS, onNetStatus); ns.client = this; sphericalVideo.attachNetStream(ns); ns.play(FILE_NAME); }

Spherical video meta-data

Now, an event is received which tells that there is meta-data attached to the video stream.

Spherical related information is delivered by a property called sphericalVideoInfo. A validity check for this property should be conducted before referring to it because this property is optional.

The property sphericalVideoInfo has a form of array of String, which means that there is more than one track and related spherical information. When referring to this property, the 0th String should be used because Flash Player always plays the 0th video track by default.

Meta-data is read and the sphericalVideo.setProjectionType (projectionType) sets the projection type of the video using the dynamic data received at runtime.

public function onMetaData (event:Object):void { if (event.hasOwnProperty("sphericalVideoInfo") && event.sphericalVideoInfo[0]) parseXML(new XML(event.sphericalVideoInfo[0])); sphericalVideo.setProjectionType(projectionType); }

Parse metadata

The meta-data received as an array of string type in XML format is parsed to read the tag values.

private function parseXML(info:XML):void { for each(var child:XML in info.children()) { if (child.localName() == "Spherical") { isSpherialVideo = child.valueOf(); } else if (child.localName() == "ProjectionType") { if (child == "equirectangular") { isSpherialVideo = true; projectionType = SphericalVideo.PROJECTION_EQUIRECTANGLUAR; } else if (child == "cubemap") { isSpherialVideo = true; projectionType = SphericalVideo.PROJECTION_CUBEMAP; } } else if (child.localName() == "InitialViewHeadingDegrees") { yaw = child.valueOf(); } else if (child.localName() == "InitialViewPitchDegrees") { pitch = child.valueOf(); } else if (child.localName() == "InitialViewRollDegrees") { roll = child.valueOf(); } else if (child.children().length() > 0) { parseXML(child); } } }

The above code retrieves and uses tag values attached to a video stream to manipulate the rendering as needed.

View port changes

Since a spherical video is supposed to provide a 360-degree view, there is a need for handling the view port changes made by user. Below, the code shows the basic handling for view port changes made by mouse (touch in case of Android).

protected function onMouseMove(event:MouseEvent):void { if (!trackMouseMove) return var newX:Number = event.stageX; var newY:Number = event.stageY; var deltaX:Number = (newX - oldX) * FOV_Y / 360.0; var deltaY:Number = (newY - oldY) * FOV_Y / 360.0; oldX = newX; oldY = newY; pitch += deltaY; if (pitch < -90) pitch = -90; else if (pitch > 90) pitch = 90; yaw += deltaX; if (yaw < -180) yaw = 180; else if (yaw > 180) yaw = -180; sphericalVideo.setCameraView(pitch, yaw, 0); }

The 'x' and 'y' values of mouse pointer are tracked and manipulated. The change detected is passed to the function setCameraView(). The setCameraView() function then transforms the view according to the values received.

public function setCameraView(pitch:Number, yaw:Number, roll:Number):void { viewTransform.identity(); viewTransform.appendRotation(roll, Vector3D.Z_AXIS); viewTransform.appendRotation(yaw, Vector3D.Y_AXIS); viewTransform.appendRotation(pitch, Vector3D.X_AXIS); if (context3D && texture) renderFrame(); }

Thus, the video is made interactive, letting user change the view port as needed.

Rendering

The actual output of SphericalVideo object is rendered onto the back buffer, which is created by the Stage3D that is connected to the VideoTexture in SphericalVideo object.

The rendering process is handled whenever SphericalVideo object receives a VideoTexture.onTextureReady event. Also, the rendering process is triggered when ActionScript calls SphericalVideo.setCameraView().

When receiving the VideoTexture.onTextureReady event, the SphericalVideo object renders spherically transformed video output with internal VideoTexture object and shader code.

The shader code uses vertex and index vectors that are generated for the given projection type.

Where to go from here