by Jesse Freeman
I've been building touchscreen applications with Flash for more than seven years, including an interactive project I created called Flash Bum. Recently, I've noticed a trend—the introduction of iOS and Android devices has caused developers to focus on building smaller touchscreen apps. It is truly amazing that today's technology allows us to carry these touch-enabled computers around in our pockets, but what about the need for large touchscreen applications in the business environment?
Last year, Roundarch built a touchscreen application for the New York Jets. The application, called the Jets Command Center, allows Jets executives to monitor stadium activity through a 42" screen display that hangs on the wall of the owner's suite. It operates on both a micro and macro scale to enhance the fan experience, as well as help the Jets franchise be profitable. It achieves both goals by taking the guesswork out of game-day operations by tracking stadium activities and making the corresponding data easily accessible and understandable.
Built with Adobe Flash technology, the Jets Command Center is a great example of a successful large touchscreen application. The development team had to address technical challenges and visual considerations while working with hardware limitations to deliver the project. In this article, I'll highlight some of the key points to consider when building your own large touchscreen applications.
By having a good understanding of the technology and the hardware that will display applications, you can avoid common pitfalls. Pay close attention to the following aspects when building projects that are designed for a large touchscreen display:
Flash is fast, and it's getting faster with each new release. However, there are still strategies you can use to optimize performance when developing high-resolution applications. I believe the two most important tasks involve breaking up large intensive processes and always trying to maintain a responsive UI. The last thing you want users to experience is an unresponsive application. Always ensure the app can quickly exit out of an intensive process, such as parsing data or displaying an animation, and strive to keep enough processing resources free to respond to user input at all times.
When it comes to developing large touchscreen apps, it is essential to consider how Flash will handle the overhead of a higher resolution screen. Due to performance challenges, the goal for data-heavy applications should be to create a thin client, which is an application that offloads heavy processing tasks to another computer or server. In order to manage the massive amount of data that the Jets Command Center processes, a dedicated server is used. This enables Flash to focus on delivering the richest visual experience possible without getting bogged down with parsing and processing all the data. This strategy is used by many large-scale Flash websites as well, but it is especially important to be aware of data processing when your application is running at a high resolution. Take advantage of as much free time in the elastic racetrack as possible for the render cycle. That was a key consideration when we developed the Jets Command Center (see Figure 1).
Figure 1. The Jets Command Center interface.
All of the data for the Jets Command Center is processed on a dedicated server that is then loaded into Flex in an easy to digest format. This ensures that loading data between sections is fast and that the UI stays responsive.
Today, it is relatively easy to purchase a touch-enabled LCD, but they have some limitations. Many people's experience with touch input involves using an iPhone or an iPad, but when it comes to LCD monitors, the experience is very different. Most affordable touchscreen displays only support two touch points. Additionally, when using a framework like Flex, a lot of the components are not designed to support more than one touch point—like a mouse. Thanks to touch API enhancements in Flash Player 10, it is now very easy to implement more complex touch interaction and gestures. However, it's still important to understand a project's touch limitations and find ways to overcome them as you develop and design your application. A general rule of thumb is to keep touch support basic; limit it to a single touch point and make sure the custom components you build can support the required touch points.
As with any new project, factor in time for a phase of design exploration. During this period, you should visually outline the goals of the application and how you plan to work out any UI issues. When it comes to large touchscreen applications, keeping the UI clean and making it easy to use is the most important challenge. Below, I've listed a few other important things to keep in mind.
Even though animation is one of the strong suits in Flash, attempting to display heavy animation or video at high resolutions is incredibly difficult. The key is to isolate animations to small areas on the screen. Avoid heavily animated backgrounds, especially on screen resolutions that are over 1024x786. Whenever possible, convert animations into video clips, because video content is hardware accelerated in Flash Player 10 and above. If you are using HD video, don't exceed 720p.
In the past, many command center applications tried to pack in as much data as humanly possible on the screen. Over the past few years, as interactive UX practices have evolved, we have learned to hide non-relevant data and focus on simplicity first. When looking at the Jets Command Center, the top data categories are consolidated into four main areas called The Pocket (see Figure 2).
Figure 2. The Pocket displays data regarding parking, tickets, concessions, and merchandise sales in the stadium.
The app highlights the four main points of interest while displaying the high level values related to each area of the stadium. This allows users to access a summary of the stadium's current activity in a single glance. For more detail, they can simply click on a category to see more in-depth information. The idea of constantly hiding non-relevant information and surfacing contextual information at the right time goes a long way towards ensuring that your application is easy to use and navigate. At any point, the user can quickly dig deeper into stadium statistics (see Figure 3).
Figure 3. Users can click to see the breakdown of concessions, which can be further analyzed on a level-by-level basis.
It is easy to forget how inaccurate and clumsy it is to use a finger for input compared to the precision of a mouse cursor. Remember to test early and often whenever you are designing touch-friendly components. Most buttons smaller than 60x60 pixels are not clickable. Also, on larger touchscreens, be sure to accommodate for inaccuracies in sensing touch input and delays. If the buttons you design are difficult to click, you'll end up frustrating users.
Choosing the right hardware is critical. Your selection will make or break the success of a large touchscreen application. Users have come to expect the responsiveness of an iPhone when it comes to any touch UI, so making the transition to larger touchscreens may be jarring for some. Remember the following considerations when selecting a touchscreen and a computer to run your application and perform testing.
Unfortunately, there aren't many options when it comes to purchasing large touchscreens, but here are my favorites:
This is the touchscreen installed in the owner's suite at New Meadowlands Stadium. It is a good-sized touchscreen and relatively affordable. We did encounter some issues getting the drivers to work and play nicely with Flash. Eventually we sorted these issues out, but early on in the development phase the Windows 7 drivers didn't register the display as a touchscreen in Flash. Also, it's important to note that it only supports two touch points.
This is the display I used when developing the Flash Bum project.
3M offers some very impressive medium-sized touchscreen displays. They are super responsive and accept multiple touch points. Lee Brimelow presented a great demo at Flash and the City last year using a 3M monitor that can support over 60 touch points. You can check out a video of his demo to learn more. 3M also makes frameless touchscreens, which are good when building custom installations.
To power an application like the Jets Command Center, a top-of-the-line computer is required. Any i5 or i7 processor with four gigs of RAM will do. Also, it is important to make sure the computer has a good video card. Although Flash isn't fully hardware accelerated, a good GPU will make a considerable difference when trying to run Flash on a high-resolution display at full screen. A dedicated server to handle processing data and formatting it in a way that your main application can quickly parse is also a good idea. The specs on a computer like this are considerably lower depending on the OS, language, and database you decide to use.
In most cases, constant testing of touchscreen apps throughout the development cycle is necessary. At Roundarch, we used HP TouchSmart software on a single computer to develop and test the app at the same time. These also make good installation units since they contain both the display and computer in the same case. The only issues to keep in mind with these computers are the two point input limitation with touch and the slow response time. However, for testing purposes, this is more than enough to work with, and they are usually powerful enough to use when developing. Finally, always perform testing on the final hardware to ensure that everything works properly. If you are relying on touch events that your development computer doesn't support, plan on using the production computer for your final development and testing.
The real advantage to using Flash for developing large touchscreen applications is the ability to build portable code. Several years ago, I used Flash to build the Flash Bum app for touchscreens and the web, but there weren't as many devices to consider. Today, there are so many options when deploying applications. A well-designed touchscreen app can easily be ported to the web, smartphones, and tablets. Obviously, each device will need some visual redesign to accommodate particular screen resolutions, but the jump from large screen apps to tablets is relatively easy. This is especially true with the new devices that are coming out, such as the BlackBerry PlayBook. They have native support for Flash and Flex, so they are prime candidates for corporate command center apps.
The ability to build a large touchscreen application that exists on a wall in a boardroom and simultaneously enable users to control and view the content on their tablets really amazes me. It's also impressive that you can have a touchscreen display in your office and drag content from the large touchscreen over to a more portable device, such as a smartphone or tablet, for on-the-go accessibility. By leveraging Flash to compile the same code base on all these devices, the cost involved in customizing the UI for each device is minimized.
Touch technology has greatly improved since I first started working with Flash as a flexible software solution for touchscreen applications. The ability to deploy Flash apps on multiple devices creates many exciting options for building apps with minimal cost. Whether you are developing art installations or corporate command centers, such as the app we designed for the New York Jets, it is important to understand the challenges you'll face when delivering large touchscreen applications. Optimize performance by breaking up large processes. Maintain a responsive UI while effectively managing the data. From an artistic standpoint, strive to design a UI that is clean and easy to use. Avoid displaying heavy animation. Focus on simplicity to ensure that the controls are user-friendly. And keep in mind that, at the end of the day, choosing the right hardware and thorough testing throughout the entire development cycle will help you deliver a successful project.
Jesse Freeman is a technical architect and technology evangelist at Roundarch. He specializes in designing and developing for the web and mobile platforms.