Judges' perspective: What makes a winning application

by Matt Chotin

When Adobe announced the Flex Developer Derby contest this past March to celebrate the Adobe Flex 2 beta 2 release, we knew we would get to see a lot of cool stuff, which is one of the most rewarding perks of my job as product manager on the Adobe Flex team — to see how developers use our software solution to build great apps and deliver great experiences.

The contest was designed to motivate and challenge Flex developers to make the most of the new features and components of Flex 2, and judging by the entries we received, the only regret we have is that we had to limit the contest to entries from North America (because of federal regulations regarding international competitions and prizes).

We received more than 60 entries in six categories — Product configurators/Self-service guided applications, Components and gadgets, Data dashboards, Mashups, Communication/Collaboration applications, and Just freakin' cool — which were submitted to a panel of five independent judges for review and evaluation. And even though not all of them could be winners, in the words of one of the judges, "They were all really cool entries."

The judges

When looking for the judges, we looked closely at individuals who provided high-quality contributions via blogs and forums discussing the Flex 2 beta release. We then sent out invitations to some members of the community who have demonstrated exceptional Flex expertise in their careers and who we could trust to be entry neutral. Five developers were brave enough to accept our invitation and made time in their busy schedules to review and evaluate the work of their peers.

Joe Berkovitz

Joe Berkovitz

Joe Berkovitz was chosen to join our panel of judges because he has an eye for clever ideas. Joe is chief architect of Allurent, where he is building an e-commerce platform based on Flex. Joe has spent more than 25 years in the software profession, as an architect, interface designer, and engineer, and he has worked as a consultant with such diverse clients as Harvard Business School, Sony, and BMG Music Club. His Flex experience dates back to the release of Flex 1.5.

Dimitrios Gianninas

Dimitrios Gianninas

Dimitrios Gianninas was selected because he has a knack for user interface design and knows his way around Flex. He was a beta tester for Flex 1.0 and has played with Flex 2 since its alpha version. Dimitrios is a software developer with Optimal Payments Inc. as well as a member of the Cairngorm committee, which oversees the maturation of the open-source Cairngorm framework and ensures that Cairngorm will reflect the collective best practices of day-to-day Flex developers.

Ryan Stewart

Ryan Stewart

Ryan Stewart stood out because he never appears to stop thinking about the future of rich Internet applications (RIAs). Ryan got into Flex as a developer for the Wharton School of the University of Pennsylvania, where he leveraged the power of RIAs to deliver world-class learning applications for the faculty. He also writes about RIAs (including Flex) for Ziff-Davis at ZDNet.

Jesse Warden

Jesse Warden

Jesse Warden is a Flex consultant and professional multimedia developer who was invited to join our panel because he has an open mind for all things Flash and Flex related. As a Flash turned Flex developer, he can appreciate the artistic aspects of a project as well as recognize what can be done in Flash without taking advantage of Flex.

Dave Wolf (no photograph) is Vice President of Consulting at Cynergy Systems and a constant contributor to the flexcoders mailing list, especially on J2EE issues with Flex.  At Cynergy, he oversees consulting operations worldwide and spends most of his time interacting directly with the field and working in every time zone, but his own

The process

In case you're wondering how we picked the winners, here's the approach we took to the judging process. All qualifying submissions were divided among the judges without particular preferences. The judges in turn reviewed and evaluated their assigned applications according to the criteria set forth in the official contest rules, which included that all applications had to be hosted on a server and submitted to the Flex Developer Derby by providing a URL link to the hosted application on the submission form. Although the judges didn't discuss any of the submissions during the evaluation phase, they did share links to what they considered to be cool applications. And if a judge couldn't get an app to work, the contestant was given ample opportunities to provide the necessary fix.

Each application was judged on a point system, with points being awarded for originality, ease of use, design, and use of Flex features. The judges submitted their final scores to us, and we compiled a list of all entries and their respective scores for each category. We then circulated the list back to the judges, so they had a chance to look at all of the entries, including the ones they didn't judge, to see how they fared in comparison. On June 7, we all got together (with yours truly as moderator) to look over the final, ranked lineup and to determine the winner in each category. If a judge didn't agree with a proposed winner, then we talked it over as a group while looking at the application and the runners-up until we reached a consensus.

The criteria

Of course, some judges were sad to see that their favorite apps didn't win first place, but a favorite may not have necessarily exhibited the qualities that ultimately distinguished the winning app. Beyond the criteria spelled out by Adobe, the judges brought some of their own expertise and discretion to the table when it was time to pick the winners. Dimitrios, for example, was "looking for something fresh and unique," and when he saw a developer using Flex to present data in a new, not just form-based, way or an app that used the advanced features of Flex 2 (such as Flex Data Services), it caught his attention.

Ryan, who is a big believer in the idea of software being delivered over the web as a rich Internet application, appreciated entries that could have been written as a regular desktop application and distinguished themselves through a high-quality user experience. "One of the major benefits of Flex is its ability to provide a very rich, interactive experience to the user. That is going to be so important as web applications mature, and I felt that the entries that focused on the experience were the best representatives of what to strive for in a Flex application... I was looking for an application that I could point my readers to and say, 'This is how software will be delivered in the future.'"

To see and learn more about the winning applications, read Announcing the Flex Developer Derby Winners in this issue of the Edge.

Separating the Best from the Rest

Now that you know what the judges were looking for in a winning app, you may also be interested to know why an otherwise cool app fell short.

One of the most common reasons for rejecting an app had to do with developers not making the most of the features in Flex 2. If an application didn't make good, creative use of Flex components and APIs, it was rejected.

Another reason an app didn't make the final cut was its design and execution. Comparing the Derby entries, the judges noticed that quite a few applications were too heavy on their use of server-side logic. For the judges, it became very difficult to determine how much of the final result could be attributed to Flex skills versus server-side skills. As Jesse put it, "Since we were judging for Flex, contestants inadvertently hurt themselves because they spent more time on server-side development than on the Flex side, even though the application may have been better."

Judges also knocked off points when an app lacked ambition, functionality, or style — which shouldn't come as a surprise in the case of a UI-centric product such as Flex. In some instances, for example, developers had not taken the time to add creative touches to their user interfaces or make their apps distinct enough from the Flex store demo. Flex has such awesome potential; you can take it as far as you want, and not taking the time to put on the finishing touches, in some cases, did make the difference between a well-designed app that met with the judges' unanimous approval and one that fell short by a couple of votes.

Related article: Announcing the Flex Developer Derby Winners

Matt Chotin is a product manager on the Adobe Flex team who focuses on the SDK. You can often find Matt answering all kinds of questions on the Flexcoders list.