Project Hendrix: A call center Customer Experience Management solution

When the Technology Experience and Innovation Group at Adobe sat alongside our call center colleagues and observed the obstacles that got in the way of delivering the best-possible customer experience, we realized two things:

• First, it was mainly a technology problem that could be solved with the design-thinking and innovation approach that Adobe advocates.
• Second, by leveraging Adobe technology such as Flex, Adobe LiveCycle Data Services, and Adobe LiveCycle Collaboration Service, we could deliver a simpler, more intuitive, and more effective experience to all Adobe support and telesales agents.

As the development team, it made sense for us to apply the same successful strategies and technologies that Adobe corporate customers have used to improve internal Adobe operations. That's how the project code named Hendrix was born. It's been more than a year now that Hendrix has been empowering Adobe agents with a single, simple, and intuitive interface (see Figure 1).

The return on investment (ROI) has been exceptional, and the benefits wide-ranging. Hendrix has:

• Reduced average call handling time by 15%
• Reduced call center agent churn rate by 30%
• Enabled real-time access to rich account information for more effective troubleshooting
• Improved customer satisfaction scores by 10%
• Shortened agent training time from one week to one day
• For more details on the business challenges addressed by the project, see the customer showcase story. This article goes a little deeper than that story, providing a glimpse under the hood and a look at the architecture of Hendrix.

The primary goal in developing the software architecture and technical foundation of the project was to enable the team to be productive at and fully focused on implementing the business and user experience requirements. Our goals included the following:

• Reduce complexity through abstraction and separation of concerns
• Avoid reinventing the wheel; leverage existing legacy systems as well as third-party frameworks
• Avoid writing any boiler plate code by using code generation tools
• Efficiently fill the gap between developers and designers by directly consuming the designer graphical assets and using programmatic skinning techniques
• The development infrastructure and workflow also had to support frequent releases and continuous quality control. Last, but not least, all the key architectural choices were driven by requirements for high availability, scalability, and performance.

Infrastructure landscape

Figure 2 shows a simplified view of the Hendrix integration landscape.

Hendrix supports multiple user interfaces; it is multichannel and multiscreen ready. The first phase of the project targeted customer service agents and telesales agents. They were given a Hendrix web browser experience that fulfilled their needs as well as Adobe IT's requirements for painless roll-out, installation, and upgrade.

The second phase of Hendrix addresses more advanced use cases and targets technical support agents. For this release, Adobe AIR will be used to deliver a desktop client application that enables tighter integration with the operating system.

Next up is Hendrix Mobile. It too will be built with AIR, and its audience is our presales and sales teams. Hendrix will push the latest information about their preferred customers (critical cases, customer temperature, customer satisfaction, and so on) directly to their mobile devices.

Each of the Hendrix client applications is built on top of the same scalable JEE middleware. They all consume LiveCycle Data Services services to perform the following tasks:

• Access and reuse existing business rules and Create Read Update Delete (CRUD) transactions implemented in Adobe's legacy SAP systems
• Pull data from various existing systems (including the products catalog, marketing materials, translation engines, and serial number servers)
• Interact with the call center's phones and the existing Alcatel Lucent Genesys computer telephony integration (CTI) system

In addition, the Flex client running in Flash Player may directly invoke LiveCycle Collaboration Service features to enable the technical support team to use chat, screen sharing, and white boarding during their support sessions.

Software architecture overview

Now that you have a better understanding of why Hendrix was developed and some of its basic infrastructure, you're likely curious about its software architecture.

As Figure 2 showed, the Hendrix integration landscape is heterogeneous. To make the integration work, the architecture team had to establish a good separation of concerns and good modularization. To this end, the public APIs for the modules, libraries, packages, and classes were kept to a minimum. This reduced system complexity and lowered development and maintenance costs. The whole system can appear complex (see Figure 3), but taken one by one, the subsystem implementations are not that complicated.

On both the Flex client and the Java server, the development team opted for the same architectural style, which facilitated loose coupling and independent development, deployment, and testing.

• Server and client applications were split into small, loosely coupled units/modules/projects that could be developed, tested, and released independently.
• The development team used extensible inversion of control (IoC) microcontainers:
  • Parsley for Flex
  • Spring for Java
• To avoid reinventing the wheel, the team leveraged mature third-party tools including logging frameworks (sl4j in Java, Thunderbolt for Flex) and other various utilities (Apache libraries, Flexlib, Cairngorm 3).

When an Adobe engineering team builds an enterprise RIA for Adobe, the Flash Platform is a natural choice, unlikely to be challenged. Nevertheless, it's worth mentioning that the Flash Platform fulfilled major technical requirements, enabling Hendrix to support:

• A wide variety of operating system and browsers (even Internet Explorer 6)
• Rich clients (this makes Hendrix more responsive because we can decrease server load by moving data and state onto the client)
• Multimedia content

Furthermore, our approach was design driven. The information architects and user-experience design team led the requirements. Their goals were to streamline business processes, hide system complexity, reduce the need for training, and lower the average handling time. They produced a clear and choreographed user interface first expressed through stories, Adobe Fireworks wireframes (see Figure 4), and behavior documents, and then later through Adobe Illustrator visuals (see Figure 5).

The Adobe Flex Framework enabled the team to efficiently and quickly implement the desired pixel-perfect result:

• Flex components acted as extensible building blocks, which we blended together to build up rich visual components (wizards, timelines, and so on), choreography, and user gesture visual effects.
• The highly customizable skinning capabilities of Flex, which can be driven by Photoshop and Illustrator assets, bridged the designer developer gap.

Recall that among the biggest changes in Flex 4 were two new concepts around skinning. First Flex 4 introduced a new set of MXML tags defining graphics, text primitives, fills, strokes, gradients, bitmaps, filters, masks, alphas, transforms, and blend modes. Second, it introduced FXG, an XML-based vector graphics interchange format for the Adobe Flash Platform. These capabilities streamlined the Hendrix developer-designer workflow.

Hendrix designers created user interfaces as vector images (via Illustrator) that Flex developers later reused in their code (see Figure 6). The Illustrator assets were exported as FXG documents, compiled by the Flex SDK compiler, and ended up as highly optimized static components.

Flash Player or AIR?

The AIR runtime enables RIAs to run outside the browser. AIR not only enables the deployment of standalone HTML, JavaScript, and Flex applications but also the use of an embedded data store through the SQLite database or the encrypted local store shipped with it.

Using the Flex SDK you can create a release that targets Flash Player or AIR from your Flex code base. The biggest advantage of running an AIR desktop application over a Flash Player web application is the ability to interact with the underlying operating system (Windows, Linux, or Mac OS), which frees the application from the Flash Player and browser sandboxes. AIR applications can create notifications, change the dock or system tray icon, access the file system, access the microphone and webcam, and interact with other processes.

AIR features were not required by our user experience team in the first release of Hendrix. Instead, the first release was designed and released as a web application for Flash Player, which facilitated the roll-out and transparent updates to all end users, including customer service and telesales agents.

AIR on mobile

After Adobe sales and presales teams expressed their need for a Mobile CRM application, implementing Hendrix Mobile on AIR (see Figure 7) became part of the roadmap.

Our user experience team will transform the existing Hendrix application into a mobile application that will take advantage of the multitouch features, accelerometer, and finger gestures that AIR supports, along with the microphone, camera, video, and geolocation services as needed.

AIR technology can now be used to deploy applications on smartphones (including iPhone 3GS, iPhone 4, iPod Touch, and Android phones) and tablets (including, iPad, Xoom, and BlackBerry Playbook).

The AIR runtime comes preinstalled on a number of devices developed by Acer, HTC, Motorola, RIM, and Samsung. It is also available for installation through application stores such as Android Market or the Amazon Appstore for Android (stores that AIR developers can also use to distribute apps).

AIR 2.6 achieved feature and performance parity for Android and iOS platforms (using OpenGL ES2 for fast graphics). Developers can even take advantage of new features in iOS 4 such as multitasking, retina display for higher screen resolutions, and front and back camera support.

However, the AIR runtime is not available on iOS devices, so developers need to package AIR applications as native applications using the AIR Developer Tool (ADT). This is, by the way, something you could choose to do for any supported platform.

Flex frameworks

Flex architects like to debate the advantages and disadvantages of specific frameworks; Java architects have been there before. There is no silver bullet. It's worth noting that Flex is a framework in itself, and that the Flex 4 Spark model and its associated skinning capabilities can be considered an MVC framework.

However, when building advanced enterprise object-orientated systems (that require automated ways to construct, retrieve, and isolate objects) IoC frameworks are worth considering.

Parsley

For Hendrix, the Flex IoC framework of choice was Parsley. The team feels that Parsley is designed to be a framework for frameworks; a few of the Cairngorm 3 libraries that the Hendrix team contributed to are actually built on top of Parsley.

The Cairngorm 3 Module library is one example. Cairngorm 3 Module provides a set of infrastructure code that helps module loading and unloading and eases intermodule communication by augmenting the Parsley Messaging API so the messages dispatched to a given module are always guaranteed to arrive at their destination.

Cairngorm 3

In contrast to the earlier Cairngorm 2 framework, many parts of Cairngorm 3 apply across frameworks. If you're using Parsley, Spring ActionScript, Robotlegs, PureMVC, Swiz, Mate, or another framework on your projects, Cairngorm 3 could well be complementary.

Cairngorm 3 consists of:

• Guidelines that describe our Domain-Driven Design approach and advocate for tooling around continuous integration and quality
• A set of independent utility libraries and sample projects

As the Hendrix team extensively used Cairngorm 3 in our project, we also contributed back to Cairngorm 3. So if you want to dive into our architecture best practices and get to know our toolkit, we invite you to visit the Cairngorm 3 wiki and try out the Cairngorm 3 libraries and sample projects.

This section describes Hendrix intersystem communications, which enable fast and reliable data exchange between the Flex client application and the back-end services.

Object serialization

If you control both sides of the wire in your application, there is rarely a good reason to serialize and deserialize text (especially verbose XML text). Object serialization will always be much faster. Speed and scalability were major requirements in the Hendrix project, so we decided against web services (and the slippery SOAP approach) and chose to use object serialization. Specifically, we used the following technologies:

• Genesys Agent Interaction SDK and its Java Agent Interaction Layer (AIL) library. The AIL API enabled Hendrix Java middleware to interact through sockets with the existing Genesys Telephony/AIL server to manage agent login activity, monitor and change agent status, and initiate outbound interactions, regardless of media.
• SAP Java Connector (SAP JCo) and its underlying high-performance JNI-based RFC middleware. This enabled the TCP/IP communication between the Java middleware and the SAP back end.
• Action Message Format (AMF). AMF is a binary object format supported by Flash Player and implemented by the Adobe LiveCycle Data Services Java middleware solution.

AMF is used to serialize ActionScript object graphs. Once serialized, an AMF encoded object graph may be used to persist and retrieve the public state of an application across sessions or allow two endpoints to communicate through the exchange of strongly typed data.

James Ward has created a Census RIA Benchmark application that compares data loading using a variety of methods, including SOAP, JSON, XML, and AMF. As you can see in Figure 8, AMF is extremely efficient across the wire (even without compression). And, it's not just faster for loading–it also speeds client side sorting and filtering and requires less client-side memory.

HTTP or RTMP

Real-Time Messaging Protocol (RTMP) is a protocol designed for high-performance transmission of data between Adobe Flash Platform technologies. RTMP is made available as an open specification to create products and technology that enable delivery of video, audio, and data in the open SWF, FLV, F4V formats as well as the AMF format, which are all compatible with Adobe Flash Player.

With Adobe LiveCycle Data Services, there are two flavors of RTMP:

• One works over TCP and uses port 1935, a non-standard port. Because this port was impossible for the Hendrix team to open on some of the outsourced call centers' firewalls, we dismissed this pure RTMP protocol alternative.
• The second encapsulates the RTMP message within HTTP(S) requests. Named RTMPT(S), this flavor enables the protocol to traverse firewalls and use standard ports.

After considering RTMPT, the Hendrix time decided to keep it simple in the near term, and simulate push by using HTTP polling and piggybacking techniques. The Hendrix network topology is complex with outsourced call-center firewalls and proxies, Adobe intranet firewalls, and load balancers, so initially simplicity is a high priority. The team will evaluate a move from HTTP to the RTMPT protocol if the business expresses the need for:

• real time data push capabilities
• clean, immediate notification when a client is closed (This is an interesting feature in a call-center application if you need to know whether reps are online or not. The more classic HTTP-based services generally don't receive notification of a client going away until the HTTP session on the server times out.)
• reduced latency and better throughput (see Figures 9 and 10)

Hendrix middleware is a Java web application built with LiveCycle Data Services and running on a JEE application server. The LiveCycle Data Services messaging infrastructure provides support for the following:

• service-oriented APIs through its Flex remoting and proxy services
• publish-subscribe messaging APIs through its Flex messaging service
• resource-oriented APIs through its Data Management and Proxy services

Channel configuration

LiveCycle Data Services is protocol agnostic. Each combination of supported protocol and serialization mechanism acts as a message channel between the client and server, without affecting how messages are exchanged or processed by the client or server.

Messaging channels come in three main families:

• The RTMP channel creates a single duplex socket RTMP connection to the server. If the direct RTMP TCP connect attempt fails, Flash Player will attempt RTMPT (a tunnel through an HTTP proxy) or eventually fall back to adaptive HTTP requests that tunnel RTMP data back and forth between client and server.
• The classic AMF/HTTP channels, which leverage the JEE servlet API and use blocking IO. They can be configured as long polling, streaming, or simple polling with piggybacking enabled. (The first release of Hendrix used long polling and piggybacking.)
• The AMF/HTTP channels using LiveCycle Data Services NIO server, which are behaviorally the same as the servlet-based AMF channel/endpoint but use an NIO server and minimal HTTP stack to support scaling up to thousands of connections. (Hendrix made the move to NIO as the service opened up to more call centers.)

If you want more details on channel configurations, read Damon Cooper's blog post titled BlazeDS and LCDS: Channels, Channels Everywhere.

Exposing SAP services through LiveCycle Data Services

To fully leverage the existing legacy SAP CRM system, an ABAP development team implemented a set of Hendrix specialized façade services, aggregating the existing business rules and structures that existed within the CRM SAP legacy system. These ABAP functions are exposed as Remote Function Calls (RFCs), which is the standard SAP interface for communication between SAP client and server over TCP/IP or CPI-C. They are consumed by the Java middleware application server through SAP Java Connector (SAP JCo).

Thanks to SAP JCo these RFCs are wrapped in Java services and the ABAP structures in Java entities. The resulting Java API is then adapted in a façade API, exposed by LiveCycle Data Services, consumed by the Flex client.

As with object relational mapping techniques, the required plumbing leads to lots of boiler plate code. The Hendrix team developed its own solution to introspect the RFC and generate most of the RFC-to-Java-to-Flex mapping code. The LiveCycle Data Services engineering team looked at it and took it a few steps further. They will release an SAP Connector for LiveCycle Data Services later this year. This will augment Flash Builder with the tooling to do model-driven development with SAP. In addition to connecting to one or more SAP systems, the connector will enable developers to introspect and find RFC modules and functions, map SAP RFC functions to Flex remote functions, customize functions in Flex (modify function names, parameters, inputs and outputs), and generate ActionScript façades to communicate with SAP. For more on this, see Damon Cooper's post Introduction to the SAP Connector in Data Services.

Exposing Genesys CTI through LiveCycle Data Services

LiveCycle Data Services and Java made the computer telephony integration (CTI) really straightforward. The Genesys Java AIL library interactions are triggered by the end user in the Flash Player UI through LiveCycle Data Services Flex remoting services. Through these Flex remoting calls, the agent can:

• set his or her status (Ready, Not Ready, and so on)
• make an outbound call
• accept an incoming call
• toggle caller information details
• hold a call
• perform cold/intelligent/warm transfers with identified customer information
• perform a consult call
• release a call

The Genesys library core also provides an event mechanism, through which the Agent Interaction (Java API) application can notify users about server statuses such as incoming calls, hang ups, disconnection, and more

LiveCycle Data Services provides a real time publish-subscribe messaging API, which Hendrix middleware uses to publish the CTI events to only the subscribed Flex clients.

The CTI solution is reliable and enterprise ready because of the messaging reliability of LiveCycle Data Services, which ensures that messages are exchanged correctly across the network regardless of failures as long as the client and server hosts remain running. Messages are delivered in order, once, and only once.

If connectivity fails and is restored, any undelivered messages are automatically and correctly retransmitted with no risk of duplicate processing by the far end.

To activate this capability, only a few small changes to the LiveCycle Data Services XML configuration files are necessary, specifically:

• advanced messaging support in service-config.xml
• reliable messaging turned in messaging-config.xml

Leveraging JEE standards

Last but not least, the Hendrix middle tier also takes advantage of many JEE standards (implemented in the JBoss application server). In addition to simple logging support (which can be critical for production monitoring and error diagnostics), Hendrix leveraged more advanced JEE features, including:

• Java transaction management to control transaction boundaries programmatically around SAP CRUD operations
• Java Naming and Directory Interface (JNDI) to enable Hendrix middleware to discover and look up data and objects via names. This helped the team externalize the configuration (including all the SAP, CTI, and other back-end connection parameters) outside of the WAR file. As a result, we could build and deploy the same WAR file in the development, QA, staging, and production environments.
• Java Message Service (JMS) to manage, filter, and queue the CTI events
• Java Management Extensions (JMX) for managing and monitoring applications, system objects, and application managed beans. This was extremely useful to remotely clear the cache, reload the JNDI tree, interact with other application services, and better control the application at runtime.
• We also used JBossCache to avoid unnecessary back-end round-trips to fetch shared metadata, the products catalog, and similar data, as well as to enable clustering and session replication (coupled with JGroups).

Though this article provided a view of the architecture of Hendrix, there are many aspects and challenges that were covered only briefly. Below are a few resources that go deeper into a few of these subjects. In a future article, Hendrix software factory techniques will also be detailed and demonstrated. In the meanwhile, if you feel we should provide greater details on a specific aspect of this project (for example, SAP integration, CTI, security, modularity, or multiscreen support) let us know (in the comments, through Twitter, or via email).

• Cairngorm 3
• Options In Loose Coupling
• Parsley Framework
• The Flexible Configuration Options of Parsley
• Evans, E. (August 2003). Domain-driven design: tackling complexity in the heart of software. Addison-Wesley Professional.
• Flex 4 Spark whitepaper
• Flex 4 in a day
• FXG and MXML graphics
• AMF benchmarked
• LiveCycle Data Services 3 Performance Brief
• EXTREME Performance: LiveCycle Data Services 3.0 Messaging
• BlazeDS and LiveCycle Data Services: Channels, Channels Everywhere
• AMF specification
• RTMP specification
• JCo on the SAP service marketplace
• Blogs by the LiveCycle Data Services team: Mete Atamel, Damon Cooper
• Blogs by my Hendrix teammates: Yaniv, Dragos, Ionut