Artwork by Matt Rae
Design is not just a visual experience. As humans, we experience the world in many different ways, not just visually, and so to create truly immersive experiences, other senses like sound should be considered. A common method many apps, devices, and web experiences take is including auditory feedback, playing sounds or speech to provide guidance to a user. Using audio and visuals in unison can create a richer experience and make these experiences more accessible to those who may only be able to experience it through sight or sound.
This guide will explore how to use audio and speech playback in Adobe XD to take your visual designs to the next level. Pair these features with voice commands to create robust auditory experiences, like virtual assistants, all in Adobe XD.
Triggers and actions
Prototyping in Adobe XD consists of two core elements, a trigger, and an action. Triggers are what initiate an action. There are a number of triggers available in Adobe XD including Tap, Time, and Voice. These triggers can be used to create actions like Transitions, Auto-Animate, Overlays and more. In this tutorial the focus will be on the actions of Speech and Audio playback, used to play sounds, or read back a selection of text to the user.
These actions pair really well with the Voice trigger to create interactive and accessible experiences using voice and sound. To learn more about how to use the Voice trigger in prototyping, visit this help guide on setting up voice triggers in Adobe XD.
The following examples will walk through how to apply the Speech and Audio playback actions in various situations, to create different effects.
Speaking a text selection
When designing with accessibility in mind, having the ability to audibly orient someone to the current screen or actions available is very important especially for those with visual impairments.
Using Speech Playback
Using the Speech Playback action and the Time trigger in Adobe XD, this is easily achievable. To start, set a Time trigger on the artboard you wish to have read.
Next, set the action to Speech Playback. A text box will appear below the action dropdown where the content to be read can be written. If the content already exists in your design, simply copy and paste content from a text box into this input field, or alternatively it can be entered manually.
It is important to note that the engine that runs the speech playback will read the text as written, including grammar and punctuation. So if you don’t use commas or line breaks, it will be read as such and may be hard to understand. Natural pauses can be added by pressing enter and adding a break between each paragraph or line.
Various voices are available spanning different ages and genders, so experiment with the sound and tone of voice that match your experience the best - Joanna is a similar sound to the default voice assistants on the market today.
Using Audio Playback
Alternatively, if the built-in voice options don’t suit the style of your design, a pre-recorded audio-file can be loaded with a narrator reading through the content of the design. The setup works the same, however rather than selecting a Speech Playback action, select the Audio Playback action, and upload the desired file rather than pasting in text.
This setup works well for custom voices, but is not as dynamic as it requires a new audio file anytime text or copy changes.
Playing error and success sounds
A common design element in many mobile or desktop applications is the use of sound to signify successes and errors. Think about the classic swoosh sound when an email is sent in Apple Mail, or the crunch of paper when the virtual trash is emptied on a computer.
Sounds like these are a key part of experience design, and are important reinforcements to user-actions. Including these in your Adobe XD prototypes is just as easy as building out speech playback and other commands.
To do this, the Audio Playback action will once again be used. In this case the action should be applied as a secondary action either on a state change (if showing an error state on an input or form, for example) or in transitioning to another artboard.
Applying sounds to component states
When applying these sounds to the transition between component states, the audio action is applied as a secondary action to a tap. So this means that as a default, each tap interaction can have a sound associated.
Connecting this button component, there are three states: the default state, an error state, and a success state. By default the click of the button is connected to a success state, but can be overridden on an instance to go to the error state.
To add sounds, simply navigate to the existing prototype links and find the second action section in the Properties Inspector. Here an Audio Playback action can be added. Select an error sound for the transition to the error state, and a success sound for the transition to the success state. If you’re looking for sound inspiration, this Sound UI Kit from Sam Anderson is a great starting point.
Preview the design and turn up the volume so you can hear the sounds as the states change. On click, the state will Auto-Animate as the first action, followed by the audio playback.
Prototyping a music player
In the past, prototyping audio apps for podcasts or music streaming left a little to be desired. Clicking through static images doesn’t give a sense of the full experience the designers are trying to create. With Audio Playback in Adobe XD that all changes, and the Play button is no longer a static asset.
In this music player interface, music should start playing when the play button is pressed. Once again states can be used here, playing a selected audio file when the button transitions from play to pause, but it can also be done with just two artboards.
Audio files will continue playing until they are complete, or until the prototype either navigates to a new artboard, or a new audio or speech playback action is initiated. Therefore it is easiest to articulate this process across artboards.
Artboard to artboard
To set this up, two artboards are used. One has the Play button with a play icon, and the other has a pause icon. This allows the user to toggle between play and pause like most music applications.
Using Auto-Animate, connect the Play button from Artboard 1 to Artboard 2, and the Pause button from Artboard 2 back to Artboard 1. Set the easing to whatever you like, but make sure the trigger is a Tap trigger.
Next, add an action to the same interaction. Select Audio Playback, and then select the audio track you wish you play from your computer, or the list of recently used files. If you’re looking for open source sample music, Adobe Stock has an extensive collection of audio files available for license.
Finally, connect the pause button back to Artboard 1. There is no additional action here required, as the action of navigating back to the first artboard will pause the track. However, you can add in an additional action to play a chime or other auditory feedback if desired.
With the play and pause functionality now connected, hit Preview to see Audio Playback in action.
With component states
To do this with component states, the approach is similar to what was done with success and error sounds. In this scenario one artboard exists, with a component that has two states: Default and Playing.
In the same way as before, connect the states together using a Tap trigger and Auto-Animate as the first action, used to navigate to the other state.
Next, add in a second action on the Tap from Default to Playing, and select Audio Playback. Select the song once again.
Next, connect the pause state back to the default state with a Tap trigger and Auto-Animate as the action. If you only use one action you will notice that this doesn’t stop the music from playing, as the prototype hasn’t transitioned artboards, or been triggered to play a different auditory file.
To ensure the music pauses, simply add a second action to the link from Played to Default. Since the pause action will be quick, using a simple click or ding sound will be enough to indicate the action is complete.
Now when the prototype is played, toggling between the states will create a smooth play/pause transition to bring this app design to life.
Whether you’re adding some fun to a design with sounds, or creating a robust voice interface, beit for accessibility or for creating a hands off experience, the audio and speech playback options in Adobe XD make it easy to create rich auditory experiences.
