This project aims to develop a novel speech-based interface (speech and graphics) to enhance e-book content.

Current e-books are able to support dynamic animation when people click on graphic objects shown in e-book pages. However, most of them don’t support automatic animation based on a reader’s speech input when the content is read out loud.

This project investigates the role of speech input in enhancing the effective use of e-books. For example, when an adult reads a fairy tale (containing visualizable events such as “the frog jumps”) to a child, if animations of the events are activated and aligned correctly with the content of the story as it is read (“the frog jumps to the pond”) then the child is likely to be more engaged in listening and learning.

We plan to develop a semantic decoding module for this purpose. The main idea is to detect the semantic meaning of the reader’s spoken utterance and activate the associated animation. The outcome of the project will be a research prototype applied to e-books with graphic animations.

The expected tasks may also include user studies to assess the benefit of using speech-based interfaces for e-books in tutoring and learning.

There are potential uses of this type of speech-based interface for other Adobe products and services such as supporting marketers to get the relevant analytics information in graphics and text format using natural language queries instead of rigid SQL-like queries.