Multimodal, Multi-Device Prototyping Using Programming by Illustration

User interface designers are increasingly faced with the challenge of targeting multi-device, multimodal applications, but do not have tools to support them. This work proposes an informal prototyping tool, named CrossWeaver, which implements the programming by illustration (PBI) technique, enabling non-programmer designers to build multimodal, multi-device user interface prototypes, test those prototypes with end users, and collect valuable feedback informing iterative design.

PBI is a technique for user interface prototyping that involves building executable prototypes from example sketches. PBI has its origin in the informal interface approach [1], supporting natural human input, such as sketching, while minimizing recognition and transformation of the input. PBI also uses programming by demonstration techniques [2], enabling a working application to be built by an end-user based on concrete examples, in this case design sketches. CrossWeaver extends informal user interface and programming by demonstration research to multimodal, multi-device applications, enabling a designer to create and test a multi-device, multimodal prototype from a set of example-sketched storyboards.

Figure 1: Screenshot of the initial CrossWeaver prototype

[1]

J. A. Landay and B. A. Myers, "Sketching Interfaces: Toward More Human Interface Design," IEEE Computer, Vol. 34, No. 3, 2001.

[2]

A. Cypher, ed., "Watch What I Do: Programming by Demonstration," D. C. Halbert et al., ed., MIT Press, Cambridge, MA, 1993.