Electrical Engineering
      and Computer Sciences

Electrical Engineering and Computer Sciences

COLLEGE OF ENGINEERING

UC Berkeley

   

2008 Research Summary

TinyMotion: Camera Phone Based Motion Sensing (TinyMotion)

View Current Project Information

Jingtao Wang and John F. Canny

TinyMotion [1,2] is a pure software approach that detects the movements of cell phones in real time by analyzing image sequences captured by the built-in camera. Typical movements that TinyMotion detects include horizontal and vertical movements, rotational movements, and tilt movements. As a result, a user can activate and access different functions of a phone (for example, scrolling and selecting the phone menu, zooming in/out pictures, moving an on-screen cursor to a given region, and even gesture/handwriting input) by moving, tilting, or rotating the phone.

Different from existing research work, TinyMotion does not require additional sensors (accelerometer, motion sensor), markers (barcodes, anchor symbols, dots), or external computing powers, and can run on today's main-stream camera phones without hardware modification. Even more, TinyMotion can detect camera movement reliably under diverse background and illumination conditions.

TinyMotion has used both image differencing and correlation of blocks for motion estimation. In the current implementation, TinyMotion generates 12 movement estimations per second, and spends 19-22 ms to process each image frame on a Motorola v710 phone. The memory needed is around 300 kb.

We have created four applications: Motion Menu, Vision TiltText, Image/Map Viewer, and Mobile Gesture, and three games: Camera Tetris, Camera BreakOut, and Camera Snake, to test the effectiveness of our algorithm.

Through both an informal evaluation and a formal 17-participant user study [2], we found that: (1) TinyMotion can detect camera movement reliably under most background and illumination conditions. (2) Target acquisition tasks based on TinyMotion follow Fitts' law and Fitts' law parameters can be used for TinyMotion based pointing performance measurement. (3) The users can use Vision TiltText, a TinyMotion enabled input method, to enter sentences faster than MultiTapwith a few minutes of practicing. (4) Using a camera phone as a handwriting capture device and conducting a large vocabulary, multilingual real time handwriting recognition on the cell phone is feasible. (5) TinyMotion-based gaming is enjoyable and immediately available for the current generation camera phones.

We are now exploring new cell phone menu layouts and navigation mechanisms based on TinyMotion. We are also researching novel domain-specific applications in areas such as presentation control, accessibility, and home automation.

Figure 1
Figure 1: A TinyMotion enabled game--camera Tetris

Figure 2
Figure 2: Text entry speed--Vision TiltText vs. MultiTap

Figure 3
Figure 3: Handwriting samples collected by Mobile Gesture

[1]
J. Wang and J. Canny, "TinyMotion: Camera Phone Based Interaction Methods," ACM CHI, Montreal, Canada, April 24-27, 2006.
[2]
J. Wang, S. Zhai, and J. Canny, "Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study," ACM UIST, Montreux, Switzerland, October 15-18, 2006.