:: projects :: (or see a selected list)
>> CS Division and the BiD Lab, UC Berkeley
MixT: Automatic Generation of Step-by-Step Mixed Media Tutorials
Users of complex software applications often learn concepts and skills through step-by-step tutorials. We hypothesize that a mixed tutorial with static instructions and per-step videos can combine the benefits of both formats. We present MixT, a system that automatically generates step-by-step mixed media tutorials from user demonstrations. MixT segments screencapture video into steps using logs of application commands and input events, applies video compositing techniques to focus on salient information, and highlights interactions through mouse trails.
Results were published in:
- UIST2012 (full paper) [pdf] [slides]
- CHI2012 (work-in-progress) [pdf]
- [Demo Video]
- The initial result was from a course project in CS260 Research Topics in HCI, Fall11, UC Berkeley. Teamwork with MS student Sally Ahn and undergrad student Amanda Ren (both in CS). Joint work with the Adobe Creative Technologies Lab.
Kinectograph: Body-Tracking Camera Control for Demonstration Videos
A large community of users creates and shares how-to videos online. It is often difficult for the authors of these videos to control camera focus, view, and position while performing their physical tasks. To help authors produce videos, we introduce Kinectograph, a recording device that automatically pans and tilts to follow specific body parts, e.g., hands, of a user in a video. It utilizes a Kinect depth sensor to track skeletal data and adjusts the camera angle via a 2D pan-tilt gimbal mount. Users control and configure Kinectograph through a tablet application with real-time video preview.
>> Tokyo, Japan
HELP JAPAN: 2011 Japanese Earthquake and Tsunami: Online Photos and Videos
A devastating earthquake of the magnitude 9.0 hit the Northeastern Japan on March 11, 2011, followed by tsunami, fire, aftershocks, and serious nuclear crisis. By collecting online media of the earthquake and related hazards presented in a map, we hope to help people learn more about this catastrophe, the serious damage it has caused, and the care and supports from the world. In this interface, users can navigate the map and see the photographs, videos, and more information provided by the mass and social media. Our goal is to encourage people to take actions to donate to the organizations and help people in need.
- Website in English, Japanese, and Traditional Chinese [link].
- The site has reached >20,000 views in two days and over 90,000 visits in total.
For more info:
- Blog post by Peggy [link]
- Blog post by Prof. Edmund W. Schuster: Important Web Site for Japan [link]
- The map is also embedded to the Android webapp designed by Ju Chun Ko: HOPE’n HELP for JAPAN EARTHQUAKE (in Chinese) [link]
- Project collaborated with Takumi Yamamoto; Map contributed with Carl Yu
>> the MIT Media Lab
Raconteur: From Chat to Stories
People who are not professional storytellers usually have difficulty composing travel photos and videos from a mundane slideshow into a coherent and engaging story, even when it is about their own experiences. However, consider putting the same person in a conversation with a friend – suddenly the story comes alive. Raconteur is a system for conversational storytelling that encourages people to make coherent points. It performs natural language processing in real-time on a text chat between a storyteller and a viewer and recommends appropriate media items from a library. A large commonsense knowledge base and a novel commonsense inference technique are used to identify story patterns such as problem and resolution or expectation violation.
Results were published in:
- IUI2011 as a long paper [pdf] [slides]
- CHI2011 as a Note paper [pdf]
- IUI2010 as a short paper [pdf]
- Peggy's M.S. thesis at MIT, September 2010
- [Introduction Video] and [Video in 20sec]
- Global Views Monthly (遠見雜誌) vol. 289, July 2010, pp. 226-227 (in Chinese)
- Ozsvald's blog (02/07/10) [link]
- Website [link].
Designing Interactive Narrative for Children
The interactive narrative system, called "A Little Quiz for the Little Hare" (story based on S. McBratney and A. Jeram's "Guess How Much I Love You"), aims to teach young children the concepts of measurement and comparison through the conversation between two characters. It explores the design space of enhancing interactive narrative using a commonsense knowledge database to understand players' intention and generate relevant narration dynamically.
- Results were published in ELO2010: the 4th Intl. Conf. & Festival of the Electronic Literature Organization [pdf]; paper co-authored with Angela Chang.
- For more info about this conference and our work, see Prof. Montfort's blog.
- This is a course project in CMS.845 Interactive Narrative [web] , Fall09.
Goal-Oriented Interfaces for Consumer Electronics
Consumer electronics devices are becoming more complicated, intimidating users. These devices do not know anything about everyday life or human goals, and they show irrelevant menus and options. Using common-sense reasoning, we are building a system, Roadie, with knowledge about the user's intentions; this knowledge will help the device to display relevant information to reach the user's goal. For example, an amplifier should suggest a play option when a new instrument is connected, or a DVD player suggest a sound configuration based on the movie it is playing. This will lead to more human-like interactions with these devices. We've constructed the Roadie interface to real consumer electronics devices: a television, set top box, and smart phone. The devices communicate over Wi-Fi, and use the UPnP protocols.
Commonsense Inside: Pursuing the Vision of Smart Home with Commonsense Thinking
Technology enables us to possess increasing numbers of functional devices at home, but also brings potential concern of resource overflow that decreases the usability. Can a home select proper Ways to Think for users based on the higher-level knowledge about the properties of different devices? This project suggests a framework design to make a home “think” and utilize home devices using commonsense reasoning. The proposed system includes a Difference-Networks to describe devices, and a Critic-Selector to react to user needs.
- This is a course project in MAS.731J The Society of Mind, Spring09.
Burn Your Memory Away: One-time Use Video Capture and Storage Device to Encourage Memory Appreciation
Although modern ease of access to technology enables many of us to obsessively document our lives, much of the captured digital content is often disregarded and forgotten on storage devices, with no concerns of cost or decay. Can we design technology that helps people better appreciate captured memories? What would people do if they only had one more chance to relive past memories? In this paper, we present a prototype design, PY-ROM, a matchstick-like video recording and storage device that burns itself away after being used. This encourages designers to consider lifecycles and human-computer relationships by integrating physical properties into digitally augmenting everyday objects.
Stress OutSourced: A Haptic Social Network via Crowdsourcing
Stress OutSourced (SOS) is a peer-to-peer network that allows anonymous users to send each other therapeutic massages to relieve stress. By applying the emerging concept of crowdsourcing to haptic therapy, SOS brings physical and affective dimensions to our already networked lifestyle while preserving the privacy of its members. This paper first describes the system, its three unique design choices regarding privacy model, combining mobility and scalability, and affective communication for an impersonal crowd, and contrasts them with other efforts in their respective areas. Finally, this paper describes future work and opportunities in the area of haptic social networks.
- Results were published in CHI2009 [pdf] and shown in SIGGRAPH 2010.
- [Project webpage]
- Fashioning Technology [web] (05/13/09)
- Makezine blog [web] (05/17/09)
- talk2myShirt [web] (05/26/09)
- TechNewsDaily [link] over MSNBC [link] (07/27/10)
- gizmag [link](07/30/10)
- Gizmodiva [link] (08/03/10)
- This is a course project in MAS.834 Tangible Interfaces in Fall08. Teamwork with Keywon Chung, Carnaven Chiu, and Xiao Xiao.
Slow Food Fast: a Connected Sustainable Kitchen Design to Promote Local Food Culture
This project addresses cultural, social, and spatial concerns, and promotes sustainability and local food cultures in the social housing community. We take the advantages of Slow Food movement that preserves the cultural cuisines and food ingredients, and combine with the efficiency of food transportation and knowledge exchange. We believe sustainability starts at home and in simple, everyday tasks, thus propose a connected sustainable kitchen design that includes a Community Storage Box, an interactive learning system, in a dynamic market-based distributed network.
>> the UbiComp Lab, National Taiwan University (or see my portfolio)
During cooking process, family cooks are commonly unaware of how many calories go into their prepared meals. This work presents a smart kitchen with Ubicomp technology to improve home cooking by providing the number of calories of ingredients that are used in prepared meals. In doing so, family cooks can more effectively control the meal calories based on family needs. Our kitchen has sensors to track the number of calories in food ingredients, and then provides real-time feedback to users on these values.
Results were published in:
- Persuasive2008 [pdf]
- CHI2007 [pdf]
- UbiComp06 [pdf]
- IEEE Pervasive Computing Magazine 2010 [pdf]
M.S. thesis 2008 from:
- Peggy Chi [pdf] [slides]
- Jenhao Chen [pdf]
- [Introduction video]
- [a cooking example video]
- Serious Games Market [link],
- Insight (台大智活) [link] (in Chinese)
- Economic Daily News (經濟日報) [link] (in Chinese)
- Teamwork with Jenhao Chen
Ubicomp Technologies for Play-Based Occupational Therapy
Ubicomp technologies can assist parents and occupational therapists in modifying behaviors in young children. In occupational therapy, an effective mean to motivate child behavior change is by designing playful activities which leverages the desire of children to play to induce their behavioral change. By embedding digital technology into activities, Ubicomp technologies can be used to enhance the effectiveness of play-based occupational therapy. We demonstrated two playful activity designs targeting on slow eating and tooth brushing behaviors in young children.
Designing Smart Everyday Objects
We surveyed and classified smart everyday objects based on the relation between the object’s digital (new) and its traditional functions, and the relation between the object’s digital (new) object-human interaction and its traditional object-human interaction. Then, we attempted to map out a design space for digital function and interaction of smart everyday objects.
- Findings were published in HCI International 2007 [pdf]
- Teamwork with Jenhao Chen
Intelligent System of Social Network Agents
As technology enables new and faster modes of communication among people, it has become increasingly difficult to manage our social contacts effectively. Existing contact management tools lack the support for intelligent analysis of our social interactions. Thus, we developed an intelligent system of social network agents called "Déjà Vu" for managing and analyzing personal social contacts over the web.
- This project was also the ImagineCup 2007 winner.
- Press: Microsoft Press [link] (in Chinese).
- Teamwork with Chia-chuan Hung, Yi-Jing Huang, and Ting-Yen Lee
Being a Food Vendor in Nightmarket
By analyzing and transforming the processes into a simulating game, we developed a multi-modal, interactive and role-playing game for people to experience selling food in a disruptive night market environment.