Hotaru: Intuitive Manipulation Techniques
for Projected Displays of Mobile Devices

Kosuke Miyahara, Hiroshi Inoue, Yuji Tsunesada, Masanori Sugimoto
Interaction Technology Lab., University of Tokyo
{miyahara, inoue, tsunesada, sugi}@itl.t.u-tokyo.ac.jp

1. OVERVIEW
A near future technology is presented: a mobile device with a projector and intuitive manipulation techniques by using a video camera mounted on the device. It is difficult to realize a mobile device with a small and light projector that still retains the feature of mobility, so we have developed a system to project displays of mobile devices by tracking their three-dimensional positions and orientations. The proposed system called Hotaru (a firefly, in English) allows users to annotate, rotate or transfer files between multiple devices by touching their projected displays with fingers.

Figure 1. Hotaru in Use

2. SYSTEM CONFIGURATION
2.1 Requirement

• 3D positions and orientations of mobile devices are automatically identified.
• Based on their 3D positions and orientations, where displays of individual devices are projected is determined.

Figure 2. System Overview

2.2 Location & Orientation Identification
A method with infrared LEDs and a stereo camera is used for identifying locations and orientations of mobile devices. Three LEDs are mounted on a circuit board designed to be attached to each PDA. By capturing the shape of the triangle formed by the LEDs through the stereo camera above users, locations and orientations of the PDA are estimated. The position and orientation errors with this method are less than six centimeters and ten degrees, respectively. Different blinking patterns of LEDs are assigned to individual PDAs for their identification.

Figure 3 PDA with IR LEDs

3. MANIPULATIONS ON PROJECTED DISPLAY
3.1 Recognition of Projected Displays
The system first performs the distortion correction of an image caused due to the wide-angle lens, and converts it into a binary format image. When four vertices of a projected display cannot be determined due to occlusion by human hands, the system applies the Hough transform and recognizes the projected display. Finally, by using the four vertices of the projected display in the camera coordinate and those of the PDA screen in the world coordinate, a transformation matrix between the two coordinates is calculated.

3.2 Recognition of Operations by Fingers

• click: a user's pointing area has not been changed for more than one second
• double click: a user's pointing area has not been changed for more than two seconds
• drag: a move of a user's finger after a click
• release: a user's finger has stopped for more than one second after a drag
• cancel: moving his finger away from the projected display

4. INTUITIVE MANIPULATION TECHNIQUES

Figure 4. Annotation (left), Rotation (center) and file transfer with a finger(right)

Figure 5. File transfer by overlapping projected displays of their devices

5. CONCLUSIONS & FUTURE WORKS
Informal user studies proved that Hotaru could effectively support collaborative tasks in co-located situations. We will further investigate the design and implementation issues through intensive user studies.

PUBLICATION (selected)

• Miyahara, K., Inoue, H., Tsunesada, Y., Sugimoto, M.: Intuitive Manipulation Techniques for Projected Displays of Mobile Devices, In Proceedings of ACM CHI2005 Extended Abstract, Portland Oregon (2005, to appear).
• Sugimoto, M., Miyahara, K., Inoue, H., Tsunesada, Y.: Hotaru: Intuitive Manipulation Techniques for Projected Displays of Mobile Devices, In Proceedings of INTERACT2005, Rome, Italy (2005, to appear).