by Tamás Kifor and Tibor Gottdank
3D visualization and communication devices are developing rapidly. A current project at SZTAKI introduces these innovative technologies to telemanipulation and communication. The main goal is to offer a novel tool for people working on a common task and enable them to cooperate in a common intelligent virtual 3D environment.
The first implementation of this effort is Mouse3D. This program facilitates the use of a mobile device (currently Apple iPhone and Apple iPod Touch) as a spatial computer mouse acting as a 3D controller.
With Mouse3D we can control applications (eg PowerPoint presentations, virtual 3D world applications) on our PC with our mobile device. It is especially handy if we want to carry out control a few steps away from our PC screen, walking freely around a room and don't want to be bound to the keyboard and mouse.
Mouse3D (for VIRCA - SZTAKI 3D Virtual Collaboration Arena application) has been developed within HUNOROB, a Hungarian–Norwegian research based innovation project. It enables the user to control their computer with a smart mobile device from a short distance. Distance parameters and some control mechanisms are similar to Wii. Smart mobile devices include PDAs or smartphones equipped with a touch screen, and an accelerometer/giroscope. Unlike Wii, such controllers are always in the user's pocket and don't need any special hardware. It is also possible to use several mobile devices simultaneously to control an application. This solution increases freedom of control (eg a user may have an iPhone in their right hand and an iPod Touch in their left) and makes possible a more complex control involving more people at the same time (eg multiplayer games).
Figure 1: Using Mouse3D in SecondLife application.
Practically any application can make use of this controller provided it has an adapter for Mouse3D.
In our implementation there is a standalone client application on the mobile device and a “Mouse3D driver” on the controlled computer. There is also an optional web application component for secure authentication (we call this a domain server).
Within the framework of the project we have created an iPhone controller for VirCA. More recently, we have been working on setting up an open API (Application Program Interface) for Mouse3D adapters and implementing sample adapters to SecondLife, Lego NXJ Robot and PowerPoint applications.
Discovery, authentication, security and control (discussed below) are the major challenges for this approach. The Graphical User Interface (GUI) on the mobile device has been kept as simple as possible to allow users to focus on the controlled application (the computer screen or the controlled robot) rather than the mobile device.
- Discovery: The mobile client can search for controllable computers on the local network (by IP address) with a broadcast message and the user can choose from a list of computer names.
- Authentication: This is an optional feature, which can be made mandatory by the computer user. There are two ways for authentication: either using LDAP (Lightweight Directory Access Protocol) or using a password.
- Security: The biggest issue of security in this application is that the mobile device and the controlled computer don’t trust one another. In case of LDAP authentication we use a domain server component (a web application) for the authentication trusted by both the mobile device and the computer. With the combination of authentication, security code and connection confirmation, the user can define the proper security level for this computer.
- Control: In order to implement the control features, we have to match possible mobile functions (eg tap, tilt or swipe) with the application functions (eg walk forward, zoom in or go to the next page). Some mobile functions have number values that have to be transformed to an application function value (eg tilting the device at 30 degrees means walk at 6 km/h).
A traditional mouse is an input-only device. A PDA however, can be considered as an input-output device. Controlled applications can send output events to their controller. Such output events are vibration, playing sound, showing images and messages. Input events are tap, tilt, swipe, drag, shake, pinch and screen rotation.
After the completion of the first development, we decided to make an open API for adapters between the Mouse3D driver and a controlled application. This is a platform independent Java API but adapters can use platform-specific libraries, as well (eg Windows DLLs).
The advantage of this API is that developers don’t have to implement a controller client on one or more mobile devices, bothering with communication between the mobile client and the computer, implementing authentication and security methods. They only have to implement the Java API and communication between their adapter and the controlled application.
Figure 2: An overview of Mouse3D architecture.
The Mouse3D technology represents the potential of next generation PC tools. Beyond the traditional PC peripherials (keyboard, traditional mouse etc.) Mouse3D, as a feature of mobile devices, would be the general 3D tool for various purposes (eg for presentations, games, research, web shopping).
We plan to extend the availability of Mouse3D in the near future. We want either to implement Mouse3D mobile clients on other mobile platforms (probably Android and WinPhone7) or to make an open API for Mouse3D mobile clients as well.
Links:
Mouse3D website: http://mouse3d.sztaki.hu/
HUNOROB project: http://www.itm-lab.org/academic-research-and-r-d-programs/hunorob-programSystem
Development Department of SZTAKI: http://www.sztaki.hu/department/RFO/
Please contact:
Tamás Kifor
SZTAKI, Hungary
Tel: +36 1 279 6287
E-mail:
