by Xenophon Zabulis, George Margetis, Panagiotis Koutlemanis and Constantine Stephanidis
The “Interactive Documents” system augments documents, books, and leaflets on the surfaces on which they are read. The augmented content complements that of the printed document interactively; the user can touch regions of interest both within the printed matter and the augmented content.
The “Interactive Documents” system augments printed documents that are placed upon a planar surface (e.g., a desk or a table) with multimedia content and interactive applications. Using a projector above the surface, augmented content is dynamically displayed around and upon the printed document, aligned in real-time with its 2D location and posture. The system was developed in the context of the Ambient Intelligence and Smart Environments Programme of the Institute of Computer Science of the Foundation for Research and Technology – Hellas (FORTH-ICS).
As soon as they are placed on the surface, printed documents become interactive. Users can select any of the interactive hot-spot areas of the document in order to view related multimedia content (e.g., images, videos) or deploy interactive applications. The system employs an RGBD sensor (a conventional and a depth camera combined) as a sensing modality to identify documents and user fingertip contact with them.
The RGB component of the signal allows recognition and localization of documents on the surface. This is achieved by retaining a database of known documents, which stores the visual appearance of each printed page, in terms of visual features in each. When running, the system matches database features with features imaged by the camera to recognize the document pages. The spatial arrangement of these features in the database enables the estimation of location and orientation of the document on the surface. As features are local, recognition is robust to document occlusions by user hands.
The depth component of the signal is employed in tangible interaction with documents, multimedia content, and applications. Tangible interaction avails detection and localization of fingertips upon the surface or document. Using the location and posture estimates of the document, touch is localized upon the printed page. This way, touch is interpreted with respect to the page, to find which word or figure in the document, or application, the user interacts with. By knowing the page location touched, underlined, or encircled by the user, the system can provide content and context-based information and assistance, as page contents are associated in the database with their semantics.
Figure 1: User interaction with augmented printed documents. Main: the system highlights regions of potential user interest; upon user touch, relevant content is augmented, juxtaposed to the document. Thumbnails: multimedia augmentation in a music and poetry book (left), map augmentation providing city tour information on the printed map (middle) and visitor information (right).
Augmentation is enabled by common calibration of the projector and camera. The system translates the location and posture of the document to the projector reference frame. This way, not only content is presented aligned and upon the document, but documents can be moved and rotated along the surface at user convenience, with the augmented content being “dragged along” in real time.
“Interactive documents” is a versatile system that can accommodate applications from different domains. It has been employed as an educational tool, to augment books with context-based multimedia information, and provide interactive exercises related to the content of each book page. Another use is to provide information to the visitors of a city upon a map; this use is installed at the Tourism Office of the Municipality of Heraklion, where tourists can place their own copy of a printed map, upon which the system highlights places of interest. The visitors can touch on the highlighted areas and see videos and images related to the selected place, as well as guidance information. The system finds use in expos and marketing, where printed leaflets are enhanced with multimedia information and interactive applications about the featured product.
Link:
http://www.ics.forth.gr/ami/projects/view/All/Interactive_Documents
References:
[1] G. Margetis, X. Zabulis, S. Ntoa, P. Koutlemanis, E. Papadaki, M. Antona, C. Stephanidis: “Enhancing education through natural interaction with physical paper”, Universal Access in the Information Society, Special Issue on “User Experience and Access using Augmented and Multimedia Technologies”, pp. 1-24, 2014, DOI: http://dx.doi.org/10.1007/s10209-014-0365-0
[2] G. Margetis, X. Zabulis, P. Koutlemanis, M. Antona, C. Stephanidis: “Augmented interaction with physical books in an Ambient Intelligence learning environment”, Multimedia Tools and Applications. Published on-line: 10 January 2012 DOI: http://link.springer.com/article/10.1007/s11042-011-0976-x
[3] G. Margetis, P.Koutlemanis, X. Zabulis, M. Antona, C. Stephanidis: “A Smart Environment for Augmented Learning through Physical Books, in Proc. of the IEEE International Conf. on Multimedia and Expo (ICME 2011), Barcelona, Spain, 2011.
Please contact:
Constantine Stephanidis
ICS-FORTH, Greece
Tel: +30 2810 391741
E-mail:
