by Giuseppe Amato, Fabrizio Falchi and Paolo Bolettieri
VISITO Tuscany (VIsual Support to Interactive TOurism in Tuscany) is a research project which investigates techniques for producing an interactive guide, accessible via smartphone, for tourists visiting cities of art. The system applies image analysis and content recognition techniques to recognize photographed monuments. The user just has to take a picture of a tourist landmark to obtain pertinent information on his or her smartphone.
In the last few years, the problem of recognizing landmarks has received growing attention from the research community. As an example, Google presented its approach to building a web-scale landmark recognition engine that was also used to implement the Google Goggles service [3]. VISITO Tuscany [1] also addresses this issue, investigating and developing technologies in order to produce an interactive and customized advanced tour guide service to visit Tuscan cities of art. More specifically, it focuses on offering services to be used:
- During the tour – through the use of new generation mobile devices in order to improve the quality of the experience. The mobile device enables users to get detailed information about the artistic objects they are looking at or about their location. While taking pictures of monuments, places and other close-up objects, the users indicate what appears to them to be most interesting. When a picture is taken it is processed by the system to infer the user's interests and to provide relevant and customized information. For example, if a user takes a picture of the bell tower of Giotto, they can get detailed information (historical, artistic, structural techniques, etc) on this monument.
- Before the tour – to plan the visit in a better way. Information from other users of the system and their experiences , together with information already included in the database system and, more generally, on the web, can be employed by users to better plan their own visit. Interaction will take place through 3D graphic-based techniques.
- After the tour – to keep the memory alive and share it with others. The user can access the pictures taken and the itinerary followed through an interaction based on 3D graphics. Information and experiences can be shared with other users by creating social networks.
One of the main novelties of VISITO Tuscany, is the possibility of obtaining information about monuments by taking a picture of the tourist landmark with a smartphone. The acquired image is analyzed and the landmark recognized so that the user can obtain relevant information related to the monument.
Figure 1: Tourist information on a smartphone.
Figure 2: The VISITO Tuscany project services.
The landmark recognition system is composed of three main components: a client application that runs on a mobile phone, an image classifier that recognizes landmarks contained in pictures, and a digital library containing descriptions of various monuments. At the moment of writing, we have created recognizers for hundreds of monuments in three cities in Tuscany: Florence, Pisa, and San Gimignano. The mobile application is already available for the Android platform [2] and will be soon available also for iPhone. When the user takes a picture of a monument, the picture is first sent to the classifier that checks if one of the available monuments is recognized. When a monument is recognized, the description is retrieved from the digital library and sent back to the mobile device.
Landmark recognition is performed using local features and kNN based classification algorithms. We defined a new approach that relies on a revision of the single label kNN classification algorithm. More specifically, we propose an algorithm that first assigns a label to each local feature of an image query. The label of the image is then assigned on the basis of the labels and confidences assigned to its local features. In other words, our kNN approach is based on the similarity among the local features of the query image and the ones in the training set rather than similarity among whole images.
The VISITO Tuscany project is funded by the Tuscan Region and is coordinated by ISTI-CNR. The consortium includes three ISTI-CNR laboratories (Networked Multimedia Information Systems, Visual Computing and High Performance Computing), the security laboratory of IIT-CNR, and three private companies: Alinari24Ore, Hyperborea, and 3Logic MK. We thank the municipalities of Florence, Pisa, and San Gimignano for providing us with all authorizations to build the demonstrator.
Links:
[1] http://www.visitotuscany.it
[2] https://market.android.com/details?id=it.visitotuscany
[3] http://www.google.com/mobile/goggles
Please contact:
Giuseppe Amato
ISTI-CNR, Italy
Tel: +39 050 3152810
E-mail:
