by Frank Hopfgartner (University of Glasgow, UK), Cathal Gurrin (Dublin City University, Ireland), and Hideo Joho (University of Tsukuba, Japan)
A satellite session of the NTCIR (Evaluation of Information Access Technologies) conference was experimentally held in Glasgow, allowing participants to present their work either in Europe or in Asia. This experience, designed to foster research and innovation across continental borders, was a great success.
Local conferences, such as TREC in North America, CLEF in Europe, and NTCIR in Asia, play a leading role in promoting information retrieval research by supporting novel campaigns and releasing datasets to share the latest research challenges. To gain access to these datasets, participants are requested to communicate their work in the form of working notes. Despite the overall success of these conferences, the main drawback is that these working notes are not peer-reviewed. This may pose problems, especially for researchers who cannot easily afford or justify travel expenses to attend such conferences. To overcome the problem of distance, we organised an experimental satellite session that allowed participants of the Asia-based evaluation campaign NTCIR [L1] to present their work either in Europe or in Asia. Given participants’ feedback, we see this as an attractive method to foster research and innovation beyond continental borders.
In 2016, we organised a benchmarking task of lifelogging technologies as part of NTCIR, an international series of evaluation workshops designed to enhance research in Information Access technologies. Our task, NTCIR-Lifelog [L2], focused on the development of novel and innovative techniques in the field of lifelogging. As outlined by Gurrin et al. (2014) [1], lifelog access and retrieval poses new challenges for the data science research community. Firstly, lifelog data is challenging to work with; by its nature, it is rich multimodal data (sensor and media data) with a significant semantic gap to user information needs. Secondly, there are many diverse reasons to access data from a lifelog; an initial suggestion is the “five R’s” of memory [2], each of which would require different access methodologies. Thirdly, lifelog data is a continuous stream of data, hence the retrieved unit of information is query-dependent. Finally, effective lifelog retrieval is a multidisciplinary activity, incorporating semantic enrichment, search, summarisation, computer vision, cognitive science.
As part of NTCIR-Lifelog, we released a novel lifelogging dataset, described in [3], and invited participants to work on one of the following two subtasks:
• Lifelog Semantic Access Task (LSAT). In this subtask, the participants had to retrieve a number of specific moments in a lifeloggers life.
• Lifelog Insight Task (LIT). The aim of this subtask was to gain insights into the lifelogger’s life. It follows the idea of the Quantified Self movement that focuses on the visualisation of self-tracking data to provide ‘self-knowledge through numbers’.
In order to participate in this evaluation campaign, participants are usually requested to present their approach at the NTCIR conference in Tokyo, which took place from 7-10 June 2016. However, since attending the conference in Japan is costly for researchers from Europe, this requirement might discourage interested parties to participate in this campaign. We therefore got the permission from NTCIR chairs to host a satellite session in Europe where teams could present their results.
Thanks to financial support from the European Science Foundation via its Research Network Programme ‘Evaluating Information Access Systems’, we were able to host such session in Glasgow at the same time and day when the Lifelog session took place in Tokyo. We identified two objectives for the satellite event in Glasgow. First of all, we wanted to familiarise researchers in Europe with the pilot task on comparative evaluation of information access and retrieval systems operating over personal lifelog that we organised as part of NTCIR this year. Moreover, we aimed to establish a network of lifelogging researchers in Europe.
Figure 1: Participants of the satellite workshop in Glasgow.
Overall, eight teams decided to participate in NTCIR Lifelog and presented their work [L3]. While six teams sent representatives to Japan, two teams from Europe presented their work in Glasgow. Participants who attended the satellite session in Glasgow, shown in Figure 1, had all worked on lifelogging before, but approached the topic from different angles, including digital preservation, quantified self, human-computer interaction, and data analysis. Judging from discussions during the workshop, we believe that both the dataset that was released as part of NTCIR Lifelog, as well as its tasks, has sparked their interest in engaging further with this evaluation task.
We received positive feedback from participants in Glasgow and Tokyo (NTCIR-12 Conference site), as well as from the organisers of the conference. Given this feedback, we are happy that the organisers of NTCIR-13 allowed us to repeat the experiment at NTCIR-13. We are therefore happy to invite interested researchers to participate in NTCIR-Lifelog 2 and present their results in December 2017 either in Japan or in Europe. For NTCIR-Lifelog 2, we are preparing an even richer dataset, consisting of at least 45 days of data from two active lifeloggers, including multimedia data, human biometrics, information access logs, and human activity records. An overview is given in Figure 2.
Figure 2: Overview of the dataset recorded for NTCIR-13 Lifelog 2
Links:
[L1] http://research.nii.ac.jp/ntcir/index-en.html
[L2] http://ntcir-lifelog.computing.dcu.ie/
[L3] http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings12/NTCIR/toc_ntcir.html#Lifelog
References:
[1] C. Gurrin, A. F. Smeaton, and A. R. Doherty. Lifelogging: Personal big data. Foundations and Trends in Information Retrieval, 8(1):1-125, 2014.
[2] A. J. Sellen and S. Whittaker. Beyond total capture: a constructive critique of lifelogging. Communications of the ACM, 53(5), 70-77, 2010.
[3] C. Gurrin, H. Joho, F. Hopfgartner, L. Zhou, R. Albatal: NTCIR Lifelog: The First Test Collection for Lifelog Research. In Proceedings of SIGIR 2016, pages 705-708, ACM, 2016.
Please contact:
Frank Hopfgarner, University of Glasgow, UK
+44 330 2472
