by AxelCyrille Ngonga Ngomo, InfAI, Alejandra García Rojas, ONTOS, and Irini Fundulaki, ICS-FORTH
The HOBBIT project aims at abolishing the barriers in the adoption and deployment of Big Linked Data. To this end, HOBBIT will provide open benchmarking reports that allow to assess the fitness of existing solutions for their purposes. These benchmarks will be based on data that reflects reality and measures industry relevant Key Performance Indicators (KPIs) with comparable results using standardized hardware.
Linked Data has grown rapidly over the last ten years [L1]. Organizations are increasingly interested in using solutions based on Linked Data. However, choosing the right solution for their needs remains a difficult task. HOBBIT’s rationale is to support organizations that aim to use Linked Data technologies at all scales (including Big Data) in the choice of appropriate solutions. To this end, HOBBIT [L2] will provide benchmarks for all the industry relevant phases of the Linked Data lifecycle [1] according to the approach shown in Figure 1.
Figure 1: HOBBIT Ecosystem
In particular, the H2020 Hobbit EU project will create benchmarks for the following stages:
1. Generation and Acquisition: Benchmarks pertaining to the transformation of unstructured, semi-structured and structured data into RDF.
2. Analysis and Processing: Benchmarks pertaining to the use of Linked Data to perform complex tasks such as supervised machine learning.
3. Storage and Curation: Benchmarks pertaining to the storage, versioning and querying of RDF data stored in corresponding solutions.
4. Visualisation and Services: Application centric benchmarks pertaining to queries used by software solutions which rely on large amounts of Linked Data.
The HOBBIT project will provide innovative benchmarks based on the following premises:
- Realistic benchmarks: Benchmarks are commonly generated with synthetic data that reflect a single and specific domain. HOBBIT aims at creating mimicking algorithms to generate synthetic data from different domains.
- Universal benchmarking platform: We will develop a generic platform that will be able to execute large scale benchmarks across the Linked Data lifecycle. The platform will provide reference implementations of the KPIs as well as dereferenceable results and automatic feedback to tools developers..
- Industry relevant Key Performance Indicators (KPIs): In addition to the classical KPIs developed over the last decades, HOBBIT will collect relevant KPIs from industry to make the assessment of technologies based on the industrial needs possible.
Comparability of results: Within its challenges, HOBBIT will provide the possibility of deploying benchmarks on hardware provided by the project but also in the cloud. Therewith, we will ensure that the results achieved by the frameworks benchmarked by HOBBIT are comparable.
Independent HOBBIT institution: HOBBIT will be grow to become a biasfree organization that will conduct regular benchmarks and provide European industry with up-to-date performance results. We target to initiate the organization in 2017. To achieve the goals of HOBBIT, we aim at bringing together three main categories of actors: solution providers, technology users and scientific community. In the first stage of the project we will target all three categories of actors that will provide us with (1) supplementary KPIs and (2) datasets from their domain for the mimicking algorithms used to create the benchmarks. Once the first version of the different benchmarks is available, we will challenge solution providers and the scientific community to evaluate their solutions.
These dereferenceable benchmark results will be provided in an open format [2]. The members of the HOBBIT community [L3] will be provided with the early results and early insights pertaining to the benchmarks and results of the project. HOBBIT’s consortium is composed of 10 organizations: InfAI (Germany, coordinator), Fraunhofer IAIS, (Germany), Foundation for Research and Technology Hellas (Greece), National Center for Scientific Research “Demokritos” (Greece), iMinds VZW (Belgium), USU Software AG (Germany), Ontos AG (Switzerland), OpenLink Software (UK), AGT Group GmbH (Germany) and TomTom Polska (Poland).
Links:
[L1] http://linkeddatacatalog.dws.informatik.uni-mannheim.de/state/
[L2] http://project-hobbit.eu/, https://twitter.com/hobbit_project
[L3] http://projecthobbit.eu/getinvolved/
References:
[1] S. Auer, et al.: “Managing the lifecycle of linked data with the LOD2 stack”, in The Semantic Web–ISWC 2012 (pp. 116), Springer.
[2] R. Usbeck, et al.: “GERBIL: general entity annotator benchmarking framework”, Proc. of the World Wide Web Conference 2015.
Please contact:
Axel Cyrille NgongaNgomo,
University Leipzig, Germany
Alejandra Garcia Rojas
Ontos AG, Switzerland
Irini Fundulaki
ICS-FORTH, Greece
