Unified Access to Internet Measurement Data

by Felix Strohmeier, Martin Nilsson and Demetres Antoniades

Optimizing control, management and flexibility of future network infrastructures requires a good understanding of network and application behaviour. The FP7 project MOMENT (Monitoring and Measurement in the Next-Generation Technologies) combines a diversity of data sources, enriches them with semantic information, and presents them via a unified interface in a user-friendly way.

Since the Internet does not by design integrate network measurement or quality-of-service assurance mechanisms, it has always been necessary to perform measurements independently from the actual communication in order to obtain status information from the network. Deployed monitoring and measurement infrastructure and tools represent essential technology for migrating to a future Internet. Besides the per-domain operational monitoring performed by Internet Service Providers, research institutes and projects are establishing monitoring infrastructures for a large-scale view of the Internet. Owners of such infrastructures are willing to share their collected data for research and development, but usually only provide proprietary interfaces such as Web-based graphical user interfaces, which do not allow programmatic subsequent processing of measurement results.

Heterogeneous Data Sources
As depicted in Figure 1, the MOMENT project targets the creation of a mediator that allows unified access to several current and future sources of network monitoring and measurement data. Existing measurement infrastructures and tools like Dimes and Etomic make their data available via different access methods. Three examples of data sources created by ERCIM members cooperating in the MOMENT project are presented in the following. First, MINER is the Measurement Infrastructure for NEtwork Research that orchestrates multiple measurement tools. It allows measurement scenarios to be specified and scheduled, and results to be retrieved from them. These services are provided by a tool-agnostic unified programming interface. Secondly, BART (Bandwidth Available in Real-Time) is a tool for measuring end-to-end available bandwidth in real time, generating a continuous stream of measurement data. Finally, Appmon is a traffic classification application able to attribute network traffic to the application of origin, by inspecting the payload of the packets. Appmon provides both real-time and historical usage patterns for a number of popular applications, such as BitTorrent, eDonkey and HTTP.

Figure 1: MOMENT Unified Interface.

Semantics Behind
For unified access to such diverse data sources, it is both a necessary and a challenging task to define a common standard for data formats and semantics. Examples requiring unification are units and names; for example, while one data source stores one-way delay as delay in microseconds, another may store it as owd in nanoseconds. Through the mapping to a common ontology (type hierarchy), infrastructures themselves can be left untouched. For this reason, MOMENT has created a domain ontology for describing measurement data as well as accompanying metadata. The concept of measurement metadata has already been introduced in earlier projects such as DatCat and MOME. The MOMENT consortium defines the metadata structure in the Web ontology language (OWL).

Service-Oriented Architecture
Due to the diversity of data sources, the MOMENT project uses a service-oriented architecture. This allows the independent development of services, and connection through Web service technologies, fulfilling one of the main requirements of the project: to provide an open interface for new measurement infrastructures and other future data sources. Single data sources provide an interface of their choice, eg through the Simple Open Access Protocol (SOAP), Representational State Transfer (REST), or even direct SQL or FTP access to the data, which is transformed into services by the MOMENT mediator. A service-oriented approach is also used inside the mediator. By connecting single services to a service chain one can reuse existing operations on various data sources. In this context we want to highlight two core services of the MOMENT mediator, the analysis service (ANAS) and the workflow service (WFS), schematically depicted in Figure 2.

Figure 2: MOMENT Simplified Architecture.

The analysis service allows different kinds of data analysis, ranging from simple histograms to complex calculations such as correlating data from different sources, discovering possible intersections and providing options towards more complete results derived from the broader coverage of the global Internet that MOMENT provides. This correlation may result in possible new metrics of interest for the future Internet, deriving from the completeness of both the measurement points and the measurement data.

Another feature of the MOMENT mediator is the workflow service. Due to the volatility of the Internet, measurement data are usually only valid for a limited time, so automatic mechanisms including data collection from different data sources and multi-level analysis are helpful services for complex data analysis. The workflow service implements a scripting service that allows workflows to be predefined, stored and repeatedly executed on the different data sources and services provided by the mediator. Workflows are represented in the business process execution language (BPEL).

Handling Large Data Amounts
Internet measurements often involve large amounts of data, which may cause problems for busy servers or tiny clients. The MOMENT mediator uses continuations in order to divide data streams into manageable pieces, along with a handshake protocol between client and server so that very large data sets can be conveniently handled.

Links:
http://www.fp7-moment.eu
http://miner.salzburgresearch.at
http://www.barttool.org
http://www.ist-lobster.org/
http://lobster.ics.forth.gr/~appmon/public_sensors
http://www.netdimes.org
http://www.etomic.org

Please contact:
Felix Strohmeier
Salzburg Research Forschungsgesellschaft mbH/AARIT, Austria
Tel: +43 662 2288 443
E-Mail: felix.strohmeiersalzburgresearch.at