Following on from previous successful grid related work, the Laboratory of Parallel and Distributed Systems (LPDS) of SZTAKI is now focusing on Grid-Cloud interoperability.
Over the last decade the e-science infrastructure eco-system has been enriched with clouds. Together with supercomputer-based grids, cluster-based grids and desktop grids, clouds now form one of the main pillars of this ecosystem. All pillars have their own advantages that make them attractive for a certain application area, however, some of them can benefit from multiple e-science infrastructure technologies. Cluster- and supercomputer-based grids can be considered as ‘service grids’ since they provide managed cluster and supercomputer resources as services with high availability. The recently emerged cloud systems can be used when fast response times, strict Quality of Service (QoS) and Service Level Agreements (SLA) are required and if (as happens in many cases) resource requirements are higher than those provided by the available grid infrastructure resources. However, the cost of porting complex research applications from one technology to another one is hard to finance. Also, unfortunately, in many cases the pillars are separated from each other and cannot be used simultaneously by the same e-scientist with a large-scale single application.
There are two possible approaches to bind the pillars. The first approach is referred as low-level interoperability which is realized at the task level. Sending tasks directly from one pillar to other one is made possible by using so-called ‘bridging technologies’. The Enabling Desktop Grids for e-Science (EDGeS) FP7 EU project developed a service called 3G-Bridge which allows task based interoperability between service and desktop grids. The recently commenced European Desktop Grid Initiative (EDGI) EU FP7 project continues the work and will develop desktop grid-cloud bridging middleware with the goal to provide instantly available additional resources for desktop grid systems if an application has QoS requirements that cannot be satisfied by the available resources of the system. EDGI will support user communities that are heavy users of e-science infrastructures and require an extremely large number of CPUs and cores. EDGI is coordinated by the Laboratory of Parallel and Distributed Systems of SZTAKI.
Figure 1: The EDGI project goals.
The second approach can be referred as high-level or workflow-level interoperability which allows executing (parts of) workflows on any pillar of the e-science ecosystem. Tasks can be sent to any pillar directly but the low-level interoperability can also be used when required. This type of high interoperability is achieved by using high-level tools like portals (sometimes referred as science gateways). In Europe, one of the most popular generic purpose science gateways is the P-GRADE (Parallel Grid Run-time and Application Development Environment) portal developed by LPDS of SZTAKI. Its basic concepts were mainly developed during the grid era, therefore P-GRADE portal is currently used by many national grids (UK NGS, Belgium Grid, SwissGrid, Turkish/ Hungarian/ Spanish Grids, Grid Ireland, etc.), as well as several regional grids (South-East European/ Baltic/ UK White Rose Grids) and several science specific virtual organizations (Chemistry Grid, Economy Grid, Math Grid, etc.). In recent years P-GRADE portal has also become popular outside Europe: in Grid Malaysia, Grid Kazakhstan, and Armenian Grid. P-GRADE portal is an open source toolset consisting of a service-rich, workflow oriented graphical front-end and a back-end enabling execution in various types of e-science systems (grids, clusters and recently clouds). It hides the complexity of the infrastructure middleware through its high-level graphical web interface, and it can be used to develop, execute and monitor workflow applications on service and desktop grid systems. P-GRADE portal installations typically provide the user with access to several middleware technologies, using a single login.
Originally, P-GRADE portal supported job submission only to service grids. The 3G-Bridge, developed by EDGeS, has been interfaced P-GRADE portal with BOINC and XtremWeb desktop grids, and cloud interoperability is now achieved in three steps:
1. Cloud resource management: allocates on-demand cloud resource for jobs when they arrive and removes the cloud resources when there are no more jobs.
2. Job submission: Submitted jobs are first queued at the 3G-Bridge and then delegated to the allocated cloud resources.
3. Job scheduling on cloud resources: load balances jobs on allocated cloud resources, thus ensures that no resource is idle while others are overwhelmed.
This approach allows high-level interoperability using on-demand cloud resources in a transparent and efficient way with improved scalability. The three steps (resource management, job submission, job scheduling) of interoperability are separate independent functional units. As an attractive advantage, this allows the independent extension of each component to support different cloud middlewares or user communities with special requirements.
Links:
http://portal.p-grade.hu
http://www.sztaki.hu
http://www.lpds.sztaki.hu
http://edgi-project.eu/
http://edges-grid.eu/
Please contact:
Attila Marosi, Miklós Kozlovszky and Péter Kacsuk
SZTAKI, Hungary
E-mail:
