Book on UCS

Status:  in preparation

Title: Ultrascale Computing Systems

Editors: Jesus Carretero, Emmanuel Jeannot, Albert Zomaya



The ever-increasing presence of digital data and computers requires pushing for new levels of scalability and sustainability of computing systems in order to address the huge data and processing requirements of the near future. Systems need are expected to be two to three orders of magnitude larger than today systems, including systems with unprecedented amounts of heterogeneous hardware, lines of source code, numbers of users, and volumes of data, sustainability is critical to ensure the feasibility of those systems. Ultrascale computing systems (UCS) are a solution. Envisioned as large-scale complex systems joining parallel and distributed computing systems, which can be located at multiple sites and cooperate to provide the required resources and performance to the users, they could extend individual systems to provide the resources needed.


The goal of this book is to present important aspects towards making more sustainable Ultrascale Computing Systems (UCS), following the results of the research work in the COST Action IC 1305 Network for Sustainable Ultrascale Computing (NESUS), held from 2014 to 2018.  UCS needs new sustainable programming and execution models suitable in the context of rapidly changing underlying computing architecture. Emerging programming models that facilitate the task of scaling and extracting performance on continuous evolving platforms, while providing resilience and fault-tolerant mechanisms to tackle the increasing probability of failures throughout the whole software stack, are needed to achieve scale handling (optimal usage of resources, faults), improve programmability, adaptation to rapidly changing underlying computing architecture, data centric programming models, resilience, energy-efficiency.  In this book, we address Ultrascale computing systems, with focus on:

  • Programming models and runtimes
  • Resilience and fault tolerance
  • Data management techniques and life-cycle
  • Energy aware Ultrascale Systems
  • Applications for Ultrascale Systems





Author: Professor Albert Zomaya, Chief Editor of the IET Book Series on Big Data



Authors:  Leonel Sousa, INESC-ID Portugal; Radu Prodan, University of Innsbruck, Austria and Pierre Kuonen, University of Applied Sciences of Western Switzerland


Synopsys: With the spread of the Internet, applications and web-based services, distributed computing infrastructures, local parallel systems and the availability of huge amounts of dispersed data, software dependent systems will be more and more connected, more and more networked, leading to the creation of “supersystems”. The phrase “Ultrascale Computing Systems (UCS)” refers to this type of IT “supersystems”. However, to really speak of UCS we must consider several orders of magnitude increase in the size of data, in the computing power and in the network complexity relative to what exists now. By making explicit the characteristics of UCS, we better understand at which extent these systems are different from what exists today. The challenges that sustainable UCS posed require a change in perspective. We need to replace the satisfaction of requirements using traditional approaches based on rational top down engineering by the orchestration of complex, decentralized systems.  Addressing this challenge as a whole is most probably an un-realistic approach and does not correspond to the characteristics of UCS. Indeed, by definition, UCS cannot be addressed as a whole. These types of systems are too large and complex to be encompassed by a single human, a single team and even a single organization. A more treatable approach is to identify the main domains in which we need to improve our knowledge and technologies in order to provide the necessary means that will make sustainable UCS being a reality.

In this book, we address Ultrascale computing systems, with focus on:

  • Programming models and runtimes
  • Resilience and fault tolerance
  • Data management techniques and life-cycle
  • Energy aware Ultrascale Systems
  • Applications for Ultrascale Systems



Authors: Juan A. Rico-Gallego and Juan C. Diaz-Martin, University of Extremadura, Spain; Alexey L. Lastovetsky and Ravi Reddy, University College Dublin, Ireland; Matthias Janetscheck and Radu Prodan, University of Innsbruck, Austria; Jorge Barbosa,  University of Porto, Portugal;  Ali Shoker, INESC TEC, Portugal; Joao Leitao,  Universidade Nova de Lisboa, Portugal; Peter Van Roy and Christopher Meiklejohn, Université catholique de Louvain, Belgium; Emmanuel Jeannot, INRIA Bordeaux Sud-Ouest , France;  Juan Luis Garcia-Zapata, University of Extremadura, Spain; and Georges DaCosta, Université Paul Sabatier, France.


2.1 Introduction

Synopsys:  UCS need new sustainable programming and execution models suitable in the context of rapidly changing underlying computing architecture. Emerging programming models that facilitate the task of scaling and extracting performance on continuous evolving platforms, while providing resilience and fault-tolerant mechanisms to tackle the increasing probability of failures throughout the whole software stack, are needed to achieve scale handling (optimal usage of resources, faults), improve programmability, adaptation to rapidly changing underlying computing architecture, data centric programming models, resilience, energy-efficiency.  This chapter presents several contributions targeting the former goals.


2.2 Using Performance and Energy Models for Ultrascale Computing Applications

Synopsys: Ultrascale systems, including high performance computing, distributed computing and big data management platforms, will demand a huge investment of heterogeneous computing and communication equipment. Ensuring the availability of current and future social, enterprise and scientific applications with efficient and reliable execution on these platforms remains nowadays an outstanding challenge. Indeed, reducing their power footprint while still achieving high performance has been identified as a leading constraint in this goal. Model-driven design and development of optimal software solutions should play a critical role in that respect. Energy consumption is one of the main limiting factors for designing and deploying Ultrascale Systems. Using monitoring and measurement data, Performance and Energy models contribute to quantify and gain insights into the performance and power consumption effects of system components and their interactions, including both hardware and the full software stack. Analysis of the information provided by the models is then used for tuning applications and predicting its behavior under different conditions, mainly at scale. This section describes methods, facilities and tools for building performance and energy models, with the goal of aiding in the design, development and tuning of data-parallel and task-parallel applications running on complex heterogeneous parallel platforms.


2.3 Impact of Workflow Enactment Modes on Scheduling and Workflow Performance.

Synopsys: Most sophisticated scheduling algorithms require as accurate and as complete as possible information about the workflow execution plan and the current enactment state to deliver optimal performance. However, providing that information may incur additional overhead which may worsen the performance. We investigate two different enactment modes, early enactment and late enactment, in an empirical analysis using the Manycore Workflow Runtime Environment (MWRE) to determine the impact of missing information on scheduling performance and to gauge the amount of additional overhead required to provide complete information to the scheduler. Early enactment mode evaluates the enactment state as soon as possible providing the most information, whereas late enactment evaluates the enactment state only when absolutely required for continuing workflow enactment providing less information but also having less overhead than early enactment mode. Experimental results show that early enactment mode provides a slightly better performance on average, but depending on the actual workflow late enactment mode may provide performance that is nearly indistinguishable from early enactment mode.


2.4 Towards General Purpose Computations on the Edge

Synopsys:  Cloud computing has become a prominent and successful computing model. However, emerging application domains, such as the Internet of Things and collaborative mobile applications, impose new challenges that cannot easily be addressed in the cloud computing model due to performance and reliability reasons, resource-constrained devices, and security and privacy issues. Edge computing is one promising alternative that shifts parts of the computational overhead from the data center — focus of cloud computing — closer to the devices, where the data is being generated, i.e., at the edge of the network. This enables applications with: (1) high reliability even if the cloud center is unreachable; (2) reduced delays as computations can execute locally, without the latency penalties caused by the distances data streams need to travel; (3) sustainable transmission through reducing the amount of data to be transmitted to the data center for processing; and (4) enabling users to retain security and privacy control of their data. In this section, we present LightKone, a proposal that enables rich, declarative, general purpose computation at the edge. Contrary to the stand-alone computation of Fog Computing devices, our approach is based on the use of synchronization-free replicated mutable data, namely Conflict-free Replicated DataTypes (CRDTs), combined with robust and efficient hybrid gossip communication primitives. Based on these concepts, we show how to build edge runtime systems that are highly available, robust, and secure, covering the spectrum from “light edge” (i.e., IoT devices, Mobiles, Motes, etc.) through “heavy edge” (i.e., data centers, clusters, servers, etc.).


2.5  Spectral Graph Partitioning for Process Placement on Heterogeneous Platforms

Synopsys:  Current heterogeneous architectures, composed of CPUs and accelerator processors (for instance, GPUs) connected by communication channels of different performance, are mainstream. In these platforms, the application workload distribution between the processes executing the algorithm must be non-uniform, that is, each process has different performance capabilities, and hence, it receives a proportional workload, with the goal of a global workload balance. Finding an optimal partition that balances the load minimizes the communications is computationally expensive. The approximation techniques used to solve this problem have traditionally been used for assignment in homogeneous architectures, with a uniform task model as well. In the following pages, we review the partitioning techniques that have been used for tasks with non-uniform loads, and we study an extension of the spectral method for this problem: to accommodate uneven computational loads in heterogeneous architectures. To find a good process placement is a hard combinatorial optimization problem. The methods to obtain approximate solutions can be roughly categorized as based on physical models, such as diffusion or spectral methods, and those based on algorithmic procedures, such as searching extreme vertexes and shortest paths. Among the physical methods, the spectral one differs because it is nonlocal, that is, while the other physical methods are based on a magnitude associated to each vertex, and that evolves in the computation by effects along adjacent vertexes, the spectral method is based in an eigenvector, which is a magnitude distributed because it is computed with non-local operations, involving non adjacent vertexes. For our purposes, the main disadvantage of the spectral method is that the vertex load is not embodied in the Laplacian. As we want to consider mappings from application graph to machine graph, the loads should be included in the model. We do it by generalizing the Laplacian, defining the Laplacian with potential. In this way we can partition with vertex loads as other approaches, and besides to have the loads inside the model.   We implemented the spectral partition with Laplacians with potential as a module in the Scotch partitioning software, and do some preliminary comparisons over several graph databases, as Walshaw, SNAP or SuiteSparse Matrix Collection. The collections of graph data sets and benchmarks come mainly from meshes of finite element method, or from social networks. In addition to this standard data sets, we enrich them with other types of graphs, from different distributed applications (for example, those arising from map-reduce schemes).


Keywords: Performance Modelling, Energy Modelling, Heterogeneous Platforms, Optimization Techniques, Cloud Computing; Edge Computing; Process Placement; Graph Partition;  Sustainability; Energy awareness; Availability; Scalability.




Authors: Pascal Bouvry, Sebastien Varrette and Muhammad Umer Wassim, University of Luxembourg, Luxembourg; Tuan Anh Trinh, Budapest University of Technology and Economics, Hungary; Szabina Fodor and Gergely Farkas, Corvinus University of Budapest, Hungary;  Sergio Iserte, Rafael Mayo and Enrique S. Quintana-Ortí, Universitat Jaume I, Spain; Antonio J. Peña, Barcelona Supercomputing Center, Spain; Kiril Dichev and Dimitrios Nikolopoulos, Queen’s University of Belfast, UK.


3.1 Introduction

Synopsys:  As discussed in the Introduction, Ultrascale computing is a new computing paradigm that comes naturally from the necessity of computing systems that should be able to handle massive data in possibly very large scale distributed systems, enabling new forms of applications that can serve a very large amount of users and in a timely manner that we have never experienced before. It is very challenging to find sustainable solutions for UCS due to their scale and a wide range of possible applications and involved technologies. For example, we need to deal with cross fertilization among HPC, largescale distributed systems, and big data management. One of the challenges regarding sustainable UCS is resilience. Traditionally, it has been an important aspect in the area of critical infrastructure protection (e.g. the traditional electrical grid and the smart grids). Furthermore, it has also become popular in the area of information and communication technology (ICT), ICT systems, computing and large-scale distributed systems. The existing practices of dependable design deal reasonably well with achieving and predicting dependability in systems that are relatively closed and unchanging. Yet, the tendency to make all kinds of large-scale systems more interconnected, open, and able to change without new intervention by designers, makes existing techniques inadequate to deliver the same levels of dependability.  For instance, evolution of the system itself and its uses impairs dependability: new components ”create” system design faults or vulnerabilities by feature interaction or  by triggering pre-existing bugs in existing components; likewise, new patterns of use  arise, new interconnections open the system to attack by new potential adversaries,  and so on.


3.2 Security and reliability in Ultra-scale System

Synopsys: There has been a convergence of the different technologies in terms of HPC, Cloud, Big Data in order to face new challenges in terms of large scale computing, simulation and data analytics. This convergence is foreseen to be extended in order to take over the opportunities offered by the Internet of Things (IoT) including mobile networks and large scale sensor networks such as those systems deployed in smart grids. Many of the security and resilience challenges opened in IoT area are shared by the Ultra-scale domain including secure data management, trust and privacy. It is thus requested to anticipate the upcoming applications enabled by this new global system. The communications versus computation challenges and the related network complexity raised by such new platforms request more distributed solutions such as Fog computing, Software-Defined-Networks (SDNs), etc. In terms of security, we will need more distributed solutions for trust management and more efficient and extended homomorphic encryption primitives. The new systems should be able to handle partial inconsistencies by using meta-heuristic/probabilistic approaches. In particular, to sustain the expected scalability and dynamics of these systems, the foreseen solutions would probably have to rely on unstructured Peer-to-Peer (P2P) approaches, where the supported overlay network relies on random and self-organized topologies to handle group membership and communications in an automated and distributed way. The self-healing properties and proven resilience against the most important types of faults met in such large-scale systems (namely crash faults and cheating fault) render these approaches ideal candidates to sustain the development of robust systems able to handle Secure operation and resilience of Ultra-scale systems.


3.3 Regulation Compliance in Ultra-scale System

SynopsysToday’s law driven societies based on the use of smart contracts (blockchains) are transforming the way economies function around the globe. Ultra-scale systems that refer to an increase in magnitude of computing power and storage by two to three folds are the underlying technical infrastructures in these societies. Thus, the reliability of ultra-scale systems is of paramount importance. Regulations compliance is the common model used to ensure desired level of reliability in any system. In this research the goal is to design and implement a bot i.e. software agent, that can monitor performance of Ultra-scale systems as per pre-define regulations and penalize the system in case of a noncompliance. In this regards, this research proposes NESUS Watchdog (a bot). The underlying concept in the bot is built upon the notion of factor analysis and stochastic modeling from the discipline of Data Science. High performance computing (HPC) cluster at University of Luxembourg (HPC @ and docker (a software container platform) were used to emulate contractual environment in Ultra-scale systems of three systems: Redis, MongoDB, and Memcached Servers. The results showed failure of Redis and MongoDB Servers to act in accordance with pre-defined regulations. Technically, this difference could be attributed to the fact that Memcached is simply used for caching and therefore, it is less prone to noncompliance. Whereas, Redis and MongoDB as databases and message brokers are performing more complex operations and are more likely to be in discord with pre-defined regulations. This research will benefit enterprises that view regulation compliance as a driving factor for implementation of reliability in Ultra-scale systems.


3.4 A study of the job malleability in large HPC facilities and its future implications in fault tolerance solutions

SynopsisIn this section we describe a methodology for dynamic job reconfiguration driven by the programming model runtime in collaboration with the global resource manager. We improve the system throughput by exploiting malleability techniques (in terms of number of MPI ranks) through the reallocation of resources assigned to a job during its execution. In our proposal, the OmpSs runtime reconfigures the number of MPI ranks during the execution of an application in cooperation with the Slurm workload manager. In addition, we take advantage of OmpSs offload semantics to allow application developers deal with data redistribution. By combining these elements a job is able to expand itself in order to exploit idle nodes or be shrunk if other queued jobs could be initiated. This novel approach adapts the system workload in order to increase the throughput as well as make a smarter use of the underlying resources. Our experiments demonstrate that this approach can reduce the total execution time of a practical workload by more than 40% while reducing the amount of resources by 30%. We conclude the report with a discussion of the integration of fault tolerance mechanisms in the malleability framework developed.


Keywords: Security, resilience, fault tolerance, compliance, AI, blockchain, Job dynamic reconfiguration, Resource dynamic reallocation, Resilience, Ultra-scale.



Authors:  Jesus Carretero, University Carlos III of Madrid, Spain; Pilar Gonzalez-Ferez, Universidad de Murcia, Spain; Anastasios Papagiannis, Giorgos Saloustros and Angelos Bilas, FORTH-ICS, Greece;   Fabrizio Marozzo, Javier Garcia-Blas, Paolo Trunfio and Domenico Talia, University of Calabria, Italy; Ali Shoker, INESC TEC, Portugal; Toni Cortes, Barcelona Supercomputing Center, Spain


4.1 Introduction

UCS requires to provide a data management lifecycle on scalable architectures combining HPC and distributed computing in a synergistic approach, as future scalable systems will require sustainable data management for addressing the predicted exponential growth and complexity of digital information. Thus we need to explore and re-think the relationship between the data management lifecycle and scalable architectures in order to pave the way towards reaching sustainable ultrascale on the next generation of computing platforms.


4.2 Analysis of Intra-node Scaling of an Efficient Key-value Store on Modern Multicore Servers

Synopsys: Tucana is an in-house key-value store designed for fast storage devices, such as SSDs, that reduces the gap between sequential and random I/O performance, especially under high degree of concurrency and relatively large I/Os (a few tens of KB). It supports variable size keys and values, versions, arbitrary data set sizes, concurrency, multi-threading, and persistence and recovery from failures. Tucana uses a Bε–tree approach that only performs buffering and batching at the lowest part of the tree, since we assume that the largest part of the tree (but not the data items) fit in memory. Our previous results show that Tucana is up to 9.2× more efficient in terms of CPU cycles/op for in-memory workloads and up to 7× for workloads that do not fit in memory compared to RocksDB. In this paper, we analyze in detail the host CPU overheads, the network and I/O traffic of Tucana. We identify several limitations to achieve higher performance. Then we focus on the client-server network protocol and we use RDMA to design and implement an efficient communication layer between clients and servers. We discuss design choices and we present a preliminary evaluation of our approach.


4.3 Data-centric workflow runtime for data-intensive applications on cloud systems

Synopsis:  As data intensive scientific computing systems become more widespread, there is a necessity of simplifying the development, deployment, and execution of complex data analysis applications for scientific discovery. The scientific workflow model is the leading approach for designing and executing data-intensive applications in high-performance computing infrastructures. Commonly, scientific workflows are built by a set of connected tasks arranged in a directed acyclic graph style, which communicate through storage abstractions. The Data Mining Cloud Framework (DMCF) is a system allowing users to design and execute data analysis workflows on cloud platforms, relying on cloud storage services for every I/O operation. Hercules is an in-memory I/O solution that can be used in DMCF as an alternative to cloud storage services, providing additional performance and flexibility features. This work improves the integration between DMCF and Hercules by using a data-aware scheduling strategy for exploiting data locality in data-intensive workflows.  This section presents also experimental results demonstrating the performance improvements achieved using the proposed data-aware scheduling strategy in the Microsoft Azure cloud platform.


4.4  Conflict-free Replicated DataTypes models

Synopsis:  Driven by the CAP theorem, Available-Partition-Tolerant (AP) systems are usually advocated to avoid the strong consistency (C) overhead in geo-scalable settings (given that network-partitions cannot be generally avoided). In this model, replicas are allowed to serve client’s requests without prior synchronization—usually done in the background. This may obviously lead to conflicts that must be handled in a deterministic and automatic manner. Conflict-free Replicated DataTypes (CRDT) are fully-replicated data types where conflict-resolution is embedded by design—following well-thought mathematical formalizations. CRDTs are recently widely adopted in industry, few examples are Facebook, TomTom, Redis, Riak Basho DB, League of Legends, etc. CRDT models are either operation-based (op-based) or state-based. In the former, commutative (maybe by design) operations are disseminated assuming an underlying reliable causal middleware, whereas in the latter, a complete state is usually disseminated and merged to other states following semi-lattices merging techniques. We presented substantially improved extensions of the two models called Pure CRDTs and Delta CRDTs. In the first, operations are sent without any ordering meta-data, which is leveraged from the underlying middle-ware itself. In Delta CRDTs, we reduced propagating entire states—sometimes very large—to simply sending recent “delta” updates made to the state which significantly reduces the dissemination overhead. These recent models are recently being adopted in industry by Akka, Redis, Basho Riak DB, etc., and are also a base for academic development in the H2020 LightKone ( project on edge computing. In this section, we aim at describing the different models, showing the tradeoffs in feasibility and performance.


Keywords:  Availability, Scalability, CRDT, DataClay, Key-value stores, SSDs, memory-mapped I/O, RDMA-based communication, Data-centric, Data-locality, Workflows, Cloud Computing, DMCF, Hercules




Authors:  Laurent Lefevre, INRIA, France; Ariel Oleksiak, Poznan Supercomputing Center, Poland; Vincenzo De Maio, Vienna University of Technology, Austria; Sebastien Lafond, Åbo Akademi University, Finland; Jean-Marc Pierson,  Paul Sabatier University – Toulouse III, France; Radu Prodan, University of Innsbruck, Austria; Enida Sheme and Neki Frasheri, Polytechnic University of Tirana, Albania;  Jean-Marc Pierson, Patricia Stolf and Georges Da Costa, Institut de Recherche en Informatique de Toulouse, France; Joel Guerrero, Instituto Tecnológico de la Laguna, Spain, Benson Muite, University of Tartu, Findland;  Wojciech Piątek and Wojciech Szeliga, Poznan University of Technology, Poland;  Sebastien Varrette, University of Luxembourg, Luxembourg; Anne-Cecile Orgerie, INRIA, France; Pedro Alonso, Universidad Politécnica de Valencia, Spain; Ravi Reddy Manumachu and Alexey Lastovetsky, University College Dublin, Ireland


5.1. Introduction 

Synopsis: Energy consumption is one of the main limiting factor for the design of ultrascale infrastructures. Multi-level hardware and software optimisations must be designed and explored in order to reduce energy consumption for these large scale equipment. This chapter addresses the issue of energy efficiency of Ultrascale Systems in front of other quality metrics. The goal of this chapter is to explore the design of metrics, analysis, frameworks and tools for putting energy awareness and energy efficiency at the next stage. Significant emphasis will be placed on the idea of “energy complexity”, reflecting the synergies between energy efficiency and Quality of Service, resilience and performance, by studying computation power, communication/data sharing power, data access power, algorithm energy consumption, etc.


5.2.Energy modelling and simulation

Synopsis:  Energy consumption has become a significant issue for data centres. For this reason, many researchers currently focus on developing energy aware algorithms to improve their energy efficiency. However, due to the difficulty of employing real data centres’ infrastructure for assessing the effectiveness of energy-aware algorithms, researchers resort on simulation tools. These tools require precise and detailed models for virtualized data centres in order to deliver accurate results. In recent years, many models have been proposed, but most of them do not consider some of the energy impacting components (e.g. CPU, network, storage). In this work, we focus on increasing the accuracy of existing energy prediction models, by providing the research community with a more accurate data centre energy consumption model. First, we focus on energy consumption of physical nodes inside the data centre. Since energy is the integral of power over time, those are the two main factors that are interested by energy modelling. Therefore, first we provide models for the component with the highest power draw inside the physical nodes, like CPU, RAM, I/O and cooling systems. After this, we focus on modelling the parameters affecting the running time of the data centre, such as the data centre’s workload and the way computational resources are managed inside the data centre. Also, we investigate how the use of renewable energy sources has effect on data centre’s energy management and other energy related parameters. Concerning the simulation part, we discuss different types of data centre simulations, such as event-based simulation and statistical simulations, discussing the current state of the art in this area and the solutions that can fit the most the Exascale scenario.


5.3 Evaluation of Renewable Energy Usage for an Exascale Computer

Synopsys: Attention paid to energy efficiency in datacenters has been increasing significantly in the last decade. One of the latest trends on this issue is supplying datacenters with renewable energy, and reducing the carbon footprint of datacenters is a major step towards green computing. Following this trend, more and more datacenters are built in cold climate areas for cheaper cooling and increased energy efficiency of the facilities. However, such geographical locations have a very varying availability of renewable energy. First, we aim to analyze the feasibility of using renewable energies for a datacenter located on the 60-degree latitude north. Based on real weather data, values of renewable energy generation are provided for a whole year. We introduce a new metric called Minimum Percentage Supply, which represents the fraction of the total data center energy consumption that can be covered by renewable energy, produced by 1 wind turbine and 1 m2 solar panel. We also analyze the ratio between number of wind turbines and m2 of solar panels to achieve a desired percentage of renewable energy coverage for a given data center. The energy cost trade-off is also evaluated for different quantities of wind and solar energy sources.  In order to generalize these concepts, we experiment the aforementioned metrics based on data for three different geographical locations. Based on the results, we discuss the competitiveness of constructing datacenters in different locations. Another issue to be addressed is how to effectively manage different energy sources which supply green powered datacenters. The sources are scheduled based on a priority scheme, aiming to maximize the renewable energy utilization, minimize the energy used from the grid and optimize battery usage. Dynamic power capping technique is used to put a threshold on the drawn energy from the grid. The results show that the proposed scheme is efficient and it is a promising direction in the field of the optimization in datacenters using renewable energy.


5.4.  A survey of cooling in data centers


Synopsys: Data centers can require upto several megawatts of power which are dissipated as heat. Dissipating this heat requires careful planning to minimize operating costs. Current methods for cooling in data centers will be surveyed. Methods of simulating heat transport to optimize data center design so that cost is minimized will also be surveyed. At the level of the computer rack the benefits of immersive solutions, phase change based solutions and heat pipes will be reviewed. In addition, strategies for heat re-use and cooling in several European data centers will be reviewed. Attention will be paid to integration with local conditions that allow for free outside cooling or reduction of building heating expenses.


5.5  A full-cost model for estimating the energy consumption of computing infrastructures

Synopsys: There are two realistic scenarios today to access High Performance Computing (HPC) capacities beyond what is available from the desktop systems. One option is to acquire and operate an HPC system, the other option relies on the Cloud Computing (CC) paradigm, and more particularly of the Infrastructure-as-a-Service (IaaS) model which is best suited for HPC workload. In this context, we propose in this paper a Total Cost of Ownership analysis for a medium-size academic HPC facility and we compare it to the Amazon EC2 cost model. The costs of current data centers are mostly driven by their energy consumption specifically by the air conditioning, computing and networking infrastructure. Yet, current cost models are usually static and rarely consider the facilities’ energy consumption per user. Current pay-as-you-go models of Cloud providers allow users to easily know how much their computing will cost. However, this model is not fully transparent as to where the costs come from (e.g., energy). In this work we explore a full-cost model for estimating the energy consumption of computing infrastructures.


5.6.  Heterogeneous computation of matrix products

Synopsys: The work presented here is an experimental study of performance in execution time and energy consumption of matrix multiplications on a heterogeneous server. The server features three different devices: a multicore CPU, an NVIDIA Tesla GPU, and an Intel Xeon Phi coprocessor. Matrix multiplication is one of the most used linear algebra kernels and, consequently, applications that make an intensive use of this operation can greatly benefit from efficient implementations. This is the case of the evaluation of matrix polynomials, a core operation used to calculate many matrix functions, which involve a very large number of products of square matrices. Although there exist many proposals for efficient implementations of matrix multiplications in heterogeneous environments, it is still difficult to find packages providing a matrix multiplication routine that is so easy to use, efficient, and versatile as its homogeneous counterparts. Our approach here is based on a simple implementation using OpenMP sections. We have also devised a functional model for the execution time that has been successfully applied to the evaluation of matrix polynomials of large degree so that it allows to balance the workload and minimizes the runtime cost.


Keywords:  sustainability, energy efficiency metrics, energy models, energy monitoring, energy simulation, energy aware software, energy consumption.




Francisco Almeida and Vincente Blanco, Universidad de la Laguna, Spain; Thomas Rauber,  University of Bayreuth, Germany; Gudula Rünger, Technische Universität Chemnitz, Germany; Maya Neytcheva, Uppsala University, Sweden; Vladimir Voevodin, Lomonosov Moscow State University, Russia; Raimondas Čiegis and Vadimas Starikovičius and Rima Kraiuzienė, Vilnius Gediminas Technical University, Lithuania; Svetozar Margenov, Bulgarian Academy of Sciences, Bulgaria; Neki Frasheri, Polytechnic University of Tirana, Albania; Emanouil Atanassov, Bulgarian Academy of Sciences, Bulgaria; Pablo San Segundo, Polytechnic University of Madrid; Sandor Szabo and Bogdan Zavalnij, University of Pecs, Hungary.


6.1. Introduction 

Many large-scale scientific applications have a need for ultrascale computing due to scientific goals to simulate larger problems within a reasonable time period. However, it is generally ciaeed that applications have to be rewritten substantially in order to reach ultrascale computing dimensions. Accompanying issues in the redesign of application codes are the inclusion of cross-cutting dimensions identified to be crucial for sustainable ultrascale computing like energy, scalability, massive parallelism, or structure data distribution.  This chapter shows several aspects to adapt the former solutions to ultrascale applications, exploring algorithms, applications, and the impact of their translation into Ultrascale Systems.


6.2. Application-specific analytic energy models

Synopsys: Power and energy consumption are increasingly important in the area of high performance computing. The implementation of energy efficient applications is thus a crucial issue, while the performance or throughput of an application execution is still an important goal for most areas. To plan and control performance and energy consumption, it is suitable to have an idea about their behaviour. One method is to use performance and energy models, which can be application independent or include knowledge about the specific application. The model can include the dependence on several parameter, including, the number of threads, frequency scaling, architecture type. In this article, we concentrate on application-specific analytical energy models and provide an overview of the state-of-the-art in this area.


6.3. Holistic approach in analysing performance and scalability for large scale scientific computing applications

Synopsys: The only way to achieve high utilization of the currently available heterogeneous architectures and the ultra- and exascale machines to come is to detect the good machine-algorithm match, to identify the performance bottlenecks and ways to overcome those. To be able to attain the above goal we need monitoring tools that enable the analysis of the efficiency of large scale computer simulations in a holistic way. Results from performance test studies, based on three typical scenarios when developing software solutions for computer simulations are presented. The used monitoring and performance analyzing system tool provides not only information on the execution of a particular application code but also how efficiently that application utilizes the computer resources on different levels in the hardware hierarchy – caches, memory, processor, IO etc. The material includes the obtained results and their analysis, the improvements and a new feature developed as well as suggests aspects to be addressed in future extensions and enhances both in application and system monitoring software.


6.4. On parallel algorithms for the numerical solution of problems with fractional powers of elliptic operators

Synopsys: In recent decades, popularity of fractional-order models is constantly increasing. Fractional-order derivatives and operators are successfully applied in various problems in physics, chemistry, biochemistry, biology, hydrology, image processing and finance. They are able to describe anomalous behaviours of various materials and phenomena. Parallel solvers of problems based on fractional powers of elliptic operators are considered. Due to the non-local nature of the operators, the numerical solution of such problems is a significant computational challenge, especially in the case of 3D domain. We employ four different transformations of the given non-local into some of the following local differential problems: (i) elliptic problem with a degenerate or singular weight; (ii) pseudo-parabolic problem; (iii) integral representation using standard Laplacian operators; (iv) transformation based on best uniform rational approximation. The scalability and efficiency of the developed parallel algorithms are analyzed and compared. The issues of optimal data partitioning and load balancing are addressed.


6.5. Scalability of parallel processing for gravity inversion

Synopsys:  In this section we present results about the performance of a recently updated algorithm for the inversion of gravity anomalies for 3D geosections on HPC systems. Because of the high complexity of the algorithm an ultra-scale parallel computing system will be necessary for complex multi-method engineering works. A relaxation iterative principle was used to update the geosection distribution of mass density step by step. The selection of updates was done on basis of least squares error matching of the update effect with the observed anomaly. Locally weighted least squares combined with the linear trend were used to obtain good inversion results for multi-body geosections. Promising results were obtained by hybrid MPI and OpenMP parallelization.


6.6 Massive parallelization of the maximum clique problem using subchromatic


Synopsys: Determining a maximum clique in a given graph is a well-known NP-hard problem with many important applications in various fields of science and technology. The overall strategy is to divide the original, also master, problem into a large number of smaller subproblems which can be solved independently of each other. In order to achieve a reasonable load balance the number of the problems should be a multiple of the number of computing cores. The partial results of the cores will then be aggregated into the final solution to the maximum clique problem of the original graph. We introduce the concepts of K-clique avoiding vertex set and 􀝇K-clique avoiding edge set. These sets are the pivotal tools to divide a problem into smaller problems; the vertex set is conceptually simpler while the edge set provides a larger number of subproblems. Determining optimal size K-clique avoiding vertex sets is computationally demanding. In this work, we consider different greedy color-based, heuristics to compute suboptimal K-clique avoiding vertex sets, and the ability of these sets to partition the maximum clique problem search space into subproblems of similar difficulty.


Keywords: scalability, programmability, massively parallel computing, graph coloring, fractional-order models, maximum clique problem, large scale scientific computing, energy-aware algorithms, Exascale.



The Book Editors


List of authors




  • Enida Sheme and Neki Frasheri, Polytechnic University of Tirana


  • Radu Prodan, Matthias Janetscheck and Radu Prodan, University of Innsbruck
  • Vincenzo De Maio, Vienna University of Technology


  • Peter Van Roy and Christopher Meiklejohn, Université catholique de Louvain


  • Sebastien Lafond, Åbo Akademi University
  • Benson Muite, University of Tartu


  • Emmanuel Jeannot, Anne-Cecile Orgerie, Laurent Lefevre, INRIA
  • Georges DaCosta, Université Paul Sabatier
  • Jean-Marc Pierson, Paul Sabatier, Patricia Stolf and Georges Da Costa, Institut de Recherche en Informatique de Toulouse


  • Anastasios Papagiannis, Giorgos Saloustros and Angelos Bilas, FORTH-ICS


  • Tuan Anh Trinh, Budapest University of Technology and Economics
  • Szabina Fodor and Gergely Farkas, Corvinus University of Budapest


  • Alexey L. Lastovetsky and Ravi Reddy, University College Dublin
  • Kiril Dichev and Dimitrios Nikolopoulos, Queen’s University of Belfast


  • Fabrizio Marozzo, Paolo Trunfio and Domenico Talia, University of Calabria


  • Pascal Bouvry, Sebastien Varrette and Muhammad Umer Wassim, University of Luxembourg


  • Oleksiak, Poznan Supercomputing Center
  • Wojciech Piątek and Wojciech Szeliga, Poznan University of Technology


  • Leonel Sousa and Ali Shoker, INESC-ID
  • Jorge Barbosa, University of Porto
  • Joao Leitao, Universidade Nova de Lisboa


  • Juan A. Rico-Gallego, Juan C. Diaz-Martin and Juan Luis Garcia-Zapata, University of Extremadura
  • Sergio Iserte, Rafael Mayo and Enrique S. Quintana-Ortí, Universitat Jaume I, Spain
  • Antonio J. Peña and Toni Cortes, Barcelona Supercomputing Center, Spain
  • Jesus Carretero, University Carlos III of Madrid, Spain
  • Javier Garcia-Blas, University Carlos III of Madrid, Spain
  • Pilar Gonzalez-Ferez, Universidad de Murcia, Spain
  • Joel Guerrero, Instituto Tecnológico de la Laguna, Spain
  • Pedro Alonso, Universidad Politécnica de Valencia, Spain


  • Pierre Kuonen, University of Applied Sciences of Western