Scalable Data Management Techniques for High-End Computing Systems

 

Nowadays, there is a great interest on developing High-End Computing Systems (HECS), to deliver high-performance parallel and distributed computing, using cluster platforms and large scale distributed systems compositions, like Grids and Clouds. This interest is specially relevant to provide high end data services due to the advent of data intensive computing and big data problems, that are creating the so-called “data deluge”. However, the increasing number of architectural I/O layers of HEC systems creates a long path from compute nodes to final storage, which hampers the performance due to long latencies and poor aggregated throughput. Additionally, storage I/O optimizations are often applied independently at each system layer, making difficult to perform global optimizations and creating mismatches between different layers.

In this project, we propose to develop new techniques for reducing the bottleneck in data access due to communication networks and the server’s overload, when compositions of large scale distributed systems and supercomputers platforms are used. The main objective of this project is to study and to propose new techniques for improving the input/output scalability in HECS from a holistic point of view, including all aspects related to the Input/Ouput operations, such as programmability of I/O software stack, providing elasticity in the I/O system, enhancing locality in the data path, and providing delegation of data intensive operations to storage nodes to create data-centric architectures. Moreover, the project proposes to create cross‐layer adaptive control mechanisms to enhance data movement and to globally address runtime events such as load, energy consumption, or failures to develop efficient and sustainable systems.

This project holistically looks for solutions to overcome the problems detected at all levels in the I/O stack, addressing major issues and problems detected in the state of the art and current High-End Computing Systems (HECS), and pursuing the following objectives. O1. To explore techniques for improving the programmability of storage I/O software stack through novel abstractions and mechanisms for controlling system-wide data management. Futhermore, to provide an integrated interface for standard data structures which is common to memory stored data structures and disk stored data structures, overcoming the limitations of POSIX-like file access interfaces. O2. To provide new strategies of delegated execution for data intensive applications. To achieve this goal, we will develop solutions based on active storage concept suitable for parallel file systems and cloud based storage. Furthermore, we will explore the use of integrated interfaces for providing distributed storage services that allow the execution of computations in storage infrastructure reducing the effective bandwidth requirements. O3. Improving distributed elastic caching and storage through the dynamic creation of intermediate I/O servers to group I/O operations of applications and to enhance I/O stack performance. O4. To investigate techniques and mechanisms for exposing and exploiting data locality in the storage stack of High-End Computing systems, to improve energy-efficiency, scalability, and performance. O5. To research applications I/O patterns to integrate solutions provided in the project in real-world prototypes to validate the behaviour and feasibility of those solutions.

Publications related to the projects can be seen here: Publications

The following technical reports are results of this research project.

Technical Reports
ID Title  
Working Group 1 Techniques for improving the programmability of I/O software stack Leader: Jesús Carretero Pérez
E1.1 Technical report on Techniques for improving the programmability of I/O software stack Download PDF
E1.2 Report on enhanced and unified I/O interface Download PDF
E1.3 Report on techniques for the programmability of I/O software stack Download PDF
Working Group 2 New strategies for delegated execution in data intensive applications Leader: Félix García Carballeira
E2.1 Technical report on strategies for delegated execution in data intensive applications Download PDF
E2.2 Design of the delegated I/O node architecture Download PDF
E2.3 Report on new services and interfaces Download PDF
E2.4 Report on scheduling and load balancing for delegated I/O nodes Download PDF
Working Group 3 Elasticity techniques for the I/O stack Leader: Jesús Carretero Pérez
E3.1 Technical report on Elasticity techniques for the I/O stack Download PDF
E3.2 Report on the dynamic creation of intermediate I/O servers Download PDF
E3.3 Design of a distributed elastic caching and storage Download PDF
Working Group 4 Techniques to expose and exploit data locality Leader: Jesús Carretero Pérez
E4.1 Technical report on techniques to expose and exploit data locality Download PDF
E4.2 Report on monitoring and analysing the storage I/O data flow Download PDF
E4.3 Report on solutions for exposing and exploiting data locality Download PDF
Working Group 5 Application requirements, use cases and evaluation Leader: Félix García Carballeira
E5.1 Technical report on Application requirements and use cases Download PDF
E5.2 Report on application adadptation and deployment Download PDF
E5.3 Report on evaluation and optimization of the applications Download PDF