Scalable Data Management Techniques for High-End Computing Systems
Nowadays, there is a great interest on developing High-End Computing Systems (HECS), to deliver high-performance parallel and distributed computing, using cluster platforms and large scale distributed systems compositions, like Grids and Clouds. This interest is specially relevant to provide high end data services due to the advent of data intensive computing and big data problems, that are creating the so-called “data deluge”. However, the increasing number of architectural I/O layers of HEC systems creates a long path from compute nodes to final storage, which hampers the performance due to long latencies and poor aggregated throughput. Additionally, storage I/O optimizations are often applied independently at each system layer, making difficult to perform global optimizations and creating mismatches between different layers.
In this project, we propose to develop new techniques for reducing the bottleneck in data access due to communication networks and the server’s overload, when compositions of large scale distributed systems and supercomputers platforms are used. The main objective of this project is to study and to propose new techniques for improving the input/output scalability in HECS from a holistic point of view, including all aspects related to the Input/Ouput operations, such as programmability of I/O software stack, providing elasticity in the I/O system, enhancing locality in the data path, and providing delegation of data intensive operations to storage nodes to create data-centric architectures. Moreover, the project proposes to create cross‐layer adaptive control mechanisms to enhance data movement and to globally address runtime events such as load, energy consumption, or failures to develop efficient and sustainable systems.
This project holistically looks for solutions to overcome the problems detected at all levels in the I/O stack, addressing major issues and problems detected in the state of the art and current High-End Computing Systems (HECS), and pursuing the following objectives. O1. To explore techniques for improving the programmability of storage I/O software stack through novel abstractions and mechanisms for controlling system-wide data management. Futhermore, to provide an integrated interface for standard data structures which is common to memory stored data structures and disk stored data structures, overcoming the limitations of POSIX-like file access interfaces. O2. To provide new strategies of delegated execution for data intensive applications. To achieve this goal, we will develop solutions based on active storage concept suitable for parallel file systems and cloud based storage. Furthermore, we will explore the use of integrated interfaces for providing distributed storage services that allow the execution of computations in storage infrastructure reducing the effective bandwidth requirements. O3. Improving distributed elastic caching and storage through the dynamic creation of intermediate I/O servers to group I/O operations of applications and to enhance I/O stack performance. O4. To investigate techniques and mechanisms for exposing and exploiting data locality in the storage stack of High-End Computing systems, to improve energy-efficiency, scalability, and performance. O5. To research applications I/O patterns to integrate solutions provided in the project in real-world prototypes to validate the behaviour and feasibility of those solutions.
Publications related to the projects can be seen here: Publications
The following technical reports are results of this research project.