The Experts below are selected from a list of 4536 Experts worldwide ranked by ideXlab platform
Karl Hahn - One of the best experts on this subject based on the ideXlab platform.
-
Hierarchical Storage Support and Management for Large-Scale Multidimensional Array Database Management Systems
2010Co-Authors: Bernd Reiner, Gabriele Höfling, Karl Hahn, Peter BaumannAbstract:Large-scale scientific experiments or simulation programs often generate large amounts of multidimensional data. Data volume may reach hundreds of terabytes (up to petabytes). In the present and the near future, the only practicable way for storing such large volumes of multidimensional data are Tertiary Storage systems. But commercial (multidimensional) database systems are optimized for performance with primary and secondary memory access. So Tertiary Storage memory is only in an insufficient way supported for storing or retrieval of multidimensional array data. To combine the advantages of both techniques, storing large amounts of data on Tertiary Storage media and optimizing data access for retrieval with multidimensional database management systems is the intention of this paper. We introduce concepts for efficient hierarchical Storage support and management for large-scale multidimensional array database management systems and their integration into the commercial array database management system RasDaMan.
-
HEAVEN: A hierarchical Storage and archive environment for multidimensional array database management systems
Lecture Notes in Computer Science, 2004Co-Authors: Bernd Reiner, Karl HahnAbstract:The intention of this paper is to present HEAVEN, a solution of intelligent management of large-scale datasets held on Tertiary Storage systems. We introduce the common state of the art technique Storage and retrieval of large spatio-temporal array data in the High Performance Computing (HPC) area An identified major bottleneck today is fast and efficient access to and evaluation of high performance computing results. We address the necessity of developing techniques for efficient retrieval of requested subsets of large datasets from mass Storage devices. Furthermore, we show the benefit of managing large spatio-temporal data sets, e.g. generated by simulations of climate models, with Database Management Systems (DMBS). This means DBMS need a smart connection to Tertiary Storage systems with optimized access strategies. HEAVEN is based on the multidimensional array DBMS RasDaMan.
-
Optimized management of large-scale data sets stored on Tertiary Storage systems
IEEE Distributed Systems Online, 2004Co-Authors: Bernd Reiner, Karl HahnAbstract:We focus on extending RasDaMan's multidimensional query language using a new concept called object framing. With this extension, users no longer are restricted to performing range queries in the shape of multidimensional hypercubes. It can be able to formulate range queries by indicating complex frames. Thus, object framing represents a generalization of geometric operations. A new system, optimized toward high-performance computing, extends the RasDaMan (Raster Data Management) database management system to allow flexible management of multidimensional spatiotemporal data and to reduce Tertiary Storage access time.
-
Tertiary Storage support for large scale multidimensional array database management systems
2002Co-Authors: Bernd Reiner, Karl HahnAbstract:Many large-scale scientific domains often generate huge amounts (hundreds of terabytes) of multidimensional data. The only practicable way for storing such large volumes of multidimensional data is a Tertiary Storage system. Unfortunately in commercial multidimensional Database Management Systems (DBMS) the access is optimized for performance with primary and secondary memory. Tertiary Storage memory is not or only in an insufficient way supported for storing or retrieval of multidimensional array data. The intention of this paper is, to combine the advantage of both techniques, storing large amounts of data on Tertiary Storage media and realizing efficient data access for retrieval with the commercial multidimensional array DBMS RasDaMan.
Bernd Reiner - One of the best experts on this subject based on the ideXlab platform.
-
Hierarchical Storage Support and Management for Large-Scale Multidimensional Array Database Management Systems
2010Co-Authors: Bernd Reiner, Gabriele Höfling, Karl Hahn, Peter BaumannAbstract:Large-scale scientific experiments or simulation programs often generate large amounts of multidimensional data. Data volume may reach hundreds of terabytes (up to petabytes). In the present and the near future, the only practicable way for storing such large volumes of multidimensional data are Tertiary Storage systems. But commercial (multidimensional) database systems are optimized for performance with primary and secondary memory access. So Tertiary Storage memory is only in an insufficient way supported for storing or retrieval of multidimensional array data. To combine the advantages of both techniques, storing large amounts of data on Tertiary Storage media and optimizing data access for retrieval with multidimensional database management systems is the intention of this paper. We introduce concepts for efficient hierarchical Storage support and management for large-scale multidimensional array database management systems and their integration into the commercial array database management system RasDaMan.
-
HEAVEN: A hierarchical Storage and archive environment for multidimensional array database management systems
Lecture Notes in Computer Science, 2004Co-Authors: Bernd Reiner, Karl HahnAbstract:The intention of this paper is to present HEAVEN, a solution of intelligent management of large-scale datasets held on Tertiary Storage systems. We introduce the common state of the art technique Storage and retrieval of large spatio-temporal array data in the High Performance Computing (HPC) area An identified major bottleneck today is fast and efficient access to and evaluation of high performance computing results. We address the necessity of developing techniques for efficient retrieval of requested subsets of large datasets from mass Storage devices. Furthermore, we show the benefit of managing large spatio-temporal data sets, e.g. generated by simulations of climate models, with Database Management Systems (DMBS). This means DBMS need a smart connection to Tertiary Storage systems with optimized access strategies. HEAVEN is based on the multidimensional array DBMS RasDaMan.
-
Optimized management of large-scale data sets stored on Tertiary Storage systems
IEEE Distributed Systems Online, 2004Co-Authors: Bernd Reiner, Karl HahnAbstract:We focus on extending RasDaMan's multidimensional query language using a new concept called object framing. With this extension, users no longer are restricted to performing range queries in the shape of multidimensional hypercubes. It can be able to formulate range queries by indicating complex frames. Thus, object framing represents a generalization of geometric operations. A new system, optimized toward high-performance computing, extends the RasDaMan (Raster Data Management) database management system to allow flexible management of multidimensional spatiotemporal data and to reduce Tertiary Storage access time.
-
Tertiary Storage support for large scale multidimensional array database management systems
2002Co-Authors: Bernd Reiner, Karl HahnAbstract:Many large-scale scientific domains often generate huge amounts (hundreds of terabytes) of multidimensional data. The only practicable way for storing such large volumes of multidimensional data is a Tertiary Storage system. Unfortunately in commercial multidimensional Database Management Systems (DBMS) the access is optimized for performance with primary and secondary memory. Tertiary Storage memory is not or only in an insufficient way supported for storing or retrieval of multidimensional array data. The intention of this paper is, to combine the advantage of both techniques, storing large amounts of data on Tertiary Storage media and realizing efficient data access for retrieval with the commercial multidimensional array DBMS RasDaMan.
Yanqiu Zhang - One of the best experts on this subject based on the ideXlab platform.
-
non blocking disk tape join algorithm for data on Tertiary Storage systems
Computer and Information Technology, 2005Co-Authors: Baoliang Liu, Lei Nie, Yanqiu ZhangAbstract:Massive data accumulated by business or scientific applications have reached such a great amount that they can be accommodated only on tapes. In order to make full use of these data, tools of data analysis and data mining should be developed. In such applications, disk resident data are needed to join with tape resident data. Many disk-tape join methods have been proposed, examples are CDT-NB and CDT-GH. But all these algorithms have blocking behaviour that user must wait quite a while before the first result can be seen. Since disk-tape join operation often takes a long time to finish, it is desirable to produce the join result as early as possible while the join performance doesn't deteriorate too much. The non-blocking disk-tape join (NDT) presented in this paper is the first disk-tape join algorithm designed with this goal in mind. It has three phases: the hashing phase, the merging phase and the probing phase. Join results can be produced in each phase. Tuples of disk resident relation and tape resident relation are read simultaneously into memory and be joined in the hashing phase. The merging phase joins those tuples that flushed onto disk during the hashing phase. After the first two phases, disk resident relation has been partitioned and is joined with remaining tape resident relation in the probing phase. Experimental results show that NDT can produce join results much earlier than the-state-of-art CDT-GH and the performance of NDT is about the same with that of CDT-GH
-
optimal data dispatching methods in near line Tertiary Storage system
Web-Age Information Management, 2004Co-Authors: Baoliang Liu, Yanqiu ZhangAbstract:Many applications like digital libraries use Near-line Tertiary Storage Systems (TSS) to store their massive data. TSS consists of main memory, disks and Tertiary devices. Typically highly referenced or recent data are stored on disks and historical data are stored on Tertiary Storage devices. We call it Data Dispatching: the determination of what kind of data should be stored on disks and what kind of data should be stored on Tertiary Storage devices. Traditionally it was up to the Database Management System (DBMS) administrator to dispatch data by hand. But DBMS has to take the responsibility if we want to bring Tertiary Storage devices under the control of DBMS. We proved in this paper that the data dispatching is an optimal problem and can be reduced to the famous binary knapsack problem. Experimental results showed that the average response time of TSS could be decreased by using optimal data dispatch method.
-
dynamical schedule algorithms based on markov model in Tertiary Storage
Web-Age Information Management, 2004Co-Authors: Yanqiu Zhang, Zhaogong Zhang, Baoliang LiuAbstract:Tertiary Storages, such as tape libraries and optical disc libraries, are becoming the important Storage devices in massive data applications. With the massive Storage space they have, the Tertiary Storages have very low access efficiency. Designing good schedule algorithms is an important method to improve the access efficiency in Tertiary Storage. Stochastic Markov model is used for predicting the expected number of accesses to the data on Tertiary Storage. Two new schedule algorithms named MarkovMR and MarkovSRF are given also. Generally, in a tape library, there is a robot arm, a few of tape drives and a magazine where a lot of tapes located in. When the tapes are kept in tape drives, we call them online tapes. Otherwise, we call the tapes in the magazine offline tapes. Weight factors are used to above schedule algorithms to favor online tapes so that the requests for online tapes are served first before online tapes are ejected. The weighted algorithms are named wMarkovMR and wMarkovSRF. By compared to the Round-robin policy, the experimental results show that the four schedule algorithms based on Markov model have higher efficiency of data access in Tertiary Storage. The efficiency of wMarkovMR is highest among all the algorithms. Furthermore, the optimal factors can be derived from experiments.
-
surrogate join for massive data on Tertiary Storage system
International Database Engineering and Applications Symposium, 2004Co-Authors: Baoliang Liu, Yanqiu ZhangAbstract:In This work surrogate join (SJ) for massive data on Tertiary Storage is presented. The relations to be joined are first split into surrogate relations and nonsurrogate relations. Surrogate relation consists of tuple identifier and join attribute and nonsurrogate relation consists of tuple identifier and nonjoin attributes. Join is first performed on the two surrogate relations and a join result index is produced which consists of the identifiers of the matching tuples of both surrogate relations, then the join result index is merged with both nonsurrogate relations to get final join result. Experimental results show that our method is better than previous ones in performance and scalability. Note that SJ can convert Tertiary join into disk join and one pass scan of both Tertiary resident nonsurrogate relations for most applications.
Baoliang Liu - One of the best experts on this subject based on the ideXlab platform.
-
Online Management of Massive Data on Tertiary Storage: Dream or Reality?
INFOCOMP Journal of Computer Science, 2006Co-Authors: Baoliang Liu, Kimutai KimeliAbstract:Despite the decrease in disk price and the increase in disk Storage capacity, the Storage requirements of many organizations still can’t be met economically using disk system alone. Tertiary Storage offers a lower-cost alternative. Whether it is feasible to manage massive data on Tertiary Storage or not is still a question with no answer. In this paper, we take the join operation, which is one of most common and time consuming database operations, for example to demonstrate the feasibility of online management of massive data on Tertiary Storage. The experimental results show that it is more important to choose the right operation algorithm than to choose the Storage devices in the analysis and mining of massive data. Tertiary Storages can be used to manage massive data as effective as disks and they provide a new solution to the problem of online querying and accommodating massive data.
-
non blocking disk tape join algorithm for data on Tertiary Storage systems
Computer and Information Technology, 2005Co-Authors: Baoliang Liu, Lei Nie, Yanqiu ZhangAbstract:Massive data accumulated by business or scientific applications have reached such a great amount that they can be accommodated only on tapes. In order to make full use of these data, tools of data analysis and data mining should be developed. In such applications, disk resident data are needed to join with tape resident data. Many disk-tape join methods have been proposed, examples are CDT-NB and CDT-GH. But all these algorithms have blocking behaviour that user must wait quite a while before the first result can be seen. Since disk-tape join operation often takes a long time to finish, it is desirable to produce the join result as early as possible while the join performance doesn't deteriorate too much. The non-blocking disk-tape join (NDT) presented in this paper is the first disk-tape join algorithm designed with this goal in mind. It has three phases: the hashing phase, the merging phase and the probing phase. Join results can be produced in each phase. Tuples of disk resident relation and tape resident relation are read simultaneously into memory and be joined in the hashing phase. The merging phase joins those tuples that flushed onto disk during the hashing phase. After the first two phases, disk resident relation has been partitioned and is joined with remaining tape resident relation in the probing phase. Experimental results show that NDT can produce join results much earlier than the-state-of-art CDT-GH and the performance of NDT is about the same with that of CDT-GH
-
optimal data dispatching methods in near line Tertiary Storage system
Web-Age Information Management, 2004Co-Authors: Baoliang Liu, Yanqiu ZhangAbstract:Many applications like digital libraries use Near-line Tertiary Storage Systems (TSS) to store their massive data. TSS consists of main memory, disks and Tertiary devices. Typically highly referenced or recent data are stored on disks and historical data are stored on Tertiary Storage devices. We call it Data Dispatching: the determination of what kind of data should be stored on disks and what kind of data should be stored on Tertiary Storage devices. Traditionally it was up to the Database Management System (DBMS) administrator to dispatch data by hand. But DBMS has to take the responsibility if we want to bring Tertiary Storage devices under the control of DBMS. We proved in this paper that the data dispatching is an optimal problem and can be reduced to the famous binary knapsack problem. Experimental results showed that the average response time of TSS could be decreased by using optimal data dispatch method.
-
dynamical schedule algorithms based on markov model in Tertiary Storage
Web-Age Information Management, 2004Co-Authors: Yanqiu Zhang, Zhaogong Zhang, Baoliang LiuAbstract:Tertiary Storages, such as tape libraries and optical disc libraries, are becoming the important Storage devices in massive data applications. With the massive Storage space they have, the Tertiary Storages have very low access efficiency. Designing good schedule algorithms is an important method to improve the access efficiency in Tertiary Storage. Stochastic Markov model is used for predicting the expected number of accesses to the data on Tertiary Storage. Two new schedule algorithms named MarkovMR and MarkovSRF are given also. Generally, in a tape library, there is a robot arm, a few of tape drives and a magazine where a lot of tapes located in. When the tapes are kept in tape drives, we call them online tapes. Otherwise, we call the tapes in the magazine offline tapes. Weight factors are used to above schedule algorithms to favor online tapes so that the requests for online tapes are served first before online tapes are ejected. The weighted algorithms are named wMarkovMR and wMarkovSRF. By compared to the Round-robin policy, the experimental results show that the four schedule algorithms based on Markov model have higher efficiency of data access in Tertiary Storage. The efficiency of wMarkovMR is highest among all the algorithms. Furthermore, the optimal factors can be derived from experiments.
-
surrogate join for massive data on Tertiary Storage system
International Database Engineering and Applications Symposium, 2004Co-Authors: Baoliang Liu, Yanqiu ZhangAbstract:In This work surrogate join (SJ) for massive data on Tertiary Storage is presented. The relations to be joined are first split into surrogate relations and nonsurrogate relations. Surrogate relation consists of tuple identifier and join attribute and nonsurrogate relation consists of tuple identifier and nonjoin attributes. Join is first performed on the two surrogate relations and a join result index is produced which consists of the identifiers of the matching tuples of both surrogate relations, then the join result index is merged with both nonsurrogate relations to get final join result. Experimental results show that our method is better than previous ones in performance and scalability. Note that SJ can convert Tertiary join into disk join and one pass scan of both Tertiary resident nonsurrogate relations for most applications.
Thomas Papadakis - One of the best experts on this subject based on the ideXlab platform.
-
Continuous Data Block Placement in and Elevation from Tertiary Storage in Hierarchical Storage Servers
Cluster Computing, 2001Co-Authors: Peter Triantafillou, Thomas PapadakisAbstract:Given the cost of memories and the very large Storage and bandwidth requirements of large-scale multimedia databases, hierarchical Storage servers (which consist of disk-based secondary Storage and tape-library-based Tertiary Storage) are becoming increasingly popular. Such server applications rely upon tape libraries to store all media, exploiting their excellent Storage capacity and cost per MB characteristics. They also rely upon disk arrays, exploiting their high bandwidth, to satisfy a very large number of requests. Given typical access patterns and server configurations, the tape drives are fully utilized uploading data for requests that “fall through” to the Tertiary level. Such upload operations consume significant secondary Storage device and bus bandwidth. In addition, with present technology (and trends) the disk array can serve fewer requests to continuous objects than it can store, mainly due to IO and/or backplane bus bandwidth limitations. In this work we address comprehensively the performance of these hierarchical, continuous-media, Storage servers by looking at all three main system resources: the tape drive bandwidth, the secondary-Storage bandwidth, and the host's RAM. We provide techniques which, while fully utilizing the tape drive bandwidth (an expensive resource) they introduce bandwidth savings, which allow the secondary Storage devices to serve more requests and do so without increasing demands for the host's RAM space. Specifically, we consider the issue of elevating continuous data from its permanent place in Tertiary for display purposes. We develop algorithms for sharing the responsibility for the playback between the secondary and Tertiary devices and for placing the blocks of continuous objects on tapes, and show how they achieve the above goals. We study these issues for different commercial tape library products with different bandwidth and tape capacity and in environments with and without the multiplexing of tape libraries.