The Experts below are selected from a list of 4131 Experts worldwide ranked by ideXlab platform

Sridhar Mahadevan - One of the best experts on this subject based on the ideXlab platform.

  • Efficient and Scalable Metadata Management in EB-Scale File Systems
    IEEE Transactions on Parallel and Distributed Systems, 2014
    Co-Authors: Quanqing Xu, Rajesh Vellore Arumugam, Khai Leong Yong, Sridhar Mahadevan
    Abstract:

    Efficient and scalable distributed Metadata Management is critically important to overall system performance in large-scale distributed file systems, especially in the EB-scale era. Hash-based mapping and subtree partitioning are state-of-the-art distributed Metadata Management schemes. Hash-based mapping evenly distributes workload among Metadata servers, but it eliminates all hierarchical locality of Metadata. Subtree partitioning does not uniformly distribute workload among Metadata servers, and Metadata needs to be migrated to keep the load balanced roughly. Distributed Metadata Management is relatively difficult since it has to guarantee Metadata consistency. Meanwhile, scaling Metadata performance is more complicated than scaling raw I/O performance. The complexity further rises with distributed Metadata. It results in a primary goal that is to improve Metadata Management scalability while paying attention to Metadata consistency. In this paper, we present a ring-based Metadata Management mechanism named Dynamic Ring Online Partitioning (DROP). It can preserve Metadata locality using locality-preserving hashing, keep Metadata consistency, as well as dynamically distribute Metadata among Metadata server cluster to keep load balancing. By conducting performance evaluation through extensive trace-driven simulations and a prototype implementation, experimental results demonstrate the efficiency and scalability of DROP.

  • DROP: Facilitating distributed Metadata Management in EB-scale storage systems
    2013 IEEE 29th Symposium on Mass Storage Systems and Technologies (MSST), 2013
    Co-Authors: Quanqing Xu, Rajesh Vellore Arumugam, Khai Leong Yang, Sridhar Mahadevan
    Abstract:

    Efficient and scalable distributed Metadata Management is critically important to overall system performance in large-scale distributed storage systems, especially in the EB era. Traditional state-of-the-art distributed Metadata Management schemes include hash-based mapping and subtree partitioning. The former evenly distributes workload among Metadata servers, but it eliminates all hierarchical locality of Metadata. It cannot efficiently handle some operations, e.g., renaming or moving a directory that requires Metadata to be migrated among Metadata servers. The latter does not uniformly distribute workload among Metadata servers, and Metadata need to be migrated to keep the load balanced roughly. In this paper, we present a ring-based Metadata Management scheme, called Dynamic Ring Online Partitioning (DROP). It can preserve Metadata locality using locality-preserving hashing, as well as dynamically distribute Metadata among Metadata server cluster to keep load balancing. By conducting performance evaluation, experimental results demonstrate the effectiveness and scalability of DROP.

  • MSST - DROP: Facilitating distributed Metadata Management in EB-scale storage systems
    2013 IEEE 29th Symposium on Mass Storage Systems and Technologies (MSST), 2013
    Co-Authors: Quanqing Xu, Rajesh Vellore Arumugam, Khai Leong Yong, Sridhar Mahadevan
    Abstract:

    Efficient and scalable distributed Metadata Management is critically important to overall system performance in large-scale distributed storage systems, especially in the EB era. Traditional state-of-the-art distributed Metadata Management schemes include hash-based mapping and subtree partitioning. The former evenly distributes workload among Metadata servers, but it eliminates all hierarchical locality of Metadata. It cannot efficiently handle some operations, e.g., renaming or moving a directory that requires Metadata to be migrated among Metadata servers. The latter does not uniformly distribute workload among Metadata servers, and Metadata need to be migrated to keep the load balanced roughly. In this paper, we present a ring-based Metadata Management scheme, called Dynamic Ring Online Partitioning (DROP). It can preserve Metadata locality using locality-preserving hashing, as well as dynamically distribute Metadata among Metadata server cluster to keep load balancing. By conducting performance evaluation, experimental results demonstrate the effectiveness and scalability of DROP.

Quanqing Xu - One of the best experts on this subject based on the ideXlab platform.

  • Efficient and Scalable Metadata Management in EB-Scale File Systems
    IEEE Transactions on Parallel and Distributed Systems, 2014
    Co-Authors: Quanqing Xu, Rajesh Vellore Arumugam, Khai Leong Yong, Sridhar Mahadevan
    Abstract:

    Efficient and scalable distributed Metadata Management is critically important to overall system performance in large-scale distributed file systems, especially in the EB-scale era. Hash-based mapping and subtree partitioning are state-of-the-art distributed Metadata Management schemes. Hash-based mapping evenly distributes workload among Metadata servers, but it eliminates all hierarchical locality of Metadata. Subtree partitioning does not uniformly distribute workload among Metadata servers, and Metadata needs to be migrated to keep the load balanced roughly. Distributed Metadata Management is relatively difficult since it has to guarantee Metadata consistency. Meanwhile, scaling Metadata performance is more complicated than scaling raw I/O performance. The complexity further rises with distributed Metadata. It results in a primary goal that is to improve Metadata Management scalability while paying attention to Metadata consistency. In this paper, we present a ring-based Metadata Management mechanism named Dynamic Ring Online Partitioning (DROP). It can preserve Metadata locality using locality-preserving hashing, keep Metadata consistency, as well as dynamically distribute Metadata among Metadata server cluster to keep load balancing. By conducting performance evaluation through extensive trace-driven simulations and a prototype implementation, experimental results demonstrate the efficiency and scalability of DROP.

  • DROP: Facilitating distributed Metadata Management in EB-scale storage systems
    2013 IEEE 29th Symposium on Mass Storage Systems and Technologies (MSST), 2013
    Co-Authors: Quanqing Xu, Rajesh Vellore Arumugam, Khai Leong Yang, Sridhar Mahadevan
    Abstract:

    Efficient and scalable distributed Metadata Management is critically important to overall system performance in large-scale distributed storage systems, especially in the EB era. Traditional state-of-the-art distributed Metadata Management schemes include hash-based mapping and subtree partitioning. The former evenly distributes workload among Metadata servers, but it eliminates all hierarchical locality of Metadata. It cannot efficiently handle some operations, e.g., renaming or moving a directory that requires Metadata to be migrated among Metadata servers. The latter does not uniformly distribute workload among Metadata servers, and Metadata need to be migrated to keep the load balanced roughly. In this paper, we present a ring-based Metadata Management scheme, called Dynamic Ring Online Partitioning (DROP). It can preserve Metadata locality using locality-preserving hashing, as well as dynamically distribute Metadata among Metadata server cluster to keep load balancing. By conducting performance evaluation, experimental results demonstrate the effectiveness and scalability of DROP.

  • MSST - DROP: Facilitating distributed Metadata Management in EB-scale storage systems
    2013 IEEE 29th Symposium on Mass Storage Systems and Technologies (MSST), 2013
    Co-Authors: Quanqing Xu, Rajesh Vellore Arumugam, Khai Leong Yong, Sridhar Mahadevan
    Abstract:

    Efficient and scalable distributed Metadata Management is critically important to overall system performance in large-scale distributed storage systems, especially in the EB era. Traditional state-of-the-art distributed Metadata Management schemes include hash-based mapping and subtree partitioning. The former evenly distributes workload among Metadata servers, but it eliminates all hierarchical locality of Metadata. It cannot efficiently handle some operations, e.g., renaming or moving a directory that requires Metadata to be migrated among Metadata servers. The latter does not uniformly distribute workload among Metadata servers, and Metadata need to be migrated to keep the load balanced roughly. In this paper, we present a ring-based Metadata Management scheme, called Dynamic Ring Online Partitioning (DROP). It can preserve Metadata locality using locality-preserving hashing, as well as dynamically distribute Metadata among Metadata server cluster to keep load balancing. By conducting performance evaluation, experimental results demonstrate the effectiveness and scalability of DROP.

K.r. Dittrich - One of the best experts on this subject based on the ideXlab platform.

  • Metadata Management for data warehousing: between vision and reality
    Proceedings 2001 International Database Engineering and Applications Symposium, 2001
    Co-Authors: A. Vaduva, K.r. Dittrich
    Abstract:

    Capturing, representing and processing Metadata promises to facilitate the Management, consistent use and understanding of data and thus better support the exploitation of masses of information that is available online today. Despite the increasing interest in Metadata Management, its purpose, requirements and problems are still not clear. This is particularly true in the area of data warehousing. The reasons are multiple. Compared to the past, today's Metadata Management considers a significantly larger spectrum of information (including even certain pieces of programs). Moreover, Metadata are produced by various tools and reside in different sources which need to be integrated in order to ensure consistency and provide uniform access, impact analysis and data tracking. Existing work has only partially covered some of these aspects. The paper summarizes the most important issues of Metadata Management for data warehousing, including the role of Metadata and solved and unsolved problems of the available solutions. The design of an appropriate information model, Metadata integration and advanced user interaction facilities are crucial questions to be answered.

  • IDEAS - Metadata Management for data warehousing: between vision and reality
    Proceedings 2001 International Database Engineering and Applications Symposium, 2001
    Co-Authors: A. Vaduva, K.r. Dittrich
    Abstract:

    Capturing, representing and processing Metadata promises to facilitate the Management, consistent use and understanding of data and thus better support the exploitation of masses of information that is available online today. Despite the increasing interest in Metadata Management, its purpose, requirements and problems are still not clear. This is particularly true in the area of data warehousing. The reasons are multiple. Compared to the past, today's Metadata Management considers a significantly larger spectrum of information (including even certain pieces of programs). Moreover, Metadata are produced by various tools and reside in different sources which need to be integrated in order to ensure consistency and provide uniform access, impact analysis and data tracking. Existing work has only partially covered some of these aspects. The paper summarizes the most important issues of Metadata Management for data warehousing, including the role of Metadata and solved and unsolved problems of the available solutions. The design of an appropriate information model, Metadata integration and advanced user interaction facilities are crucial questions to be answered.

Rajesh Vellore Arumugam - One of the best experts on this subject based on the ideXlab platform.

  • Efficient and Scalable Metadata Management in EB-Scale File Systems
    IEEE Transactions on Parallel and Distributed Systems, 2014
    Co-Authors: Quanqing Xu, Rajesh Vellore Arumugam, Khai Leong Yong, Sridhar Mahadevan
    Abstract:

    Efficient and scalable distributed Metadata Management is critically important to overall system performance in large-scale distributed file systems, especially in the EB-scale era. Hash-based mapping and subtree partitioning are state-of-the-art distributed Metadata Management schemes. Hash-based mapping evenly distributes workload among Metadata servers, but it eliminates all hierarchical locality of Metadata. Subtree partitioning does not uniformly distribute workload among Metadata servers, and Metadata needs to be migrated to keep the load balanced roughly. Distributed Metadata Management is relatively difficult since it has to guarantee Metadata consistency. Meanwhile, scaling Metadata performance is more complicated than scaling raw I/O performance. The complexity further rises with distributed Metadata. It results in a primary goal that is to improve Metadata Management scalability while paying attention to Metadata consistency. In this paper, we present a ring-based Metadata Management mechanism named Dynamic Ring Online Partitioning (DROP). It can preserve Metadata locality using locality-preserving hashing, keep Metadata consistency, as well as dynamically distribute Metadata among Metadata server cluster to keep load balancing. By conducting performance evaluation through extensive trace-driven simulations and a prototype implementation, experimental results demonstrate the efficiency and scalability of DROP.

  • DROP: Facilitating distributed Metadata Management in EB-scale storage systems
    2013 IEEE 29th Symposium on Mass Storage Systems and Technologies (MSST), 2013
    Co-Authors: Quanqing Xu, Rajesh Vellore Arumugam, Khai Leong Yang, Sridhar Mahadevan
    Abstract:

    Efficient and scalable distributed Metadata Management is critically important to overall system performance in large-scale distributed storage systems, especially in the EB era. Traditional state-of-the-art distributed Metadata Management schemes include hash-based mapping and subtree partitioning. The former evenly distributes workload among Metadata servers, but it eliminates all hierarchical locality of Metadata. It cannot efficiently handle some operations, e.g., renaming or moving a directory that requires Metadata to be migrated among Metadata servers. The latter does not uniformly distribute workload among Metadata servers, and Metadata need to be migrated to keep the load balanced roughly. In this paper, we present a ring-based Metadata Management scheme, called Dynamic Ring Online Partitioning (DROP). It can preserve Metadata locality using locality-preserving hashing, as well as dynamically distribute Metadata among Metadata server cluster to keep load balancing. By conducting performance evaluation, experimental results demonstrate the effectiveness and scalability of DROP.

  • MSST - DROP: Facilitating distributed Metadata Management in EB-scale storage systems
    2013 IEEE 29th Symposium on Mass Storage Systems and Technologies (MSST), 2013
    Co-Authors: Quanqing Xu, Rajesh Vellore Arumugam, Khai Leong Yong, Sridhar Mahadevan
    Abstract:

    Efficient and scalable distributed Metadata Management is critically important to overall system performance in large-scale distributed storage systems, especially in the EB era. Traditional state-of-the-art distributed Metadata Management schemes include hash-based mapping and subtree partitioning. The former evenly distributes workload among Metadata servers, but it eliminates all hierarchical locality of Metadata. It cannot efficiently handle some operations, e.g., renaming or moving a directory that requires Metadata to be migrated among Metadata servers. The latter does not uniformly distribute workload among Metadata servers, and Metadata need to be migrated to keep the load balanced roughly. In this paper, we present a ring-based Metadata Management scheme, called Dynamic Ring Online Partitioning (DROP). It can preserve Metadata locality using locality-preserving hashing, as well as dynamically distribute Metadata among Metadata server cluster to keep load balancing. By conducting performance evaluation, experimental results demonstrate the effectiveness and scalability of DROP.

Quincey Koziol - One of the best experts on this subject based on the ideXlab platform.

  • CLUSTER - SoMeta: Scalable Object-Centric Metadata Management for High Performance Computing
    2017 IEEE International Conference on Cluster Computing (CLUSTER), 2017
    Co-Authors: Houjun Tang, Suren Byna, Bin Dong, Quincey Koziol
    Abstract:

    Scientific data sets, which grow rapidly in volume, are often attached with plentiful Metadata, such as their associated experiment or simulation information. Thus, it becomes difficult for them to be utilized and their value is lost over time. Ideally, Metadata should be managed along with its corresponding data by a single storage system, and can be accessed and updated directly. However, existing storage systems in high-performance computing (HPC) environments, such as Lustre parallel file system, still use a static Metadata structure composed of non-extensible and fixed amount of information. The burden of Metadata Management falls upon the end-users and require ad-hoc Metadata Management software to be developed.With the advent of "object-centric" storage systems, there is an opportunity to solve this issue. In this paper, we present SoMeta, a scalable and decentralized Metadata Management approach for object-centric storage in HPC systems. It provides a flat namespace that is dynamically partitioned, a tagging approach to manage Metadata that can be efficiently searched and updated, and a light-weight and fault tolerant Management strategy. In our experiments, SoMeta achieves up to 3.7X speedup over Lustre in performing common Metadata operations, and up to 16X faster than SciDB and MongoDB for advanced Metadata operations, such as adding and searching tags. Additionally, in contrast to existing storage systems, SoMeta offers scalable user-space Metadata Management by allowing users with the capability to specify the number of Metadata servers depending on their workload.

  • SoMeta: Scalable Object-Centric Metadata Management for High Performance Computing
    2017 IEEE International Conference on Cluster Computing (CLUSTER), 2017
    Co-Authors: Houjun Tang, Suren Byna, Bin Dong, Quincey Koziol
    Abstract:

    Scientific data sets, which grow rapidly in volume, are often attached with plentiful Metadata, such as their associated experiment or simulation information. Thus, it becomes difficult for them to be utilized and their value is lost over time. Ideally, Metadata should be managed along with its corresponding data by a single storage system, and can be accessed and updated directly. However, existing storage systems in high-performance computing (HPC) environments, such as Lustre parallel file system, still use a static Metadata structure composed of non-extensible and fixed amount of information. The burden of Metadata Management falls upon the end-users and require ad-hoc Metadata Management software to be developed.With the advent of "object-centric" storage systems, there is an opportunity to solve this issue. In this paper, we present SoMeta, a scalable and decentralized Metadata Management approach for object-centric storage in HPC systems. It provides a flat namespace that is dynamically partitioned, a tagging approach to manage Metadata that can be efficiently searched and updated, and a light-weight and fault tolerant Management strategy. In our experiments, SoMeta achieves up to 3.7X speedup over Lustre in performing common Metadata operations, and up to 16X faster than SciDB and MongoDB for advanced Metadata operations, such as adding and searching tags. Additionally, in contrast to existing storage systems, SoMeta offers scalable user-space Metadata Management by allowing users with the capability to specify the number of Metadata servers depending on their workload.