The Experts below are selected from a list of 13662 Experts worldwide ranked by ideXlab platform
Yu-chang Chao - One of the best experts on this subject based on the ideXlab platform.
-
Load Rebalancing for Distributed File Systems in Clouds
IEEE Transactions on Parallel and Distributed Systems, 2013Co-Authors: Hung-chang Hsiao, Hsueh-yi Chung, Haiying Shen, Yu-chang ChaoAbstract:Distributed File Systems are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such File Systems, nodes simultaneously serve computing and storage functions; a File is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance in a Distributed File system; that is, the File chunks are not Distributed as uniformly as possible among the nodes. Emerging Distributed File Systems in production Systems strongly depend on a central node for chunk reallocation. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the performance bottleneck and the single point of failure. In this paper, a fully Distributed load rebalancing algorithm is presented to cope with the load imbalance problem. Our algorithm is compared against a centralized approach in a production system and a competing Distributed solution presented in the literature. The simulation results indicate that our proposal is comparable with the existing centralized approach and considerably outperforms the prior Distributed algorithm in terms of load imbalance factor, movement cost, and algorithmic overhead. The performance of our proposal implemented in the Hadoop Distributed File system is further investigated in a cluster environment.
-
The Load Rebalancing Problem in Distributed File Systems
2012 IEEE International Conference on Cluster Computing, 2012Co-Authors: Hsueh-yi Chung, Hung-chang Hsiao, Che-wei Chang, Yu-chang ChaoAbstract:Distributed File Systems (DFS) are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such File Systems, nodes simultaneously serve computing and storage functions, a File is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance, that is, the File chunks are not Distributed as uniformly as possible in the nodes. Although Distributed load balancing algorithms exist in the literature to deal with the load imbalance problem, emerging DFSs in production Systems strongly depend on a central node for chunk reallocation. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the performance bottleneck and the single point of failure. In this paper, we illustrate and define the load rebalancing problem in cloud DFSs. We advocate File Systems in clouds shall incorporate decentralized load rebalancing algorithms to eliminate the performance bottleneck and the single point of failure. Simulation results for a potential Distributed load balancing algorithm are illustrated. The performance of our proposal implemented in the Hadoop Distributed File system is also demonstrated.
Michael Bauer - One of the best experts on this subject based on the ideXlab platform.
-
a data management in a private cloud storage environment utilizing high performance Distributed File Systems
Workshops on Enabling Technologies: Infrastracture for Collaborative Enterprises, 2013Co-Authors: Tiago S Soares, M A R Dantas, Douglas Dyllon Jeronimo De Macedo, Michael BauerAbstract:The new trend in the process of data-intensive management indicates the importance of a Distributed File system for both Internet large scale services and cloud computing environments. I/O latency and application buffering sizes are two of a number of issues that are essential to be analysed on different class of Distributed File Systems. In this paper, it is presented a research work comparing four different high performance Distributed File Systems. Those Systems were employed to support a medical image server application in a private storage environment. Experimental results highlight the importance of an appropriate Distributed File system to provide a differential level of performance considering application specific characteristics.
Guihai Chen - One of the best experts on this subject based on the ideXlab platform.
-
an end to end learning based metadata management approach for Distributed File Systems
IEEE Transactions on Computers, 2021Co-Authors: Yuanning Gao, Xiaofeng Gao, Ruisi Zhang, Guihai ChenAbstract:Current Distributed File Systems are designed to support PB-scale even EB-scale data storage. Metadata service, which manages File attribute information and the global namespace tree, is crucial to system performance. Distributed metadata management, using multiple metadata servers (MDS's) to store metadata, provides effective approaches to alleviate the workload of a single server. However, maintaining good metadata locality and keeping load balancing among MDS's is a nontrivial problem. In this paper, we present the first machine learning based model called DeepHash, which leverages the neural network to learn a locality preserving hashing (LPH) mapping scheme. DeepHash first converts the metadata nodes to feature vectors by the network embedding technology. Due to the absence of training labels, i.e., the hash values of metadata nodes, we design a pair loss function with distinctive characters to train DeepHash, and introduce the sampling strategy to improve the training efficiency. Besides, we propose an efficient algorithm to dynamically balance the workload and adopt the cache model to improve query efficiency. The experiments on the Amazon EC2 platform demonstrate that the DeepHash can preserve the metadata locality meanwhile maintaining a high load balancing, which denotes the effectiveness and efficiency of DeepHash compared with traditional and state-of-the-art schemes.
-
An Efficient Ring-Based Metadata Management Policy for Large-Scale Distributed File Systems
IEEE Transactions on Parallel and Distributed Systems, 2019Co-Authors: Xiaochun Yang, Guihai ChenAbstract:The growing size of modern File system is expected to reach EB-scale. Therefore, an efficient and scalable metadata service is critical to system performance. Distributed metadata management schemes, which use multiple metadata servers (MDS's) to store metadata, provide a highly effective approach to alleviate the workload of a single server. However, it is difficult to maintain good metadata locality and load balancing among MDS's at the same time. In this paper, we propose a novel hashing scheme called AngleCut to partition metadata namespace tree and serve large-scale Distributed storage Systems. AngleCut first uses a locality preserving hashing (LPH) function to project the namespace tree into linear keyspace, i.e., multiple Chord-like rings. Then we design a history-based allocation strategy to adjust the workload of MDS's dynamically. Besides, we propose a two-layer metadata cache mechanism, including server-side cache and client-side cache to provide the two stage access acceleration. Last but not least, we introduce a Distributed metadata processing 2PC Protocol Based on Message Queue (2PC-MQ) to ensure data consistency. In general, our scheme preserves good metadata locality as well as maintains a high load balancing between MDS's. The theoretical proof and extensive experiments on Amazon EC2 demonstrate the superiority of AngleCut over previous literature.
Jiwu Shu - One of the best experts on this subject based on the ideXlab platform.
-
a flattened metadata service for Distributed File Systems
IEEE Transactions on Parallel and Distributed Systems, 2018Co-Authors: Fenlin Liu, Jiwu ShuAbstract:Key-Value stores provide scalable metadata service for Distributed File Systems. However, the metadata's organization itself, which is organized using a directory tree structure, does not fit the key-value access pattern, thereby limiting the performance. To address this issues, we propose a Distributed File system with a flattened and fine-grained division metadata service, LocoMeta, to bridge the performance gap between File system metadata and key-value stores. LocoMeta is designed to bridge the gap between File metadata to key-value store with two techniques. First, LocoMeta flattens the directory content and structure, which organizes File and directory index nodes in a flat space while reversely indexing the directory entries. Second, it exploits a fine-grained division method to improve the key-value access performance. Evaluations show that LocoMeta with eight nodes boosts the metadata throughput by five times, which approaches 93 percent throughput of a single-node key-value store, compared to 18 percent in the state-of-the-art IndexFS.
-
locofs a loosely coupled metadata service for Distributed File Systems
IEEE International Conference on High Performance Computing Data and Analytics, 2017Co-Authors: Jiwu ShuAbstract:Key-Value stores provide scalable metadata service for Distributed File Systems. However, the metadata's organization itself, which is organized using a directory tree structure, does not fit the key-value access pattern, thereby limiting the performance. To address this issue, we propose a Distributed File system with a loosely-coupled metadata service, LocoFS, to bridge the performance gap between File system metadata and key-value stores. LocoFS is designed to decouple the dependencies between different kinds of metadata with two techniques. First, LocoFS decouples the directory content and structure, which organizes File and directory index nodes in a flat space while reversely indexing the directory entries. Second, it decouples the File metadata to further improve the key-value access performance. Evaluations show that LocoFS with eight nodes boosts the metadata throughput by 5 times, which approaches 93% throughput of a single-node key-value store, compared to 18% in the state-of-the-art IndexFS.
-
octopus an rdma enabled Distributed persistent memory File system
USENIX Annual Technical Conference, 2017Co-Authors: Jiwu Shu, Youmin ChenAbstract:Non-volatile memory (NVM) and remote direct memory access (RDMA) provide extremely high performance in storage and network hardware. However, existing Distributed File Systems strictly isolate File system and network layers, and the heavy layered software designs leave high-speed hardware under-exploited. In this paper, we propose an RDMA-enabled Distributed persistent memory File system, Octopus, to redesign File system internal mechanisms by closely coupling NVM and RDMA features. For data operations, Octopus directly accesses a shared persistent memory pool to reduce memory copying overhead, and actively fetches and pushes data all in clients to re-balance the load between the server and network. For metadata operations, Octopus introduces self-identified RPC for immediate notification between File Systems and networking, and an efficient Distributed transaction mechanism for consistency. Evaluations show that Octopus achieves nearly the raw bandwidth for large I/Os and orders of magnitude better performance than existing Distributed File Systems.
Tiago S Soares - One of the best experts on this subject based on the ideXlab platform.
-
a data management in a private cloud storage environment utilizing high performance Distributed File Systems
Workshops on Enabling Technologies: Infrastracture for Collaborative Enterprises, 2013Co-Authors: Tiago S Soares, M A R Dantas, Douglas Dyllon Jeronimo De Macedo, Michael BauerAbstract:The new trend in the process of data-intensive management indicates the importance of a Distributed File system for both Internet large scale services and cloud computing environments. I/O latency and application buffering sizes are two of a number of issues that are essential to be analysed on different class of Distributed File Systems. In this paper, it is presented a research work comparing four different high performance Distributed File Systems. Those Systems were employed to support a medical image server application in a private storage environment. Experimental results highlight the importance of an appropriate Distributed File system to provide a differential level of performance considering application specific characteristics.