The Experts below are selected from a list of 3450 Experts worldwide ranked by ideXlab platform
Yuliang Shi - One of the best experts on this subject based on the ideXlab platform.
-
resource and job execution context aware hadoop configuration tuning
International Conference on Cloud Computing, 2020Co-Authors: Xinhe Wang, Jianlin Zhang, Yuliang ShiAbstract:Mapreduce is a programming model, which is widely-used in parallel processing of big data. Its' performance is significantly affected by configuration parameters. However, the huge parameter space and interact of parameters make it impossible to explore all the parameter combinations manually. In this paper, we propose RJHCT, a novel approach to automatically tune the configuration parameters for Mapreduce Applications. We use random forest regression to predict the execution time of single task, and then we predict the job execution time by packing algorithm. Leveraging the prediction model, we use genetic algorithm to search the optimal configuration parameters automatically for a given Mapreduce Application. Experimental results demonstrated that RJHCT improves the performance of Mapreduce Applications by factors of 0.28x compared with the default configuration.
Miguel Correia - One of the best experts on this subject based on the ideXlab platform.
-
medusa an efficient cloud fault tolerant Mapreduce
IEEE ACM International Symposium Cluster Cloud and Grid Computing, 2016Co-Authors: Pedro Costa, Xiao Bai, Fernando M V Ramos, Miguel CorreiaAbstract:Applications such as web search and social networking have been moving from centralized to decentralized cloud architectures to improve their scalability. Mapreduce, a programming framework for processing large amounts of data using thousands of machines in a single cloud, also needs to be scaled out to multiple clouds to adapt to this evolution. The challenge of building a multi-cloud distributed architecture is substantial. Notwithstanding, the ability to deal with the new types of faults introduced by such setting, such as the outage of a whole datacenter or an arbitrary fault caused by a malicious cloud insider, increases the endeavor considerably. In this paper we propose Medusa, a platform that allows Mapreduce computations to scale out to multiple clouds and tolerate several types of faults. Our solution fulfills four objectives. First, it is transparent to the user, who writes her typical Mapreduce Application without modification. Second, it does not require any modification to the widely used Hadoop framework. Third, the proposed system goes well beyond the fault-tolerance offered by Mapreduce to tolerate arbitrary faults, cloud outages, and even malicious faults caused by corrupt cloud insiders. Fourth, it achieves this increased level of fault tolerance at reasonable cost. We performed an extensive experimental evaluation in the ExoGENI testbed, demonstrating that our solution significantly reduces execution time when compared to traditional methods that achieve the same level of resilience.
Xinhe Wang - One of the best experts on this subject based on the ideXlab platform.
-
resource and job execution context aware hadoop configuration tuning
International Conference on Cloud Computing, 2020Co-Authors: Xinhe Wang, Jianlin Zhang, Yuliang ShiAbstract:Mapreduce is a programming model, which is widely-used in parallel processing of big data. Its' performance is significantly affected by configuration parameters. However, the huge parameter space and interact of parameters make it impossible to explore all the parameter combinations manually. In this paper, we propose RJHCT, a novel approach to automatically tune the configuration parameters for Mapreduce Applications. We use random forest regression to predict the execution time of single task, and then we predict the job execution time by packing algorithm. Leveraging the prediction model, we use genetic algorithm to search the optimal configuration parameters automatically for a given Mapreduce Application. Experimental results demonstrated that RJHCT improves the performance of Mapreduce Applications by factors of 0.28x compared with the default configuration.
Ian Gorton - One of the best experts on this subject based on the ideXlab platform.
-
model driven performance simulation of cloud provisioned hadoop Mapreduce Applications
2016 IEEE ACM 8th International Workshop on Modeling in Software Engineering (MiSE), 2016Co-Authors: Hanieh Alipour, Abdelwahab Hamoulhadj, Ian GortonAbstract:Hadoop is a widely adopted open source implementation of Mapreduce. A Hadoop cluster can be fully provisioned by a Cloud service provider to provide elasticity in computational resource allocation. Understanding the performance characteristics of a Hadoop job can help achieve an optimal balance between resource usage (cost) and job latency on a cloud-based cluster. This paper presents a method that estimates the performance of a Mapreduce Application in a Cloud provisioned Hadoop cluster. We develop a model-driven approach that models a cloud provided independent Hadoop Mapreduce model and customizes it for a specific Cloud deployment. These models are further transformed into a simulation model that produces estimations of end-to-end job latency. We explore this method in the design space of Mapreduce Applications to estimate the performance for different sizes of input data. Our approach provides a model-to-simulation-to-prediction method for observing the performance behaviour of Mapreduce Applications given a configuration of a Mapreduce platform.
Pedro Costa - One of the best experts on this subject based on the ideXlab platform.
-
medusa an efficient cloud fault tolerant Mapreduce
IEEE ACM International Symposium Cluster Cloud and Grid Computing, 2016Co-Authors: Pedro Costa, Xiao Bai, Fernando M V Ramos, Miguel CorreiaAbstract:Applications such as web search and social networking have been moving from centralized to decentralized cloud architectures to improve their scalability. Mapreduce, a programming framework for processing large amounts of data using thousands of machines in a single cloud, also needs to be scaled out to multiple clouds to adapt to this evolution. The challenge of building a multi-cloud distributed architecture is substantial. Notwithstanding, the ability to deal with the new types of faults introduced by such setting, such as the outage of a whole datacenter or an arbitrary fault caused by a malicious cloud insider, increases the endeavor considerably. In this paper we propose Medusa, a platform that allows Mapreduce computations to scale out to multiple clouds and tolerate several types of faults. Our solution fulfills four objectives. First, it is transparent to the user, who writes her typical Mapreduce Application without modification. Second, it does not require any modification to the widely used Hadoop framework. Third, the proposed system goes well beyond the fault-tolerance offered by Mapreduce to tolerate arbitrary faults, cloud outages, and even malicious faults caused by corrupt cloud insiders. Fourth, it achieves this increased level of fault tolerance at reasonable cost. We performed an extensive experimental evaluation in the ExoGENI testbed, demonstrating that our solution significantly reduces execution time when compared to traditional methods that achieve the same level of resilience.