The Experts below are selected from a list of 36 Experts worldwide ranked by ideXlab platform
Winfried K Grassmann - One of the best experts on this subject based on the ideXlab platform.
-
simulation and performance evaluation of the hadoop Capacity Scheduler
Computer Science and Software Engineering, 2014Co-Authors: Jagmohan Chauhan, Dwight Makaroff, Winfried K GrassmannAbstract:Hadoop task Schedulers like Fair Share and Capacity have been specially designed to share hardware resources among multiple organizations. The Capacity Scheduler provides a complex set of parameters to give fine control over resource allocation of a shared MapReduce cluster. Administrators and users often run into performance problems because they do not understand the performance influence of the task Scheduler parameters on MapReduce workloads. Interaction between parameter settings is particularly problematic. In this paper, we implemented a Capacity Scheduler simulator component, integrated it into an existing simulator and then validated the simulator with small test cases, consisting of standard benchmark sort programs. We then studied the impact of Capacity Scheduler parameters on different MapReduce workload submission patterns with a more complex set of benchmark programs. Among other results, we found maxCapacity and minUser-LimitPCT to be influential parameters suggested by previous work and that using separate queues for short and long jobs provides the best performance in terms of response ratio, execution time and makespan compared to submitting both types of jobs in the same queue.
-
the impact of Capacity Scheduler configuration settings on mapreduce jobs
International Conference on Cloud and Green Computing, 2012Co-Authors: Jagmohan Chauhan, Dwight Makaroff, Winfried K GrassmannAbstract:MapReduce is a parallel programming paradigm used for processing huge datasets on certain classes of distributable problems using a cluster. Budgetary constraints and the need for better usage of resources in a MapReduce cluster often influence an organization to rent or share hardware resources for their main data processing and analysis tasks. Thus, there may be many competing jobs from different clients performing simultaneous requests to the MapReduce framework on a particular cluster. Schedulers like Fair Share and Capacity have been specially designed for such purposes. Administrators and users run into performance problems, however, because they do not know the exact meaning of different task Scheduler settings and what impact they can have with respect to the application execution time and resource allocation policy decisions. Existing work shows that the performance of MapReduce jobs depends on the cluster configuration, input data type and job configuration settings. However, that work fails to take into account the task Scheduler settings. We show, through experimental evaluation, that task Scheduler configuration parameters make a significant difference to the performance of the cluster and it is important to understand the influence of such parameters. Based on our findings, we also identified some of the open issues in the existing area of research.
Jagmohan Chauhan - One of the best experts on this subject based on the ideXlab platform.
-
simulation and performance evaluation of the hadoop Capacity Scheduler
Computer Science and Software Engineering, 2014Co-Authors: Jagmohan Chauhan, Dwight Makaroff, Winfried K GrassmannAbstract:Hadoop task Schedulers like Fair Share and Capacity have been specially designed to share hardware resources among multiple organizations. The Capacity Scheduler provides a complex set of parameters to give fine control over resource allocation of a shared MapReduce cluster. Administrators and users often run into performance problems because they do not understand the performance influence of the task Scheduler parameters on MapReduce workloads. Interaction between parameter settings is particularly problematic. In this paper, we implemented a Capacity Scheduler simulator component, integrated it into an existing simulator and then validated the simulator with small test cases, consisting of standard benchmark sort programs. We then studied the impact of Capacity Scheduler parameters on different MapReduce workload submission patterns with a more complex set of benchmark programs. Among other results, we found maxCapacity and minUser-LimitPCT to be influential parameters suggested by previous work and that using separate queues for short and long jobs provides the best performance in terms of response ratio, execution time and makespan compared to submitting both types of jobs in the same queue.
-
simulation and performance evaluation of hadoop Capacity Scheduler
2013Co-Authors: Jagmohan ChauhanAbstract:MapReduce is a parallel programming paradigm used for processing huge datasets on certain classes of distributable problems using a cluster. Budgetary constraints and the need for better usage of resources in a MapReduce cluster often make organizations rent or share hardware resources for their main data processing and analysis tasks. Thus, there may be many competing jobs from different clients performing simultaneous requests to the MapReduce framework on a particular cluster. Schedulers like Fair Share and Capacity have been specially designed for such purposes. Administrators and users run into performance problems, however, because they do not know the exact meaning of different task Scheduler settings and what impact they can have with respect to the resource allocation scheme across organizations for a shared MapReduce cluster. In this work, Capacity Scheduler is integrated into an existing MRPERF simulator to predict the performance of MapReduce jobs in a shared cluster under different settings for Capacity Scheduler. A few case studies on the behaviour of Capacity Scheduler across different job patterns etc. using integrated simulator are also conducted.
-
the impact of Capacity Scheduler configuration settings on mapreduce jobs
International Conference on Cloud and Green Computing, 2012Co-Authors: Jagmohan Chauhan, Dwight Makaroff, Winfried K GrassmannAbstract:MapReduce is a parallel programming paradigm used for processing huge datasets on certain classes of distributable problems using a cluster. Budgetary constraints and the need for better usage of resources in a MapReduce cluster often influence an organization to rent or share hardware resources for their main data processing and analysis tasks. Thus, there may be many competing jobs from different clients performing simultaneous requests to the MapReduce framework on a particular cluster. Schedulers like Fair Share and Capacity have been specially designed for such purposes. Administrators and users run into performance problems, however, because they do not know the exact meaning of different task Scheduler settings and what impact they can have with respect to the application execution time and resource allocation policy decisions. Existing work shows that the performance of MapReduce jobs depends on the cluster configuration, input data type and job configuration settings. However, that work fails to take into account the task Scheduler settings. We show, through experimental evaluation, that task Scheduler configuration parameters make a significant difference to the performance of the cluster and it is important to understand the influence of such parameters. Based on our findings, we also identified some of the open issues in the existing area of research.
Dwight Makaroff - One of the best experts on this subject based on the ideXlab platform.
-
simulation and performance evaluation of the hadoop Capacity Scheduler
Computer Science and Software Engineering, 2014Co-Authors: Jagmohan Chauhan, Dwight Makaroff, Winfried K GrassmannAbstract:Hadoop task Schedulers like Fair Share and Capacity have been specially designed to share hardware resources among multiple organizations. The Capacity Scheduler provides a complex set of parameters to give fine control over resource allocation of a shared MapReduce cluster. Administrators and users often run into performance problems because they do not understand the performance influence of the task Scheduler parameters on MapReduce workloads. Interaction between parameter settings is particularly problematic. In this paper, we implemented a Capacity Scheduler simulator component, integrated it into an existing simulator and then validated the simulator with small test cases, consisting of standard benchmark sort programs. We then studied the impact of Capacity Scheduler parameters on different MapReduce workload submission patterns with a more complex set of benchmark programs. Among other results, we found maxCapacity and minUser-LimitPCT to be influential parameters suggested by previous work and that using separate queues for short and long jobs provides the best performance in terms of response ratio, execution time and makespan compared to submitting both types of jobs in the same queue.
-
the impact of Capacity Scheduler configuration settings on mapreduce jobs
International Conference on Cloud and Green Computing, 2012Co-Authors: Jagmohan Chauhan, Dwight Makaroff, Winfried K GrassmannAbstract:MapReduce is a parallel programming paradigm used for processing huge datasets on certain classes of distributable problems using a cluster. Budgetary constraints and the need for better usage of resources in a MapReduce cluster often influence an organization to rent or share hardware resources for their main data processing and analysis tasks. Thus, there may be many competing jobs from different clients performing simultaneous requests to the MapReduce framework on a particular cluster. Schedulers like Fair Share and Capacity have been specially designed for such purposes. Administrators and users run into performance problems, however, because they do not know the exact meaning of different task Scheduler settings and what impact they can have with respect to the application execution time and resource allocation policy decisions. Existing work shows that the performance of MapReduce jobs depends on the cluster configuration, input data type and job configuration settings. However, that work fails to take into account the task Scheduler settings. We show, through experimental evaluation, that task Scheduler configuration parameters make a significant difference to the performance of the cluster and it is important to understand the influence of such parameters. Based on our findings, we also identified some of the open issues in the existing area of research.
J M Santamaria - One of the best experts on this subject based on the ideXlab platform.
-
lessons learned for building agile and flexible scheduling tool for turbulent environments in the extended enterprise
Robotics and Computer-integrated Manufacturing, 2006Co-Authors: M C Palacios, Esther Alvarez, M Alvarez, J M SantamariaAbstract:Abstract This paper presents results of a 2.5-year, multidisciplinary, university–industry collaborative effort investigating design of “Internet-based Scheduler for material Optimisation and agile ProducTion In MUlti-Site enterprises in agile manufacturing” (IS-OPTIMUS), a four nation collaborative project aimed at improvements in turbulent manufacturing environments. The focus of this paper is specifically on the content of the work carried out, along with the main benefits and results. Key to achieving the goals is following a complete project-life-cycle path from the initial stages where the industrial users requirements were identified and the system specification took place to the development and tuning of the final system. Design choices for software must strike a balance between the user flexibility required and the representing environment constraints, i.e. finite Capacity scheduling, which takes production requirements from existing production planning systems, to schedule production resources like plant workers, critical tools and machines. The system consists of a material optimiser working closely with the finite Capacity Scheduler and a dynamic Scheduler providing automatic reaction to real-time exceptions, thus deriving in solutions of higher performance.
Jeff Markham - One of the best experts on this subject based on the ideXlab platform.
-
apache hadoop yarn moving beyond mapreduce and batch processing with apache hadoop 2
2014Co-Authors: Arun C Murthy, Vinod Kumar Vavilapalli, Doug Eadline, Joseph S Niemiec, Jeff MarkhamAbstract:This book is a critically needed resource for the newly released Apache Hadoop 2.0, highlighting YARN as the significant breakthrough that broadens Hadoop beyond the MapReduce paradigm. From the Foreword by Raymie Stata, CEO of Altiscale The Insiders Guide to Building Distributed, Big Data Applications with Apache Hadoop YARN Apache Hadoop is helping drive the Big Data revolution. Now, its data processing has been completely overhauled: Apache Hadoop YARN provides resource management at data center scale and easier ways to create distributed applications that process petabytes of data. And now in Apache Hadoop YARN, two Hadoop technical leaders show you how to develop new applications and adapt existing code to fully leverage these revolutionary advances. YARN project founder Arun Murthy and project lead Vinod Kumar Vavilapalli demonstrate how YARN increases scalability and cluster utilization, enables new programming models and services, and opens new options beyond Java and batch processing. They walk you through the entire YARN project lifecycle, from installation through deployment. Youll find many examples drawn from the authors cutting-edge experiencefirst as Hadoops earliest developers and implementers at Yahoo! and now as Hortonworks developers moving the platform forward and helping customers succeed with it. Coverage includes YARNs goals, design, architecture, and componentshow it expands the Apache Hadoop ecosystem Exploring YARN on a single node Administering YARN clusters and Capacity Scheduler Running existing MapReduce applications Developing a large-scale clustered YARN application Discovering new open source frameworks that run under YARN
