The Experts below are selected from a list of 297 Experts worldwide ranked by ideXlab platform
Patrick Valduriez - One of the best experts on this subject based on the ideXlab platform.
-
FP-Hadoop: Efficient processing of skewed MapReduce jobs
Information Systems, 2016Co-Authors: Miguel Liroz-gistau, Reza Akbarinia, Divyakant Agrawal, Patrick ValduriezAbstract:Nowadays, we are witnessing the fast production of very large amount of data, particularly by the users of online systems on the Web. However, processing this big data is very challenging since both space and computational requirements are hard to satisfy. One solution for dealing with such requirements is to take advantage of parallel frameworks, such as MapReduce or Spark, that allow to make powerful computing and storage units on top of ordinary machines. Although these key-based frameworks have been praised for their high scalability and fault tolerance, they show poor performance in the case of data skew. There are important cases where a high percentage of processing in the reduce side ends up being done by only one node. In this paper, we present FP-Hadoop, a Hadoop-based system that renders the reduce side of MapReduce more parallel by efficiently tackling the problem of reduce data skew. FP-Hadoop introduces a new phase, denoted intermediate reduce (IR), where blocks of intermediate values are processed by intermediate reduce Workers in parallel. With this approach, even when all intermediate values are associated to the same key, the main part of the reducing work can be performed in parallel taking benefit of the computing power of all Available Workers. We implemented a prototype of FP-Hadoop, and conducted extensive experiments over synthetic and real datasets. We achieved excellent performance gains compared to native Hadoop, e.g. more than 10 times in reduce time and 5 times in total execution time.
-
An Efficient Solution for Processing Skewed MapReduce Jobs
2015Co-Authors: Reza Akbarinia, Miguel Liroz-gistau, Divyakant Agrawal, Patrick ValduriezAbstract:Although MapReduce has been praised for its high scalability and fault tolerance, it has been criticized in some points, in particular, its poor performance in the case of data skew. There are important cases where a high percentage of processing in the reduce side is done by a few nodes, or even one node, while the others remain idle. There have been some attempts to address the problem of data skew, but only for specific cases. In particular, there is no proposed solution for the cases where most of the intermediate values correspond to a single key, or when the number of keys is less than the number of reduce Workers. In this paper, we propose FP-Hadoop, a system that makes the reduce side of MapReduce more parallel, and efficiently deals with the problem of data skew in the reduce side. In FP-Hadoop, there is a new phase, called intermediate reduce (IR), in which blocks of intermediate values, constructed dynamically, are processed by intermediate reduce Workers in parallel, by using a scheduling strategy. By using the IR phase, even if all intermediate values belong to only one key, the main part of the reducing work can be done in parallel by using the computing resources of all Available Workers. We implemented a prototype of FP-Hadoop, and conducted extensive experiments over synthetic and real datasets. We achieved excellent performance gains compared to native Hadoop, e.g. more than 10 times in reduce time and 5 times in total execution time.
-
FP-Hadoop : Efficient Execution of Parallel Jobs Over Skewed Data
Proceedings of the 41st International Conference on Very Large Data Bases, 2015Co-Authors: Miguel Liroz-gistau, Reza Akbarinia, Patrick ValduriezAbstract:Big data parallel frameworks, such as MapReduce or Spark have been praised for their high scalability and performance, but show poor performance in the case of data skew. There are important cases where a high percentage of processing in the reduce side ends up being done by only one node. In this demonstration, we illustrate the use of FP-Hadoop, a system that efficiently deals with data skew in MapReduce jobs. In FP-Hadoop, there is a new phase, called inter-mediate reduce (IR), in which blocks of intermediate val-ues, constructed dynamically, are processed by intermediate reduce Workers in parallel, by using a scheduling strategy. Within the IR phase, even if all intermediate values belong to only one key, the main part of the reducing work can be done in parallel using the computing resources of all avail-able Workers. We implemented a prototype of FP-Hadoop, and conducted extensive experiments over synthetic and real datasets. We achieve excellent performance gains compared to native Hadoop, e.g. more than 10 times in reduce time and 5 times in total execution time. During our demonstration, we give the users the possibil-ity to execute and compare job executions in FP-Hadoop and Hadoop. They can retrieve general information about the job and the tasks and a summary of the phases. They can also visually compare different configurations to explore the difference between the approaches.
Miguel Liroz-gistau - One of the best experts on this subject based on the ideXlab platform.
-
FP-Hadoop: Efficient processing of skewed MapReduce jobs
Information Systems, 2016Co-Authors: Miguel Liroz-gistau, Reza Akbarinia, Divyakant Agrawal, Patrick ValduriezAbstract:Nowadays, we are witnessing the fast production of very large amount of data, particularly by the users of online systems on the Web. However, processing this big data is very challenging since both space and computational requirements are hard to satisfy. One solution for dealing with such requirements is to take advantage of parallel frameworks, such as MapReduce or Spark, that allow to make powerful computing and storage units on top of ordinary machines. Although these key-based frameworks have been praised for their high scalability and fault tolerance, they show poor performance in the case of data skew. There are important cases where a high percentage of processing in the reduce side ends up being done by only one node. In this paper, we present FP-Hadoop, a Hadoop-based system that renders the reduce side of MapReduce more parallel by efficiently tackling the problem of reduce data skew. FP-Hadoop introduces a new phase, denoted intermediate reduce (IR), where blocks of intermediate values are processed by intermediate reduce Workers in parallel. With this approach, even when all intermediate values are associated to the same key, the main part of the reducing work can be performed in parallel taking benefit of the computing power of all Available Workers. We implemented a prototype of FP-Hadoop, and conducted extensive experiments over synthetic and real datasets. We achieved excellent performance gains compared to native Hadoop, e.g. more than 10 times in reduce time and 5 times in total execution time.
-
An Efficient Solution for Processing Skewed MapReduce Jobs
2015Co-Authors: Reza Akbarinia, Miguel Liroz-gistau, Divyakant Agrawal, Patrick ValduriezAbstract:Although MapReduce has been praised for its high scalability and fault tolerance, it has been criticized in some points, in particular, its poor performance in the case of data skew. There are important cases where a high percentage of processing in the reduce side is done by a few nodes, or even one node, while the others remain idle. There have been some attempts to address the problem of data skew, but only for specific cases. In particular, there is no proposed solution for the cases where most of the intermediate values correspond to a single key, or when the number of keys is less than the number of reduce Workers. In this paper, we propose FP-Hadoop, a system that makes the reduce side of MapReduce more parallel, and efficiently deals with the problem of data skew in the reduce side. In FP-Hadoop, there is a new phase, called intermediate reduce (IR), in which blocks of intermediate values, constructed dynamically, are processed by intermediate reduce Workers in parallel, by using a scheduling strategy. By using the IR phase, even if all intermediate values belong to only one key, the main part of the reducing work can be done in parallel by using the computing resources of all Available Workers. We implemented a prototype of FP-Hadoop, and conducted extensive experiments over synthetic and real datasets. We achieved excellent performance gains compared to native Hadoop, e.g. more than 10 times in reduce time and 5 times in total execution time.
-
FP-Hadoop : Efficient Execution of Parallel Jobs Over Skewed Data
Proceedings of the 41st International Conference on Very Large Data Bases, 2015Co-Authors: Miguel Liroz-gistau, Reza Akbarinia, Patrick ValduriezAbstract:Big data parallel frameworks, such as MapReduce or Spark have been praised for their high scalability and performance, but show poor performance in the case of data skew. There are important cases where a high percentage of processing in the reduce side ends up being done by only one node. In this demonstration, we illustrate the use of FP-Hadoop, a system that efficiently deals with data skew in MapReduce jobs. In FP-Hadoop, there is a new phase, called inter-mediate reduce (IR), in which blocks of intermediate val-ues, constructed dynamically, are processed by intermediate reduce Workers in parallel, by using a scheduling strategy. Within the IR phase, even if all intermediate values belong to only one key, the main part of the reducing work can be done in parallel using the computing resources of all avail-able Workers. We implemented a prototype of FP-Hadoop, and conducted extensive experiments over synthetic and real datasets. We achieve excellent performance gains compared to native Hadoop, e.g. more than 10 times in reduce time and 5 times in total execution time. During our demonstration, we give the users the possibil-ity to execute and compare job executions in FP-Hadoop and Hadoop. They can retrieve general information about the job and the tasks and a summary of the phases. They can also visually compare different configurations to explore the difference between the approaches.
Gabriel Luque - One of the best experts on this subject based on the ideXlab platform.
-
Ant Colony Optimization Algorithm for Workforce Planning: Influence of the Algorithm Parameters
Advanced Computing in Industrial Mathematics, 2018Co-Authors: Stefka Fidanova, Olympia Roeva, Gabriel LuqueAbstract:The workforce planning is a difficult optimization problem. It is important real life problem which helps organizations to determine workforce which they need. A workforce planning problem is very complex and needs special algorithms to be solved using reasonable computational resources. The problem consists to select set of employers from a set of Available Workers and to assign this staff to the tasks to be performed. The objective is to minimize the costs associated to the human resources needed to fulfil the work requirements. A good workforce planing is important for an organization to accomplish its objectives. The complexity of this problem does not allow the application of exact methods for instances of realistic size. Therefore we will apply Ant Colony Optimization (ACO) method which is a stochastic method for solving combinatorial optimization problems. On this paper we focus on influence of the parameters on ACO algorithm performance.
-
WCO@FedCSIS - Intercriteria Analysis of ACO Performance for Workforce Planning Problem
Recent Advances in Computational Optimization, 2018Co-Authors: Olympia Roeva, Gabriel Luque, Stefka Fidanova, Marcin PaprzyckiAbstract:The workforce planning helps organizations to optimize the production process with the aim to minimize the assigning costs. The problem is to select a set of employees from a set of Available Workers and to assign this staff to the jobs to be performed. A workforce planning problem is very complex and requires special algorithms to be solved. The complexity of this problem does not allow the application of exact methods for instances of realistic size. Therefore, we will apply Ant Colony Optimization (ACO) algorithm, which is a stochastic method for solving combinatorial optimization problems. The ACO algorithm is tested on a set of 20 workforce planning problem instances. The obtained solutions are compared with other methods, as scatter search and genetic algorithm. The results show that ACO algorithm performs better than other the two algorithms. Further, we focus on the influence of the number of ants and the number of iterations on ACO algorithm performance. The tests are done on 16 different problem instances – ten structured and six unstructured problems. The results from ACO optimization procedures are discussed. In order to evaluate the influence of considered ACO parameters additional investigation is done. InterCriteria Analysis is performed on the ACO results for the regarded 16 problems. The results show that for the considered here workforce planning problem the best performance is achieved by the ACO algorithm with five ants in population.
-
FedCSIS - Ant colony optimization algorithm for workforce planning
Proceedings of the 2017 Federated Conference on Computer Science and Information Systems, 2017Co-Authors: Stefka Fidanova, Gabriel Luque, Olympia Roeva, Marcin Paprzycki, Pawel GepnerAbstract:The workforce planning helps organizations to optimize the production process with aim to minimize the assigning costs. A workforce planning problem is very complex and needs special algorithms to be solved. The problem is to select set of employers from a set of Available Workers and to assign this staff to the jobs to be performed. Each job requires a time to be completed. For efficiency, a worker must performs a minimum number of hours of any assigned job. There is a maximum number of jobs that can be assigned and a maximum number of Workers that can be assigned. There is a set of jobs that shows the jobs on which the worker is qualified. The objective is to minimize the costs associated to the human resources needed to fulfill the work requirements. On this work we propose a variant of Ant Colony Optimization (ACO) algorithm to solve workforce optimization problem. The algorithm is tested on a set of 20 test problems. Achieved solutions are compared with other methods, as scatter search and genetic algorithm. Obtained results show that ACO algorithm performs better than other two algorithms.
-
Parallel Genetic Algorithm for the Workforce Planning Problem
Studies in Computational Intelligence, 2011Co-Authors: Gabriel Luque, Enrique AlbaAbstract:Decision making associated with workforce planning results in difficult optimization problems, this is so it involves multiple levels of complexity. In fact, the workforce planning problem that we tackle in this chapter consists of two sets of decisions: selection and assignment. The first step selects a small set of employees from a large number of Available Workers and the second (decision) assigns this staff to the tasks to be performed. The objective is to minimize the costs associated to the human resources needed to fulfill the work requirements. An effective workforce plan is an essential tool to identify appropriate workload staffing levels and justify budget allocations so that organizations can meet their objectives.
Reza Akbarinia - One of the best experts on this subject based on the ideXlab platform.
-
FP-Hadoop: Efficient processing of skewed MapReduce jobs
Information Systems, 2016Co-Authors: Miguel Liroz-gistau, Reza Akbarinia, Divyakant Agrawal, Patrick ValduriezAbstract:Nowadays, we are witnessing the fast production of very large amount of data, particularly by the users of online systems on the Web. However, processing this big data is very challenging since both space and computational requirements are hard to satisfy. One solution for dealing with such requirements is to take advantage of parallel frameworks, such as MapReduce or Spark, that allow to make powerful computing and storage units on top of ordinary machines. Although these key-based frameworks have been praised for their high scalability and fault tolerance, they show poor performance in the case of data skew. There are important cases where a high percentage of processing in the reduce side ends up being done by only one node. In this paper, we present FP-Hadoop, a Hadoop-based system that renders the reduce side of MapReduce more parallel by efficiently tackling the problem of reduce data skew. FP-Hadoop introduces a new phase, denoted intermediate reduce (IR), where blocks of intermediate values are processed by intermediate reduce Workers in parallel. With this approach, even when all intermediate values are associated to the same key, the main part of the reducing work can be performed in parallel taking benefit of the computing power of all Available Workers. We implemented a prototype of FP-Hadoop, and conducted extensive experiments over synthetic and real datasets. We achieved excellent performance gains compared to native Hadoop, e.g. more than 10 times in reduce time and 5 times in total execution time.
-
An Efficient Solution for Processing Skewed MapReduce Jobs
2015Co-Authors: Reza Akbarinia, Miguel Liroz-gistau, Divyakant Agrawal, Patrick ValduriezAbstract:Although MapReduce has been praised for its high scalability and fault tolerance, it has been criticized in some points, in particular, its poor performance in the case of data skew. There are important cases where a high percentage of processing in the reduce side is done by a few nodes, or even one node, while the others remain idle. There have been some attempts to address the problem of data skew, but only for specific cases. In particular, there is no proposed solution for the cases where most of the intermediate values correspond to a single key, or when the number of keys is less than the number of reduce Workers. In this paper, we propose FP-Hadoop, a system that makes the reduce side of MapReduce more parallel, and efficiently deals with the problem of data skew in the reduce side. In FP-Hadoop, there is a new phase, called intermediate reduce (IR), in which blocks of intermediate values, constructed dynamically, are processed by intermediate reduce Workers in parallel, by using a scheduling strategy. By using the IR phase, even if all intermediate values belong to only one key, the main part of the reducing work can be done in parallel by using the computing resources of all Available Workers. We implemented a prototype of FP-Hadoop, and conducted extensive experiments over synthetic and real datasets. We achieved excellent performance gains compared to native Hadoop, e.g. more than 10 times in reduce time and 5 times in total execution time.
-
FP-Hadoop : Efficient Execution of Parallel Jobs Over Skewed Data
Proceedings of the 41st International Conference on Very Large Data Bases, 2015Co-Authors: Miguel Liroz-gistau, Reza Akbarinia, Patrick ValduriezAbstract:Big data parallel frameworks, such as MapReduce or Spark have been praised for their high scalability and performance, but show poor performance in the case of data skew. There are important cases where a high percentage of processing in the reduce side ends up being done by only one node. In this demonstration, we illustrate the use of FP-Hadoop, a system that efficiently deals with data skew in MapReduce jobs. In FP-Hadoop, there is a new phase, called inter-mediate reduce (IR), in which blocks of intermediate val-ues, constructed dynamically, are processed by intermediate reduce Workers in parallel, by using a scheduling strategy. Within the IR phase, even if all intermediate values belong to only one key, the main part of the reducing work can be done in parallel using the computing resources of all avail-able Workers. We implemented a prototype of FP-Hadoop, and conducted extensive experiments over synthetic and real datasets. We achieve excellent performance gains compared to native Hadoop, e.g. more than 10 times in reduce time and 5 times in total execution time. During our demonstration, we give the users the possibil-ity to execute and compare job executions in FP-Hadoop and Hadoop. They can retrieve general information about the job and the tasks and a summary of the phases. They can also visually compare different configurations to explore the difference between the approaches.
Sang-hyun Choi - One of the best experts on this subject based on the ideXlab platform.
-
Manufacturing cycle time reduction for batch production in a shared worker environment
International Journal of Production Research, 2013Co-Authors: Kwan Hee Han, Geon Lee, Sang-hyun ChoiAbstract:Responsive delivery without inefficient excess inventory requires short manufacturing cycle times. The manufacturing cycle time is comprised of operation time, loading/unloading time, set up time and machine idle time. The sum of loading/unloading, set up and machine idle times is called downtime. To shorten batch processing time, reduction in downtime must be the first priority. One way to reduce downtime per work unit is to increase batch quantity, while another is to shorten the sum of set up and machine idle times. Whereas fast cycle time is critical for time-based competition, efficient utilisation of labour is also essential to keep a manufacturing company competitive due to the rapid increase in labour costs. Therefore, in many small and medium-sized businesses, one worker simultaneously handles multiple machines. In such a shared worker environment, machine idle time inevitably occurs due to a lack of Available Workers. The purpose of this paper is to propose a downtime reduction method based on p...