The Experts below are selected from a list of 6 Experts worldwide ranked by ideXlab platform

Muhan Deng - One of the best experts on this subject based on the ideXlab platform.

  • Technical Methods and Application Modes of Lancang River Waterway Modern Management
    ICTE 2015, 2015
    Co-Authors: Xue Wang, Dongsheng Li, Muhan Deng
    Abstract:

    Lancang River is an important navigable river in Yunnan province, at the same time it is a cross-border international river in southwest China, however, compared to other rivers such as Yangtze River, the waterway management and maintenance level is relatively backward, and modernization and Information Lever is lower. Therefore, according to the characteristics of the Lancang River, there is a need to achieve the modernization and efficient of Lancang River waterway management by means of Information, so that the Lancang River waterway management and maintenance can reflect the unique pattern of the Lancang River shipping, and promote the development of the Lancang River shipping. Based on analysis of the current situation and needs of the Lancang River waterway management, the article put forward the suitable advanced, modern management and maintenance mode for Lancang River, which is based on the building principles of conducting top-level design, constructing the overall framework, improving infrastructure, achieving breakthroughs in key business systems by small strides. This study is an important foundation of constructing the Lancang River intelligent waterway, and realizing full coverage, all-weather operation and rapid response modern waterway management system. It is helpful to improve the management and maintenance level of the Lancang River waterway, and promote the shipping economic development and international shipping Exchanges development of the Lancang - Mekong River. It can accelerate the Lancang - Mekong international shipping development and construction, improve the status of the Lancang River shipping, and have important significance in promoting economic and social development in Yunnan Province, developing international trade, establishing a good international environment on the neighboring countries.

Xue Wang - One of the best experts on this subject based on the ideXlab platform.

  • Technical Methods and Application Modes of Lancang River Waterway Modern Management
    ICTE 2015, 2015
    Co-Authors: Xue Wang, Dongsheng Li, Muhan Deng
    Abstract:

    Lancang River is an important navigable river in Yunnan province, at the same time it is a cross-border international river in southwest China, however, compared to other rivers such as Yangtze River, the waterway management and maintenance level is relatively backward, and modernization and Information Lever is lower. Therefore, according to the characteristics of the Lancang River, there is a need to achieve the modernization and efficient of Lancang River waterway management by means of Information, so that the Lancang River waterway management and maintenance can reflect the unique pattern of the Lancang River shipping, and promote the development of the Lancang River shipping. Based on analysis of the current situation and needs of the Lancang River waterway management, the article put forward the suitable advanced, modern management and maintenance mode for Lancang River, which is based on the building principles of conducting top-level design, constructing the overall framework, improving infrastructure, achieving breakthroughs in key business systems by small strides. This study is an important foundation of constructing the Lancang River intelligent waterway, and realizing full coverage, all-weather operation and rapid response modern waterway management system. It is helpful to improve the management and maintenance level of the Lancang River waterway, and promote the shipping economic development and international shipping Exchanges development of the Lancang - Mekong River. It can accelerate the Lancang - Mekong international shipping development and construction, improve the status of the Lancang River shipping, and have important significance in promoting economic and social development in Yunnan Province, developing international trade, establishing a good international environment on the neighboring countries.

Yongluan Zhou - One of the best experts on this subject based on the ideXlab platform.

  • SoCC - Lever: towards low-latency batched stream processing by pre-scheduling
    Proceedings of the 2017 Symposium on Cloud Computing, 2017
    Co-Authors: Fei Chen, Song Wu, Lin Gu, Yongluan Zhou
    Abstract:

    With the vast involvement of streaming big data in many applications (e.g., stock market data, sensor data, social network data, etc.), quickly mining and analyzing such data is becoming more and more important. To provide fault tolerance and efficient stream processing at scale, recent stream processing frameworks have proposed to adapt batch processing systems, such as MapReduce and Spark, to handle streaming data by putting the streams into micro-batches and treating the workloads as a continuous series of small jobs [1]. The fundamental challenge of building a batched stream processing system is to minimize the processing latency of each micro-batch. In this paper, we focus on the straggler problem, where a subset of workers are straggling behind and significantly affecting the job completion time. The straggler problem is a well-known critical problem in parallel processing systems. In comparing to large batch processing, the straggler problems in micro-batch processing are more severe and harder to tackle. We argue that the problem of using the existing straggler mitigation solutions for micro-batch processing is that they detect (or predict) stragglers and re-schedule stragglers too late in the data handling pipeline. The re-scheduling actions are carried out during the task execution period, hence it would inevitably increase the processing time of the micro-batches. Furthermore, as the data have already been dispatched, re-scheduling would inherently incur expensive data relocation. Such overhead would become significant in micro-batch processing due to the short processing time of each micro-batch. We refer to this type of methods as post-scheduling techniques. To address the problem, we propose a new pre-scheduling framework, called Lever, which predicts stragglers and makes timely scheduling decisions to minimize the processing latency. As shown in Figure 1, Lever periodically collects and analyzes the historical job profiles of the recurring micro-batch jobs. Based on such Information, Lever pre-schedules the data through three main steps, i.e. identify potential stragglers, evaluate node capacity and choose suitable helpers. More importantly, Lever makes the re-scheduling decisions before the batching module dispatches the data. As the scheduling is done while the data are being batched, it would not increase the processing time of the micro-batch. We implemented Lever in Spark Streaming, which has been contributed to the open source community as an extension of Apache Spark Streaming. To the best of our knowledge, this is the first work specifically addressing the straggler problem in continuous micro-batch processing. We conduct various experiments to validate the effectiveness of Lever. The experimental results demonstrate that Lever reduces job completion time by 30.72% to 42.19% and outperforms traditional techniques significantly.

Dongsheng Li - One of the best experts on this subject based on the ideXlab platform.

  • Technical Methods and Application Modes of Lancang River Waterway Modern Management
    ICTE 2015, 2015
    Co-Authors: Xue Wang, Dongsheng Li, Muhan Deng
    Abstract:

    Lancang River is an important navigable river in Yunnan province, at the same time it is a cross-border international river in southwest China, however, compared to other rivers such as Yangtze River, the waterway management and maintenance level is relatively backward, and modernization and Information Lever is lower. Therefore, according to the characteristics of the Lancang River, there is a need to achieve the modernization and efficient of Lancang River waterway management by means of Information, so that the Lancang River waterway management and maintenance can reflect the unique pattern of the Lancang River shipping, and promote the development of the Lancang River shipping. Based on analysis of the current situation and needs of the Lancang River waterway management, the article put forward the suitable advanced, modern management and maintenance mode for Lancang River, which is based on the building principles of conducting top-level design, constructing the overall framework, improving infrastructure, achieving breakthroughs in key business systems by small strides. This study is an important foundation of constructing the Lancang River intelligent waterway, and realizing full coverage, all-weather operation and rapid response modern waterway management system. It is helpful to improve the management and maintenance level of the Lancang River waterway, and promote the shipping economic development and international shipping Exchanges development of the Lancang - Mekong River. It can accelerate the Lancang - Mekong international shipping development and construction, improve the status of the Lancang River shipping, and have important significance in promoting economic and social development in Yunnan Province, developing international trade, establishing a good international environment on the neighboring countries.

Fei Chen - One of the best experts on this subject based on the ideXlab platform.

  • SoCC - Lever: towards low-latency batched stream processing by pre-scheduling
    Proceedings of the 2017 Symposium on Cloud Computing, 2017
    Co-Authors: Fei Chen, Song Wu, Lin Gu, Yongluan Zhou
    Abstract:

    With the vast involvement of streaming big data in many applications (e.g., stock market data, sensor data, social network data, etc.), quickly mining and analyzing such data is becoming more and more important. To provide fault tolerance and efficient stream processing at scale, recent stream processing frameworks have proposed to adapt batch processing systems, such as MapReduce and Spark, to handle streaming data by putting the streams into micro-batches and treating the workloads as a continuous series of small jobs [1]. The fundamental challenge of building a batched stream processing system is to minimize the processing latency of each micro-batch. In this paper, we focus on the straggler problem, where a subset of workers are straggling behind and significantly affecting the job completion time. The straggler problem is a well-known critical problem in parallel processing systems. In comparing to large batch processing, the straggler problems in micro-batch processing are more severe and harder to tackle. We argue that the problem of using the existing straggler mitigation solutions for micro-batch processing is that they detect (or predict) stragglers and re-schedule stragglers too late in the data handling pipeline. The re-scheduling actions are carried out during the task execution period, hence it would inevitably increase the processing time of the micro-batches. Furthermore, as the data have already been dispatched, re-scheduling would inherently incur expensive data relocation. Such overhead would become significant in micro-batch processing due to the short processing time of each micro-batch. We refer to this type of methods as post-scheduling techniques. To address the problem, we propose a new pre-scheduling framework, called Lever, which predicts stragglers and makes timely scheduling decisions to minimize the processing latency. As shown in Figure 1, Lever periodically collects and analyzes the historical job profiles of the recurring micro-batch jobs. Based on such Information, Lever pre-schedules the data through three main steps, i.e. identify potential stragglers, evaluate node capacity and choose suitable helpers. More importantly, Lever makes the re-scheduling decisions before the batching module dispatches the data. As the scheduling is done while the data are being batched, it would not increase the processing time of the micro-batch. We implemented Lever in Spark Streaming, which has been contributed to the open source community as an extension of Apache Spark Streaming. To the best of our knowledge, this is the first work specifically addressing the straggler problem in continuous micro-batch processing. We conduct various experiments to validate the effectiveness of Lever. The experimental results demonstrate that Lever reduces job completion time by 30.72% to 42.19% and outperforms traditional techniques significantly.