The Experts below are selected from a list of 1401 Experts worldwide ranked by ideXlab platform

Cho-li Wang - One of the best experts on this subject based on the ideXlab platform.

  • A segment-based DSM supporting large shared object space
    Proceedings 20th IEEE International Parallel & Distributed Processing Symposium, 2006
    Co-Authors: B.w.-l. Cheung, Cho-li Wang
    Abstract:

    This paper introduces a software DSM that can extend its shared object space exceeding 4GB in a 32-bit Commodity Cluster environment. This is achieved through the dynamic memory mapping mechanism, with local hard disks as backing store. We introduce the new concept of segments with intelligent splitting to reduce network traffic, false sharing as well as adapt better to the shared memory access patterns. A priority-based swapping algorithm is designed to reduce disk accesses for efficient dynamic memory mapping, and maximize the use of disk space as shared object space. A new queue-based scheme is also devised for efficient and simple management of memory blocks. The proposed solutions were implemented in LOTS V.2, and it can outperform its previous version when running small applications, while the maximum shared object space is increased to one-third of the total free disk space available among all the nodes

  • Cluster - Contention-free complete exchange algorithm on Clusters
    Proceedings IEEE International Conference on Cluster Computing. CLUSTER 2000, 2000
    Co-Authors: Cho-li Wang
    Abstract:

    To construct a large Commodity Cluster a hierarchical network is generally adopted for connecting the host machines, where a Gigabit backbone switch connects a few Commodity switches with uplinks to achieve scaled bisectional bandwidth. This type of interconnection usually results in link contention and has congestion developed at the uplink ports. Moreover the non-deterministic delays on scheduling communication events in Clusters accelerate the building up of congestion amongst these uplink ports, which lead to severe packets drop and hinder the overall performance. In this paper, we focus on the practical design of high-speed complete exchange algorithm on a Commodity Cluster interconnected by a hierarchical Ethernet-based network. By exploiting some architectural characteristics of the interconnection in optimizing the performance of a complete exchange algorithm, we introduce a congestion control mechanism-global windowing that monitors and regulates the traffic load, together with a permutation scheme-reorder scheme that effectively alleviates the congestion problem. We evaluate our algorithm and compare its performance with other algorithms in a PC Cluster connected by various types of switches, including Gigabit Ethernet, input-buffered and shared-memory fast Ethernet switches.

  • Efficient reliable broadcast for Commodity Clusters
    Proceedings Fourth International Conference Exhibition on High Performance Computing in the Asia-Pacific Region, 2000
    Co-Authors: Kwan-po Wong, Cho-li Wang
    Abstract:

    High-speed collective communication is the key to achieve high-performance computing in parallel computing. In the past, collective operations are usually implemented using unicast operations. We proposed a new architecture EQA (Enhanced Queue Architecture) for implementing high-speed collective operations in a Cluster. With the incorporation of EQA and the hardware broadcast facility in network switches, an efficient reliable broadcast operation is implemented in a DP-SMP communication subsystem. With EQA, the computation, memory and network resources can be utilized efficiently. We evaluated the performance of the broadcast operation in a Commodity Cluster with fast Ethernet connection. We found that the hardware-based broadcast from DP-SMP with EQA outperforms the software-based broadcast operation. The use of EQA in broadcast operation could reduce the memory consumption by almost 40%. DP-SMP with EQA has proven to be an efficient communication mechanism for coupling Commodity Clusters.

  • Contention-free complete exchange algorithm on Clusters
    Proceedings IEEE International Conference on Cluster Computing. CLUSTER 2000, 2000
    Co-Authors: Cho-li Wang
    Abstract:

    To construct a large Commodity Cluster a hierarchical network is generally adopted for connecting the host machines, where a Gigabit backbone switch connects a few Commodity switches with uplinks to achieve scaled bisectional bandwidth. This type of interconnection usually results in link contention and has congestion developed at the uplink ports. Moreover the non-deterministic delays on scheduling communication events in Clusters accelerate the building up of congestion amongst these uplink ports, which lead to severe packets drop and hinder the overall performance. In this paper, we focus on the practical design of high-speed complete exchange algorithm on a Commodity Cluster interconnected by a hierarchical Ethernet-based network. By exploiting some architectural characteristics of the interconnection in optimizing the performance of a complete exchange algorithm, we introduce a congestion control mechanism-global windowing that monitors and regulates the traffic load, together with a permutation scheme-reorder scheme that effectively alleviates the congestion problem. We evaluate our algorithm and compare its performance with other algorithms in a PC Cluster connected by various types of switches, including Gigabit Ethernet, input-buffered and shared-memory fast Ethernet switches.

  • Realistic communication model for parallel computing on Cluster
    ICWC 99. IEEE Computer Society International Workshop on Cluster Computing, 1999
    Co-Authors: Cho-li Wang
    Abstract:

    We present a model for parallel computation on Commodity Cluster. Our Cluster model is targeted as a tool for performance analysis and algorithm design. We abstract the communication event by means of local and remote data movements, and explicitly expose the contention problems by capturing them in our parameters. To validate our model, we compare the prediction accuracy of our model with the Postal model for the popular tree-based broadcast algorithm. Our model provides good prediction accuracy and answers to some performance issues that are missing in existing models. We examine the gather collective operation, in which contention delay dominates its overall execution time. Based on the model, we design a communication schedule for the gather operation that is based on the upper and lower bounds, in which the congestion behavior could be under our control and achieve optimal results by avoiding data loss.

Peng Li - One of the best experts on this subject based on the ideXlab platform.

  • On Traffic-Aware Partition and Aggregation in MapReduce for Big Data Applications
    IEEE Transactions on Parallel and Distributed Systems, 2016
    Co-Authors: Huan Ke, Peng Li
    Abstract:

    The MapReduce programming model simplifies large-scale data processing on Commodity Cluster by exploiting parallel map tasks and reduce tasks. Although many efforts have been made to improve the performance of MapReduce jobs, they ignore the network traffic generated in the shuffle phase, which plays a critical role in performance enhancement. Traditionally, a hash function is used to partition intermediate data among reduce tasks, which, however, is not traffic-efficient because network topology and data size associated with each key are not taken into consideration. In this paper, we study to reduce network traffic cost for a MapReduce job by designing a novel intermediate data partition scheme. Furthermore, we jointly consider the aggregator placement problem, where each aggregator can reduce merged traffic from multiple map tasks. A decomposition-based distributed algorithm is proposed to deal with the large-scale optimization problem for big data application and an online algorithm is also designed to adjust data partition and aggregation in a dynamic manner. Finally, extensive simulation results demonstrate that our proposals can significantly reduce network traffic cost under both offline and online cases.

Iko Pramudiono - One of the best experts on this subject based on the ideXlab platform.

  • web community mining and web log mining Commodity Cluster based execution
    Australasian Database Conference, 2002
    Co-Authors: Masaru Kitsuregawa, Masashi Toyoda, Iko Pramudiono
    Abstract:

    The emergence of WWW has drawn new frontiers for database research. Web mining has become a hot topic since WWW rapid expansion rate and chaotic nature have exposed some technical challenges as well as interesting discoveries. In general web mining can be classified into web structure mining and web usage mining. Here we introduce two applications of web mining, first from mining the web structure we identify web communities, and the second we mine web usage of mobile internet users on location aware search engine. Those applications require heavy computational power as well as good scalability. Cluster of Commodity PCs is suitable as the platform to handle such applications. Here we also report some approaches for optimal parallel execution of mining algorithms on PC Cluster.

David Valencia - One of the best experts on this subject based on the ideXlab platform.

  • Cluster - Parallel Morphological/Neural Classification of Remote Sensing Images Using Fully Heterogeneous and Homogeneous Commodity Clusters
    2006 IEEE International Conference on Cluster Computing, 2006
    Co-Authors: Javier Plaza, Pablo Martínez, Antonio Plaza, Rosa M. Pérez, David Valencia
    Abstract:

    The wealth spatial and spectral information available from last-generation Earth observation instruments has introduced extremely high computational requirements in many applications. Most currently available parallel techniques treat remotely sensed data not as images, but as unordered listings of spectral measurements with no spatial arrangement. In thematic classification applications, however, the integration of spatial and spectral information can be greatly beneficial. Although such integrated approaches can be efficiently mapped in homogeneous Commodity Clusters, low-cost heterogeneous networks of computers (HNOCs) have soon become a standard tool of choice in Earth and planetary missions. In this paper, we develop a new morphological/neural parallel algorithm for Commodity Cluster-based analysis of high-dimensional remotely sensed image data sets. The algorithms accuracy and parallel performance are tested (in the context of a real precision agriculture application) using two parallel platforms: a fully heterogeneous Cluster made up of 16 workstations at University of Maryland, and a massively parallel Beowulf Cluster at NASA's Goddard Space Flight Center.

  • Commodity Cluster and hardware based massively parallel implementations of hyperspectral imaging algorithms
    Algorithms and Technologies for Multispectral Hyperspectral and Ultraspectral Imagery XII, 2006
    Co-Authors: Antonio Plaza, Javier Plaza, Cheini Chang, David Valencia
    Abstract:

    The incorporation of hyperspectral sensors aboard airborne/satellite platforms is currently producing a nearly continual stream of multidimensional image data, and this high data volume has soon introduced new processing challenges. The price paid for the wealth spatial and spectral information available from hyperspectral sensors is the enormous amounts of data that they generate. Several applications exist, however, where having the desired information calculated quickly enough for practical use is highly desirable. High computing performance of algorithm analysis is particularly important in homeland defense and security applications, in which swift decisions often involve detection of (sub-pixel) military targets (including hostile weaponry, camouflage, concealment, and decoys) or chemical/biological agents. In order to speed-up computational performance of hyperspectral imaging algorithms, this paper develops several fast parallel data processing techniques. Techniques include four classes of algorithms: (1) unsupervised classification, (2) spectral unmixing, and (3) automatic target recognition, and (4) onboard data compression. A massively parallel Beowulf Cluster (Thunderhead) at NASA's Goddard Space Flight Center in Maryland is used to measure parallel performance of the proposed algorithms. In order to explore the viability of developing onboard, real-time hyperspectral data compression algorithms, a Xilinx Virtex-II field programmable gate array (FPGA) is also used in experiments. Our quantitative and comparative assessment of parallel techniques and strategies may help image analysts in selection of parallel hyperspectral algorithms for specific applications.

  • Commodity Cluster based parallel processing of hyperspectral imagery
    Journal of Parallel and Distributed Computing, 2006
    Co-Authors: Antonio Plaza, David Valencia, Javier Plaza, Pablo Martínez
    Abstract:

    The rapid development of space and computer technologies has made possible to store a large amount of remotely sensed image data, collected from heterogeneous sources. In particular, NASA is continuously gathering imagery data with hyperspectral Earth observing sensors such as the Airborne Visible-Infrared Imaging Spectrometer (AVIRIS) or the Hyperion imager aboard Earth Observing-1 (EO-1) spacecraft. The development of fast techniques for transforming the massive amount of collected data into scientific understanding is critical for space-based Earth science and planetary exploration. This paper describes Commodity Cluster-based parallel data analysis strategies for hyperspectral imagery, a new class of image data that comprises hundreds of spectral bands at different wavelength channels for the same area on the surface of the Earth. An unsupervised technique that integrates the spatial and spectral information in the image data using multi-channel morphological transformations is parallelized and compared to other available parallel algorithms. The code's portability, reusability and scalability are illustrated by using two high-performance parallel computing architectures: a distributed memory, multiple instruction multiple data (MIMD)-style multicomputer at European Center for Parallelism of Barcelona, and a Beowulf Cluster at NASA's Goddard Space Flight Center. Experimental results suggest that Beowulf Clusters are a source of computational power that is both accessible and applicable to obtaining results in valid response times in information extraction applications from hyperspectral imagery.

  • Commodity Cluster-based parallel processing of hyperspectral imagery
    Journal of Parallel and Distributed Computing, 2006
    Co-Authors: Antonio J. Plaza, David Valencia, Javier Plaza, Pablo Martínez
    Abstract:

    The rapid development of space and computer technologies has made possible to store a large amount of remotely sensed image data, collected from heterogeneous sources. In particular, NASA is continuously gathering imagery data with hyperspectral Earth observing sensors such as the Airborne Visible-Infrared Imaging Spectrometer (AVIRIS) or the Hyperion imager aboard Earth Observing-1 (EO-1) spacecraft. The development of fast techniques for transforming the massive amount of collected data into scientific understanding is critical for space-based Earth science and planetary exploration. This paper describes Commodity Cluster-based parallel data analysis strategies for hyperspectral imagery, a new class of image data that comprises hundreds of spectral bands at different wavelength channels for the same area on the surface of the Earth. An unsupervised technique that integrates the spatial and spectral information in the image data using multi-channel morphological transformations is parallelized and compared to other available parallel algorithms. The code's portability, reusability and scalability are illustrated by using two high-performance parallel computing architectures: a distributed memory, multiple instruction multiple data (MIMD)-style multicomputer at European Center for Parallelism of Barcelona, and a Beowulf Cluster at NASA's Goddard Space Flight Center. Experimental results suggest that Beowulf Clusters are a source of computational power that is both accessible and applicable to obtaining results in valid response times in information extraction applications from hyperspectral imagery. © 2005 Elsevier Inc. All rights reserved.

  • Parallel Morphological/Neural Classification of Remote Sensing Images Using Fully Heterogeneous and Homogeneous Commodity Clusters
    2006 IEEE International Conference on Cluster Computing, 2006
    Co-Authors: Javier Plaza, Pablo Martínez, Antonio Plaza, Rosa Perez, David Valencia
    Abstract:

    The wealth spatial and spectral information available from last-generation Earth observation instruments has introduced extremely high computational requirements in many applications. Most currently available parallel techniques treat remotely sensed data not as images, but as unordered listings of spectral measurements with no spatial arrangement. In thematic classification applications, however, the integration of spatial and spectral information can be greatly beneficial. Although such integrated approaches can be efficiently mapped in homogeneous Commodity Clusters, low-cost heterogeneous networks of computers (HNOCs) have soon become a standard tool of choice in Earth and planetary missions. In this paper, we develop a new morphological/neural parallel algorithm for Commodity Cluster-based analysis of high-dimensional remotely sensed image data sets. The algorithms accuracy and parallel performance are tested (in the context of a real precision agriculture application) using two parallel platforms: a fully heterogeneous Cluster made up of 16 workstations at University of Maryland, and a massively parallel Beowulf Cluster at NASA's Goddard Space Flight Center

Huan Ke - One of the best experts on this subject based on the ideXlab platform.

  • On Traffic-Aware Partition and Aggregation in MapReduce for Big Data Applications
    IEEE Transactions on Parallel and Distributed Systems, 2016
    Co-Authors: Huan Ke, Peng Li
    Abstract:

    The MapReduce programming model simplifies large-scale data processing on Commodity Cluster by exploiting parallel map tasks and reduce tasks. Although many efforts have been made to improve the performance of MapReduce jobs, they ignore the network traffic generated in the shuffle phase, which plays a critical role in performance enhancement. Traditionally, a hash function is used to partition intermediate data among reduce tasks, which, however, is not traffic-efficient because network topology and data size associated with each key are not taken into consideration. In this paper, we study to reduce network traffic cost for a MapReduce job by designing a novel intermediate data partition scheme. Furthermore, we jointly consider the aggregator placement problem, where each aggregator can reduce merged traffic from multiple map tasks. A decomposition-based distributed algorithm is proposed to deal with the large-scale optimization problem for big data application and an online algorithm is also designed to adjust data partition and aggregation in a dynamic manner. Finally, extensive simulation results demonstrate that our proposals can significantly reduce network traffic cost under both offline and online cases.