The Experts below are selected from a list of 27942 Experts worldwide ranked by ideXlab platform

Aydin Buluc - One of the best experts on this subject based on the ideXlab platform.

  • integrated model batch and domain parallelism in training neural networks
    2018
    Co-Authors: Amir Gholami, Ariful Azad, Peter H Jin, Kurt Keutzer, Aydin Buluc
    Abstract:

    We propose a new integrated method of exploiting model, batch and domain parallelism for the training of deep neural networks (DNNs) on large distributed-Memory Computers using minibatch stochastic gradient descent (SGD). Our goal is to find an efficient parallelization strategy for a fixed batch size using P processes. Our method is inspired by the communication-avoiding algorithms in numerical linear algebra. We see P processes as logically divided into a P_r x P_c grid where the P_r dimension is implicitly responsible for model/domain parallelism and the P_c dimension is implicitly responsible for batch parallelism. In practice, the integrated matrix-based parallel algorithm encapsulates these types of parallelism automatically. We analyze the communication complexity and analytically demonstrate that the lowest communication costs are often achieved neither with pure model nor with pure data parallelism. We also show how the domain parallel approach can help in extending the theoretical scaling limit of the typical batch parallel method.

  • hipmcl a high performance parallel implementation of the markov clustering algorithm for large scale networks
    2018
    Co-Authors: Ariful Azad, Aydin Buluc, Georgios A Pavlopoulos, Christos A Ouzounis, Nikos C Kyrpides
    Abstract:

    Biological networks capture structural or functional properties of relevant entities such as molecules, proteins or genes. Characteristic examples are gene expression networks or protein-protein interaction networks, which hold information about functional affinities or structural similarities. Such networks have been expanding in size due to increasing scale and abundance of biological data. While various clustering algorithms have been proposed to find highly connected regions, Markov Clustering (MCL) has been one of the most successful approaches to cluster sequence similarity or expression networks. Despite its popularity, MCL's scalability to cluster large datasets still remains a bottleneck due to high running times and Memory demands. Here, we present High-performance MCL (HipMCL), a parallel implementation of the original MCL algorithm that can run on distributed-Memory Computers. We show that HipMCL can efficiently utilize 2000 compute nodes and cluster a network of ∼70 million nodes with ∼68 billion edges in ∼2.4 h. By exploiting distributed-Memory environments, HipMCL clusters large-scale networks several orders of magnitude faster than MCL and enables clustering of even bigger networks. HipMCL is based on MPI and OpenMP and is freely available under a modified BSD license.

  • integrated model batch and domain parallelism in training neural networks
    2017
    Co-Authors: Amir Gholami, Ariful Azad, Peter H Jin, Kurt Keutzer, Aydin Buluc
    Abstract:

    We propose a new integrated method of exploiting model, batch and domain parallelism for the training of deep neural networks (DNNs) on large distributed-Memory Computers using minibatch stochastic gradient descent (SGD). Our goal is to find an efficient parallelization strategy for a fixed batch size using $P$ processes. Our method is inspired by the communication-avoiding algorithms in numerical linear algebra. We see $P$ processes as logically divided into a $P_r \times P_c$ grid where the $P_r$ dimension is implicitly responsible for model/domain parallelism and the $P_c$ dimension is implicitly responsible for batch parallelism. In practice, the integrated matrix-based parallel algorithm encapsulates these types of parallelism automatically. We analyze the communication complexity and analytically demonstrate that the lowest communication costs are often achieved neither with pure model nor with pure data parallelism. We also show how the domain parallel approach can help in extending the theoretical scaling limit of the typical batch parallel method.

Fredrik Manne - One of the best experts on this subject based on the ideXlab platform.

  • parallel algorithms for bipartite matching problems on distributed Memory Computers
    2011
    Co-Authors: Johannes Langguth, Md Mostofa Ali Patwary, Fredrik Manne
    Abstract:

    We present a new parallel algorithm for computing a maximum cardinality matching in a bipartite graph suitable for distributed Memory Computers. The presented algorithm is based on the Push-Relabel algorithm which is known to be one of the fastest algorithms for the bipartite matching problem. Previous attempts at developing parallel implementations of it have focused on shared Memory Computers using only a limited number of processors. We first present a straightforward adaptation of these shared Memory algorithms to distributed Memory Computers. However, this is not a viable approach as it requires too much communication. We then develop our new algorithm by modifying the previous approach through a sequence of steps with the main goal being to reduce the amount of communication and to increase load balance. The first goal is achieved by changing the algorithm so that many push and relabel operations can be performed locally between communication rounds and also by selecting augmenting paths that cross processor boundaries infrequently. To achieve good load balance, we limit the speed at which global relabelings traverse the graph. In several experiments on a large number of instances, we study weak and strong scalability of our algorithm using up to 128 processors. The algorithm can also be used to find @e-approximate matchings quickly.

  • A scalable parallel union-find algorithm for distributed Memory Computers
    2010
    Co-Authors: Fredrik Manne, Md Shahadath Hossain Patwary
    Abstract:

    The Union-Find algorithm is used for maintaining a number of non-overlapping sets from a finite universe of elements. The algorithm has applications in a number of areas including the computation of spanning trees, sparse linear algebra, and in image processing. Although the algorithm is inherently sequential there has been some previous efforts at constructing parallel implementations. These have mainly focused on shared Memory Computers. In this paper we present the first scalable parallel implementation of the Union-Find algorithm suitable for distributed Memory Computers. Our new parallel algorithm is based on an observation of how the Find part of the sequential algorithm can be executed more efficiently. We show the efficiency of our implementation through a series of tests to compute spanning forests of very large graphs.

  • maximum weighted matching using the partitioned global address space model
    2009
    Co-Authors: Alicia Thorsen, Phillip Merkey, Fredrik Manne
    Abstract:

    Efficient parallel algorithms for problems such as maximum weighted matching are central to many areas of combinatorial scientific computing. Manne and Bisseling [13] presented a parallel approximation algorithm which is well suited to distributed Memory Computers. This algorithm is based on a distributed protocol due to Hoepman [9]. In the current paper, a partitioned global address space (PGAS) implementation is presented. PGAS programmers have the conveniences of using a shared Memory model, which provides implicit communication between processes using normal loads and stores. Since the shared Memory is partitioned according to the affinity of a process, one is also able to exploit data locality. This paper addresses the main differences between the PGAS and MPI implementations of the Manne-Bisseling algorithm. It highlights some advantages of using the PGAS model such as shorter, simpler code, similarity to the sequential algorithm, and options for fine-grained and coarse-grained communication.

  • a framework for scalable greedy coloring on distributed Memory parallel Computers
    2008
    Co-Authors: Doruk Bozdag, Fredrik Manne, Erik G Boman, Assefaw H Gebremedhin, Umit V Catalyurek
    Abstract:

    We present a scalable framework for parallelizing greedy graph coloring algorithms on distributed-Memory Computers. The framework unifies several existing algorithms and blends a variety of techniques for creating or facilitating concurrency. The latter techniques include exploiting features of the initial data distribution, the use of speculative coloring and randomization, and a BSP-style organization of computation and communication. We experimentally study the performance of several specialized algorithms designed using the framework and implemented using MPI. The experiments are conducted on two different platforms and the test cases include large-size synthetic graphs as well as real graphs drawn from various application areas. Computational results show that implementations that yield good speedup while at the same time using about the same number of colors as a sequential greedy algorithm can be achieved by setting parameters of the framework in accordance with the size and structure of the graph being colored. Our implementation is freely available as part of the Zoltan parallel data management and load-balancing library.

  • a parallel distance 2 graph coloring algorithm for distributed Memory Computers
    2005
    Co-Authors: Doruk Bozdag, Fredrik Manne, Erik G Boman, Umit V Catalyurek, Assefaw H Gebremedhin, Fusun Ozguner
    Abstract:

    The distance-2 graph coloring problem aims at partitioning the vertex set of a graph into the fewest sets consisting of vertices pairwise at distance greater than two from each other. Application examples include numerical optimization and channel assignment. We present the first distributed-Memory heuristic algorithm for this NP-hard problem. Parallel speedup is achieved through graph partitioning, speculative (iterative) coloring, and a BSP-like organization of computation. Experimental results show that the algorithm is scalable, and compares favorably with an alternative approach—solving the problem on a graph G by first constructing the square graph G2 and then applying a parallel distance-1 coloring algorithm on G2.

Francoisxavier Roux - One of the best experts on this subject based on the ideXlab platform.

  • a dissection solver with kernel detection for symmetric finite element matrices on shared Memory Computers
    2014
    Co-Authors: Atsushi Suzuki, Francoisxavier Roux
    Abstract:

    SUMMARY A direct solver for symmetric sparse matrices from finite element problems is presented. The solver is supposed to work as a local solver of domain decomposition methods for hybrid parallelization on cluster systems of multi-core CPUs, and then it is required to run on shared Memory Computers and to have an ability of kernel detection. Symmetric pivoting with a given threshold factorizes a matrix with a decomposition introduced by a nested bisection and selects suspicious null pivots from the threshold. The Schur complement constructed from the suspicious null pivots is examined by a factorization with 1 × 1 and 2 × 2 pivoting and by a robust kernel detection algorithm based on measurement of residuals with orthogonal projections onto supposed image spaces. A static data structure from the nested bisection and a block sub-structure for Schur complements at all bisection levels can use level 3 BLAS routines efficiently. Asynchronous task execution for each block can reduce idle time of processors drastically, and as a result, the solver has high parallel efficiency. Competitive performance of the developed solver to Intel Pardiso on shared Memory Computers is shown by numerical experiments. Copyright © 2014 John Wiley & Sons, Ltd.

  • a dissection solver with kernel detection for symmetric finite element matrices on shared Memory Computers
    2014
    Co-Authors: Atsushi Suzuki, Francoisxavier Roux
    Abstract:

    A direct solver for symmetric sparse matrices from finite element problems is presented. The solver is supposed to work as a local solver of domain decomposition methods for hybrid parallelization on cluster systems of multi-core CPUs, and then it is required to run on shared Memory Computers and to have an ability of kernel detection. Symmetric pivoting with a given threshold factorizes a matrix with a decomposition introduced by a nested bisection and selects suspicious null pivots from the threshold. The Schur complement constructed from the suspicious null pivots is examined by a factorization with 1x1 and 2x2 pivoting and by a robust kernel detection algorithm based on measurement of residuals with orthogonal projections onto supposed image spaces. A static data structure from the nested bisection and a block sub-structure for Schur complements at all bisection-levels can use level 3 BLAS routines efficiently. Asynchronous task execution for each block can reduce idle time of processors drastically and as a result, the solver has high parallel efficiency. Competitive performance of the developed solver to Intel Pardiso on shared Memory Computers is shown by numerical experiments.

Huazhong Yang - One of the best experts on this subject based on the ideXlab platform.

  • nicslu an adaptive sparse matrix solver for parallel circuit simulation
    2013
    Co-Authors: Xiaoming Chen, Yu Wang, Huazhong Yang
    Abstract:

    The sparse matrix solver has become a bottleneck in simulation program with integrated circuit emphasis (SPICE)-like circuit simulators. It is difficult to parallelize the solver because of the high data dependency during the numeric LU factorization and the irregular structure of circuit matrices. This paper proposes an adaptive sparse matrix solver called NICSLU, which uses a multithreaded parallel LU factorization algorithm on shared-Memory Computers with multicore/multisocket central processing units to accelerate circuit simulation. The solver can be used in all the SPICE-like circuit simulators. A simple method is proposed to predict whether a matrix is suitable for parallel factorization, such that each matrix can achieve optimal performance. The experimental results on 35 matrices reveal that NICSLU achieves speedups of 2.08× ~ 8.57×(on the geometric mean), compared with KLU, with 1-12 threads, for the matrices which are suitable for the parallel algorithm. NICSLU can be downloaded from http://nicslu.weebly.com.

  • an adaptive lu factorization algorithm for parallel circuit simulation
    2012
    Co-Authors: Xiaoming Chen, Yu Wang, Huazhong Yang
    Abstract:

    Sparse matrix solver has become the bottleneck in SPICE simulator. It is difficult to parallelize the solver because of the high data-dependency during the numerical LU factorization. This paper proposes a parallel LU factorization (with partial pivoting) algorithm on shared-Memory Computers with multi-core CPUs, to accelerate circuit simulation. Since not every matrix is suitable for parallel algorithm, a predictive method is proposed to decide whether a matrix should use parallel or sequential algorithm. The experimental results on 35 circuit matrices reveal that the developed algorithm achieves speedups of 2.11×∼8.38× (on geometric-average), compared with KLU, with 1∼8 threads, on the matrices which are suitable for parallel algorithm. Our solver can be downloaded from http://nicslu.weebly.com.

  • An EScheduler-Based Data Dependence Analysis and Task Scheduling for Parallel Circuit Simulation
    2011
    Co-Authors: Xiaoming Chen, Yu Wang, Wei Wu, Hao Yu, Huazhong Yang
    Abstract:

    The sparse matrix solver has become the bottleneck in a Simulation Program with Integrated Circuit Emphasis circuit simulator. It is difficult to parallelize the sparse matrix solver because of the high data dependence during the numerical LU factorization. In this brief, a parallel LU factorization algorithm is developed on shared-Memory Computers with multicore central processing units, based on KLU algorithms. An Elimination Scheduler (EScheduler) is proposed to represent the data dependence during the LU factorization. Based on the EScheduler, the parallel tasks are scheduled in two modes to achieve a high level of concurrence, i.e., cluster mode and pipeline mode . The experimental results on 26 circuit matrices reveal that the developed algorithm can achieve speedup of 1.18-4.55t (on geometric average), as compared with KLU, with 1-8 threads. The result analysis indicates that for different data dependence, different parallel strategies should be dynamically selected to obtain optimal performance.

Iddo Friedberg - One of the best experts on this subject based on the ideXlab platform.

  • swiftortho a fast Memory efficient multiple genome orthology classifier
    2019
    Co-Authors: Xiao Hu, Iddo Friedberg
    Abstract:

    BACKGROUND: Gene homology type classification is required for many types of genome analyses, including comparative genomics, phylogenetics, and protein function annotation. Consequently, a large variety of tools have been developed to perform homology classification across genomes of different species. However, when applied to large genomic data sets, these tools require high Memory and CPU usage, typically available only in computational clusters. FINDINGS: Here we present a new graph-based orthology analysis tool, SwiftOrtho, which is optimized for speed and Memory usage when applied to large-scale data. SwiftOrtho uses long k-mers to speed up homology search, while using a reduced amino acid alphabet and spaced seeds to compensate for the loss of sensitivity due to long k-mers. In addition, it uses an affinity propagation algorithm to reduce the Memory usage when clustering large-scale orthology relationships into orthologous groups. In our tests, SwiftOrtho was the only tool that completed orthology analysis of proteins from 1,760 bacterial genomes on a computer with only 4 GB RAM. Using various standard orthology data sets, we also show that SwiftOrtho has a high accuracy. CONCLUSIONS: SwiftOrtho enables the accurate comparative genomic analyses of thousands of genomes using low-Memory Computers. SwiftOrtho is available at https://github.com/Rinoahu/SwiftOrtho.

  • swiftortho a fast Memory efficient multiple genome orthology classifier
    2019
    Co-Authors: Xiao Hu, Iddo Friedberg
    Abstract:

    Abstract Introduction Gene homology type classification is a requisite for many types of genome analyses, including comparative genomics, phylogenetics, and protein function annotation. A large variety of tools have been developed to perform homology classification across genomes of different species. However, when applied to large genomic datasets, these tools require high Memory and CPU usage, typically available only in costly computational clusters. To address this problem, we developed a new graph-based orthology analysis tool, SwiftOrtho, which is optimized for speed and Memory usage when applied to large-scale data. Results In our tests, SwiftOrtho is the only tool that completed orthology analysis of 1,760 bacterial genomes on a computer with only 4GB RAM. Using various standard orthology datasets, we also show that SwiftOrtho has a high accuracy. SwiftOrtho enables the accurate comparative genomic analyses of thousands of genomes using low Memory Computers. Availability https://github.com/Rinoahu/SwiftOrtho