The Experts below are selected from a list of 5187 Experts worldwide ranked by ideXlab platform
James Demmel - One of the best experts on this subject based on the ideXlab platform.
-
author retrospective for optimizing Matrix Multiply using phipac a portable high performance ansi c coding methodology
International Conference on Supercomputing, 2014Co-Authors: Jeff Bilmes, Krste Asanovic, Cheewhye Chin, James DemmelAbstract:PHiPAC was an early attempt to improve software performance by searching in a large design space of possible implementations to find the best one. At the time, in the early 1990s, the most efficient numerical linear algebra libraries were carefully hand tuned for specific microarchitectures and compilers, and were often written in assembly language. This allowed very precise tuning of an algorithm to the specifics of a current platform, and provided great opportunity for high efficiency. The prevailing thought at the time was that such an approach was necessary to produce near-peak performance. On the other hand, this approach was brittle, and required great human effort to try each code variant, and so only a tiny subset of the possible code design points could be explored. Worse, given the combined complexities of the compiler and microarchitecture, it was difficult to predict which code variants would be worth the implementation effort.PHiPAC circumvented this effort by using code generators that could easily generate a vast assortment of very different points within a design space, and even across very different design spaces altogether. By following a set of carefully crafted coding guidelines, the generated code was reasonably efficient for any point in the design space. To search the design space, PHiPAC took a rather naive but effective approach. Due to the human-designed and deterministic nature of computing systems, one might reasonably think that smart modeling of the microprocessor and compiler would be sufficient to predict, without performing any timing, the optimal point for a given algorithm. But the combination of an optimizing compiler and a dynamically scheduled microarchitecture with multiple levels of cache and prefetching results in a very complex, highly non-linear system. Small changes in the code can cause large changes in performance, thwarting "smart" search algorithms that attempt to build models of the performance surface. PHiPAC instead used massive, diverse, offline, and only loosely informed grid or randomized search, where each point was evaluated by empirically measuring the code running on the machine, which succeeded due to its indifference to the shape of the performance surface.
-
brief announcement lower bounds on communication for sparse cholesky factorization of a model problem
ACM Symposium on Parallel Algorithms and Architectures, 2010Co-Authors: Laura Grigori, James Demmel, Pierreyves David, Sylvain PeyronnetAbstract:Previous work has shown that a lower bound on the number of words moved between large, slow memory and small, fast memory of size M by any conventional (non-Strassen like) direct linear algebra algorithm (Matrix Multiply, the LU, Cholesky, QR factorizations,...) is Ω(# flops / √ (M)). This holds for dense or sparse matrices. There are analogous lower bounds for the number of messages, and for parallel algorithms instead of sequential algorithms. Our goal here is to find algorithms that attain these lower bounds on interesting classes of sparse matrices. We focus on matrices for which there is a lower bound on the number of flops of their Cholesky factorization. Our Cholesky lower bounds on communication hold for any possible ordering of the rows and columns of the Matrix, and so are globally optimal in this sense. For matrices arising from discretization on two dimensional and three dimensional regular grids, we discuss sequential and parallel algorithms that are optimal in terms of communication. The algorithms turn out to require combining previously known sparse and dense Cholesky algorithms in simple ways
-
benchmarking gpus to tune dense linear algebra
IEEE International Conference on High Performance Computing Data and Analytics, 2008Co-Authors: Vasily Volkov, James DemmelAbstract:We present performance results for dense linear algebra using recent NVIDIA GPUs. Our Matrix-Matrix Multiply routine (GEMM) runs up to 60% faster than the vendor's implementation and approaches the peak of hardware capabilities. Our LU, QR and Cholesky factorizations achieve up to 80--90% of the peak GEMM rate. Our parallel LU running on two GPUs achieves up to ~540 Gflop/s. These results are accomplished by challenging the accepted view of the GPU architecture and programming guidelines. We argue that modern GPUs should be viewed as multithreaded multicore vector units. We exploit blocking similarly to vector computers and heterogeneity of the system by computing both on GPU and CPU. This study includes detailed benchmarking of the GPU memory system that reveals sizes and latencies of caches and TLB. We present a couple of algorithmic optimizations aimed at increasing parallelism and regularity in the problem that provide us with slightly higher performance.
-
benchmarking gpus to tune dense linear algebra
IEEE International Conference on High Performance Computing Data and Analytics, 2008Co-Authors: Vasily Volkov, James DemmelAbstract:We present performance results for dense linear algebra using recent NVIDIA GPUs. Our Matrix-Matrix Multiply routine (GEMM) runs up to 60% faster than the vendor's implementation and approaches the peak of hardware capabilities. Our LU, QR and Cholesky factorizations achieve up to 80--90% of the peak GEMM rate. Our parallel LU running on two GPUs achieves up to ~540 Gflop/s. These results are accomplished by challenging the accepted view of the GPU architecture and programming guidelines. We argue that modern GPUs should be viewed as multithreaded multicore vector units. We exploit blocking similarly to vector computers and heterogeneity of the system by computing both on GPU and CPU. This study includes detailed benchmarking of the GPU memory system that reveals sizes and latencies of caches and TLB. We present a couple of algorithmic optimizations aimed at increasing parallelism and regularity in the problem that provide us with slightly higher performance.
-
Using PHiPAC to speed error back-propagation learning
2002Co-Authors: Jeff Bilmes, Krste Asanovic, James DemmelAbstract:We introduce PHiPAC, a coding methodology for develop-ing portable high-performance numerical libraries in ANSI C. Using this methodology, we have developed code for optimized Matrix Multiply routines. These routines can achieve over 90% of peak performance on a variety of cur-rent workstations, and are often faster than vendor-supplied optimized libraries. We then describe the bunch-mode back-propagation algorithm and how it can use the PHiPAC derived Matrix Multiply routines. Using a set of plots, we investigate the tradeoffs between bunch size, convergence rate, and training speed using a standard speech recogni-tion data set and show how use of the PHiPAC routines can lead to a significantly faster back-propagation learning
Mark S Gordon - One of the best experts on this subject based on the ideXlab platform.
-
quantum chemical calculations using accelerators migrating Matrix operations to the nvidia kepler gpu and the intel xeon phi
Journal of Chemical Theory and Computation, 2014Co-Authors: Sarom S Leang, Alistair P Rendell, Mark S GordonAbstract:Increasingly, modern computer systems comprise a multicore general-purpose processor augmented with a number of special purpose devices or accelerators connected via an external interface such as a PCI bus. The NVIDIA Kepler Graphical Processing Unit (GPU) and the Intel Phi are two examples of such accelerators. Accelerators offer peak performances that can be well above those of the host processor. How to exploit this heterogeneous environment for legacy application codes is not, however, straightforward. This paper considers how Matrix operations in typical quantum chemical calculations can be migrated to the GPU and Phi systems. Double precision general Matrix Multiply operations are endemic in electronic structure calculations, especially methods that include electron correlation, such as density functional theory, second order perturbation theory, and coupled cluster theory. The use of approaches that automatically determine whether to use the host or an accelerator, based on problem size, is explore...
-
quantum chemical calculations using accelerators migrating Matrix operations to the nvidia kepler gpu and the intel xeon phi
Journal of Chemical Theory and Computation, 2014Co-Authors: Sarom S Leang, Alistair P Rendell, Mark S GordonAbstract:Increasingly, modern computer systems comprise a multicore general-purpose processor augmented with a number of special purpose devices or accelerators connected via an external interface such as a PCI bus. The NVIDIA Kepler Graphical Processing Unit (GPU) and the Intel Phi are two examples of such accelerators. Accelerators offer peak performances that can be well above those of the host processor. How to exploit this heterogeneous environment for legacy application codes is not, however, straightforward. This paper considers how Matrix operations in typical quantum chemical calculations can be migrated to the GPU and Phi systems. Double precision general Matrix Multiply operations are endemic in electronic structure calculations, especially methods that include electron correlation, such as density functional theory, second order perturbation theory, and coupled cluster theory. The use of approaches that automatically determine whether to use the host or an accelerator, based on problem size, is explored, with computations that are occurring on the accelerator and/or the host. For data-transfers over PCI-e, the GPU provides the best overall performance for data sizes up to 4096 MB with consistent upload and download rates between 5-5.6 GB/s and 5.4-6.3 GB/s, respectively. The GPU outperforms the Phi for both square and nonsquare Matrix multiplications.
Matt Challacombe - One of the best experts on this subject based on the ideXlab platform.
-
an optimized sparse approximate Matrix Multiply for matrices with decay
SIAM Journal on Scientific Computing, 2013Co-Authors: Nicolas Bock, Matt ChallacombeAbstract:We present an optimized single-precision implementation of the sparse approximate Matrix Multiply (SpAMM}) [M. Challacombe and N. Bock, arXiv 1011.3534, 2010], a fast algorithm for Matrix-Matrix multiplication for matrices with decay that achieves an $\mathcal{O} ( n \log n )$ computational complexity with respect to Matrix dimension $n$. We find that the max norm of the error achieved with a SpAMM tolerance below $2 \times 10^{-8}$ is lower than that of the single-precision general Matrix-Matrix Multiply (SGEMM}) for dense quantum chemical matrices, while outperforming SGEMM with a cross-over already for small matrices ($n \sim 1000$). Relative to naive implementations of SpAMM using Intel's Math Kernel Library or AMD's Core Math Library, our optimized version is found to be significantly faster. Detailed performance comparisons are made for quantum chemical matrices with differently structured sub-blocks. Finally, we discuss the potential of improved hardware prefetch to yield 2x to 3x speedups.
-
an optimized sparse approximate Matrix Multiply for matrices with decay
arXiv: Numerical Analysis, 2012Co-Authors: Nicolas Bock, Matt ChallacombeAbstract:We present an optimized single-precision implementation of the Sparse Approximate Matrix Multiply (\SpAMM{}) [M. Challacombe and N. Bock, arXiv {\bf 1011.3534} (2010)], a fast algorithm for Matrix-Matrix multiplication for matrices with decay that achieves an $\mathcal{O} (n \log n)$ computational complexity with respect to Matrix dimension $n$. We find that the max norm of the error achieved with a \SpAMM{} tolerance below $2 \times 10^{-8}$ is lower than that of the single-precision {\tt SGEMM} for dense quantum chemical matrices, while outperforming {\tt SGEMM} with a cross-over already for small matrices ($n \sim 1000$). Relative to naive implementations of \SpAMM{} using Intel's Math Kernel Library ({\tt MKL}) or AMD's Core Math Library ({\tt ACML}), our optimized version is found to be significantly faster. Detailed performance comparisons are made for quantum chemical matrices with differently structured sub-blocks. Finally, we discuss the potential of improved hardware prefetch to yield 2--3x speedups.
-
an optimized sparse approximate Matrix Multiply
2012Co-Authors: Nicolas Bock, Matt ChallacombeAbstract:Group T-1, Theoretical Division, Los Alamos National Laboratory, Los Alamos, NM 87544(Dated: March 22, 2012)We present an optimized single-precision implementation of the Sparse Approximate Matrix Mul-tiply (SpAMM) [M. Challacombe and N. Bock, arXiv 1011.3534 (2010)], a fast algorithm for Matrix-Matrix multiplication for matrices with decay that achieves an O(nlnn) computational complexitywith respect to Matrix dimension n. We nd that the max norm of the error achieved with a SpAMMtolerance below 2 810 is lower than that of the single-precision SGEMM for dense quantum chem-ical matrices, while outperforming SGEMM with a cross-over already for small matrices (n˘1000).Relative to naive implementations of SpAMM using Intel’s Math Kernel Library (MKL) or AMD’s CoreMath Library (ACML), our optimized version is found to be signi cantly faster. Detailed perfor-mance comparisons are made for quantum chemical matrices with di erently structured sub-blocks.Finally, we discuss the potential of improved hardware prefetch to yield 2{3x speedups.
-
a general parallel sparse blocked Matrix Multiply for linear scaling scf theory
Computer Physics Communications, 2000Co-Authors: Matt ChallacombeAbstract:A general approach to the parallel sparse-blocked Matrix–Matrix Multiply is developed in the context of linear scaling self-consistent-field (SCF) theory. The data-parallel message passing method uses non-blocking communication to overlap computation and communication. The space filling curve heuristic is used to achieve data locality for sparse Matrix elements that decay with “separation”. Load balance is achieved by solving the bin packing problem for blocks with variable size. With this new method as the kernel, parallel performance of the simplified density Matrix minimization (SDMM) for solution of the SCF equations is investigated for RHF/6-31G ∗∗ water clusters and RHF/3-21G estane globules. Sustained rates above 5.7 GFLOPS for the SDMM have been achieved for (H 2 O) 200 with 95 Origin 2000 processors. Scalability is found to be limited by load imbalance, which increases with decreasing granularity, due primarily to the inhomogeneous distribution of variable block sizes.
Roger M Smith - One of the best experts on this subject based on the ideXlab platform.
-
gemmw a portable level 3 blas winograd variant of strassen s Matrix Matrix Multiply algorithm
Journal of Computational Physics, 1994Co-Authors: Craig C Douglas, Michael A Heroux, Gordon Slishman, Roger M SmithAbstract:Abstract Matrix-Matrix multiplication is normally computed using one of the BLAS or a reinvention of part of the BLAS. Unfortunately, the BLAS were designed with small matrices in mind. When huge, well-conditioned matrices are multiplied together, the BLAS perform like the blahs, even on vector machines. For matrices where the coefficients are well conditioned. Winograd's variant of Strassen's algorithm offers some relief, but is rarely available in a quality form on most computers. We reconsider this method and offer a highly portable solution based on the Level 3 BLAS interface.
Sarom S Leang - One of the best experts on this subject based on the ideXlab platform.
-
quantum chemical calculations using accelerators migrating Matrix operations to the nvidia kepler gpu and the intel xeon phi
Journal of Chemical Theory and Computation, 2014Co-Authors: Sarom S Leang, Alistair P Rendell, Mark S GordonAbstract:Increasingly, modern computer systems comprise a multicore general-purpose processor augmented with a number of special purpose devices or accelerators connected via an external interface such as a PCI bus. The NVIDIA Kepler Graphical Processing Unit (GPU) and the Intel Phi are two examples of such accelerators. Accelerators offer peak performances that can be well above those of the host processor. How to exploit this heterogeneous environment for legacy application codes is not, however, straightforward. This paper considers how Matrix operations in typical quantum chemical calculations can be migrated to the GPU and Phi systems. Double precision general Matrix Multiply operations are endemic in electronic structure calculations, especially methods that include electron correlation, such as density functional theory, second order perturbation theory, and coupled cluster theory. The use of approaches that automatically determine whether to use the host or an accelerator, based on problem size, is explore...
-
quantum chemical calculations using accelerators migrating Matrix operations to the nvidia kepler gpu and the intel xeon phi
Journal of Chemical Theory and Computation, 2014Co-Authors: Sarom S Leang, Alistair P Rendell, Mark S GordonAbstract:Increasingly, modern computer systems comprise a multicore general-purpose processor augmented with a number of special purpose devices or accelerators connected via an external interface such as a PCI bus. The NVIDIA Kepler Graphical Processing Unit (GPU) and the Intel Phi are two examples of such accelerators. Accelerators offer peak performances that can be well above those of the host processor. How to exploit this heterogeneous environment for legacy application codes is not, however, straightforward. This paper considers how Matrix operations in typical quantum chemical calculations can be migrated to the GPU and Phi systems. Double precision general Matrix Multiply operations are endemic in electronic structure calculations, especially methods that include electron correlation, such as density functional theory, second order perturbation theory, and coupled cluster theory. The use of approaches that automatically determine whether to use the host or an accelerator, based on problem size, is explored, with computations that are occurring on the accelerator and/or the host. For data-transfers over PCI-e, the GPU provides the best overall performance for data sizes up to 4096 MB with consistent upload and download rates between 5-5.6 GB/s and 5.4-6.3 GB/s, respectively. The GPU outperforms the Phi for both square and nonsquare Matrix multiplications.