Krylov Subspace

14,000,000 Leading Edge Experts on the ideXlab platform

Scan Science and Technology

Contact Leading Edge Experts & Companies

Scan Science and Technology

Contact Leading Edge Experts & Companies

The Experts below are selected from a list of 360 Experts worldwide ranked by ideXlab platform

James Demmel - One of the best experts on this subject based on the ideXlab platform.

  • s-Step Krylov Subspace Methods as Bottom Solvers for Geometric Multigrid
    2014 IEEE 28th International Parallel and Distributed Processing Symposium, 2014
    Co-Authors: Samuel Williams, Mike Lijewski, Ann Almgren, Brian Van Straalen, Erin Carson, Nicholas Knight, James Demmel
    Abstract:

    Geometric multigrid solvers within adaptive mesh refinement (AMR) applications often reach a point where further coarsening of the grid becomes impractical as individual sub domain sizes approach unity. At this point the most common solution is to use a bottom solver, such as BiCGStab, to reduce the residual by a fixed factor at the coarsest level. Each iteration of BiCGStab requires multiple global reductions (MPI collectives). As the number of BiCGStab iterations required for convergence grows with problem size, and the time for each collective operation increases with machine scale, bottom solves in large-scale applications can constitute a significant fraction of the overall multigrid solve time. In this paper, we implement, evaluate, and optimize a communication-avoiding s-step formulation of BiCGStab (CABiCGStab for short) as a high-performance, distributed-memory bottom solver for geometric multigrid solvers. This is the first time s-step Krylov Subspace methods have been leveraged to improve multigrid bottom solver performance. We use a synthetic benchmark for detailed analysis and integrate the best implementation into BoxLib in order to evaluate the benefit of a s-step Krylov Subspace method on the multigrid solves found in the applications LMC and Nyx on up to 32,768 cores on the Cray XE6 at NERSC. Overall, we see bottom solver improvements of up to 4.2x on synthetic problems and up to 2.7x in real applications. This results in as much as a 1.5x improvement in solver performance in real applications.

  • a residual replacement strategy for improving the maximum attainable accuracy of s step Krylov Subspace methods
    SIAM Journal on Matrix Analysis and Applications, 2014
    Co-Authors: Erin Carson, James Demmel
    Abstract:

    Krylov Subspace methods are a popular class of iterative methods for solving linear systems with large, sparse matrices. On modern computer architectures, both sequential and parallel performance of classical Krylov methods is limited by costly data movement, or communication, required to update the approximate solution in each iteration. These motivated communication-avoiding Krylov methods, based on $s$-step formulations, reduce data movement by a factor of $O(s)$ by reordering the computations in classical Krylov methods to exploit locality. Studies on the finite precision behavior of communication-avoiding Krylov methods in the literature have thus far been empirical in nature; in this work, we provide the first quantitative analysis of the maximum attainable accuracy of communication-avoiding Krylov Subspace methods in finite precision. Following the analysis for classical Krylov methods, we derive a bound on the deviation of the true and updated residuals in communication-avoiding conjugate gradient...

  • avoiding communication in nonsymmetric lanczos based Krylov Subspace methods
    SIAM Journal on Scientific Computing, 2013
    Co-Authors: Erin Carson, Nicholas Knight, James Demmel
    Abstract:

    Krylov Subspace methods are iterative methods for solving large, sparse linear systems and eigenvalue problems in a variety of scientific domains. On modern computer architectures, communication, or movement of data, takes much longer than the equivalent amount of computation. Classical formulations of Krylov Subspace methods require data movement in each iteration, creating a performance bottleneck, and thus increasing runtime. This motivated $s$-step, or communication-avoiding, Krylov Subspace methods, which only perform data movement every $O(s)$ iterations. We present new communication-avoiding Krylov Subspace methods, CA-BICG and CA-BICGSTAB. We are the first to provide derivations of these methods. For both sequential and parallel implementations, our methods reduce data movement by a factor of $O(s)$ versus the classical algorithms. We implement various polynomial bases and perform convergence experiments to enable comparison with the classical algorithm. We discuss recent results in improving both...

  • communication avoiding Krylov Subspace methods
    2010
    Co-Authors: James Demmel, Mark Hoemmen
    Abstract:

    Krylov Subspace methods (KSMs) are iterative algorithms for solving large, sparse linear systems and eigenvalue problems. Current KSMs rely on sparse matrix-vector multiply (SpMV) and vector-vector operations (like dot products and vector sums). All of these operations are communication-bound. Furthermore, data dependencies between them mean that only a small amount of that communication can be hidden. Many important scientific and engineering computations spend much of their time in Krylov methods, so the performance of many codes could be improved by introducing KSMs that communicate less. Our goal is to take s steps of a KSM for the same communication cost as 1 step, which would be optimal. We call the resulting KSMs "communication-avoiding Krylov methods." This thesis makes the following contributions: (1) We have fast kernels replacing SpMV, that can compute the results of s calls to SpMV for the same communication cost as one call (Section 2.1). (2) We have fast dense kernels as well, such as Tall Skinny QR (TSQR – Section 2.3) and Block Gram-Schmidt (BGS – Section 2.4), which can do the work of Modified Gram-Schmidt applied to s vectors for a factor of Θ(s2) fewer messages in parallel, and a factor of Θ(s/W) fewer words transferred between levels of the memory hierarchy (where W is the fast memory capacity in words). (3) We have new communication-avoiding Block Gram-Schmidt algorithms for orthogonalization in more general inner products (Section 2.5). (4) We have new communication-avoiding versions of the following Krylov Subspace methods for solving linear systems: the Generalized Minimum Residual method (GMRES – Section 3.4), both unpreconditioned and preconditioned, and the Method of Conjugate Gradients (CG), both unpreconditioned (Section 5.4) and left-preconditioned (Section 5.5). (5) We have new communication-avoiding versions of the following Krylov Subspace methods for solving eigenvalue problems, both standard (Ax = λx, for a nonsingular matrix A) and "generalized" (Ax = λMx, for nonsingular matrices A and M): Arnoldi iteration (Section 3.3), and Lanczos iteration, both for Ax = λx (Section 4.2) and Ax = λMx (Section 4.3). (6) We propose techniques for developing communication-avoiding versions of nonsymmetric Lanczos iteration (for solving nonsymmetric eigenvalue problems Ax = λx) and the Method of Biconjugate Gradients (BiCG) for solving linear systems. (7) We can combine more stable numerical formulations that use different bases of Krylov Subspaces with our techniques for avoiding communication. For a discussion of different bases, see Chapter 7. To see an example of how the choice of basis affects the formulation of the Krylov method, see Section 3.2.2. (8) We have faster numerical formulations. For example, in our communication-avoiding version of GMRES, CA-GMRES (see Section 3.4), we can pick the restart length r independently of the s-step basis length s. Experiments in Section 3.5.5 show that this ability improves numerical stability. We show in Section 3.6.3 that it also improves performance in practice, resulting in a 2.23× speedup in the CA-GMRES implementation described below. (9) We combine all of these numerical and performance techniques in a shared-memory parallel implementation of our communication-avoiding version of GMRES, CA-GMRES. Compared to a similarly highly optimized version of standard GMRES, when both are running in parallel on 8 cores of an Intel Clovertown (see Appendix A), CA-GMRES achieves 4.3× speedups over standard GMRES on standard sparse test matrices (described in Appendix B.5). When both are running in parallel on 8 cores of an Intel Nehalem (see Appendix A), CA-GMRES achieves 4.1× speedups. See Section 3.6 for performance results and Section 3.5 for corresponding numerical experiments. We first reported performance results for this implementation on the Intel Clovertown platform in Demmel et al. [78]. (10) We have incorporated preconditioning into our methods. Note that we have not yet developed practical communication-avoiding preconditioners; this is future work. We have accomplished the following: (a) We show (in Sections 2.2 and 4.3) what the s-step basis should compute in the preconditioned case for many different types of Krylov methods and s-step bases. We explain why this is hard in Section 4.3. (b) We have identified two different structures that a preconditioner may have, in order to achieve the desired optimal reduction of communication by a factor of s. See Section 2.2 for details. (Abstract shortened by UMI.)

Oliver G Ernst - One of the best experts on this subject based on the ideXlab platform.

  • implementation of a restarted Krylov Subspace method for the evaluation of matrix functions
    Linear Algebra and its Applications, 2008
    Co-Authors: M Afanasjew, Michael Eiermann, Oliver G Ernst, Stefan Guttel
    Abstract:

    A new implementation of restarted Krylov Subspace methods for evaluating f (A)b for a function f ,a matrix A and a vector b is proposed. In contrast to an implementation proposed previously, it requires constant work and constant storage space per restart cycle. The convergence behavior of this scheme is discussed and a new stopping criterion based on an error indicator is given. The performance of the implementation is illustrated for three parabolic initial value problems, requiring the evaluation of exp(A)b.

  • a restarted Krylov Subspace method for the evaluation of matrix functions
    SIAM Journal on Numerical Analysis, 2006
    Co-Authors: Michael Eiermann, Oliver G Ernst
    Abstract:

    We show how the Arnoldi algorithm for approximating a function of a matrix times a vector can be restarted in a manner analogous to restarted Krylov Subspace methods for solving linear systems of equations. The resulting restarted algorithm reduces to other known algorithms for the reciprocal and the exponential functions. We further show that the restarted algorithm inherits the superlinear convergence property of its unrestarted counterpart for entire functions and present the results of numerical experiments.

  • geometric aspects of the theory of Krylov Subspace methods
    Acta Numerica, 2001
    Co-Authors: Michael Eiermann, Oliver G Ernst
    Abstract:

    The development of Krylov Subspace methods for the solution of operator equations has shown that two basic construction principles underlie the most commonly used algorithms: the orthogonal residual (OR) and minimal residual (MR) approaches. It is shown that these can both be formulated as techniques for solving an approximation problem on a sequence of nested Subspaces of a Hilbert space, an abstract problem not necessarily related to an operator equation. Essentially all Krylov Subspace algorithms result when these Subspaces form a Krylov sequence. The well-known relations among the iterates and residuals of MR/OR pairs are shown to hold also in this rather general setting. We further show that a common error analysis for these methods involving the canonical angles between Subspaces allows many of the known residual and error bounds to be derived in a simple and consistent manner. An application of this analysis to compact perturbations of the identity shows that MR/OR pairs of Krylov Subspace methods converge q-superlinearly when applied to such operator equations.

Valeria Simoncini - One of the best experts on this subject based on the ideXlab platform.

  • an implicitly restarted Krylov Subspace method for real symmetric skew symmetric eigenproblems
    Linear Algebra and its Applications, 2012
    Co-Authors: Volker Mehrmann, Christian Schroder, Valeria Simoncini
    Abstract:

    Abstract A new implicitly-restarted Krylov Subspace method for real symmetric/skew-symmetric generalized eigenvalue problems is presented. The new method improves and generalizes the SHIRA method of Mehrmann and Watkins (2001) [37] to the case where the skew-symmetric matrix is singular. It computes a few eigenvalues and eigenvectors of the matrix pencil close to a given target point. Several applications from control theory are presented and the properties of the new method are illustrated by benchmark examples.

  • Krylov Subspace methods for projected Lyapunov equations
    Applied Numerical Mathematics, 2012
    Co-Authors: Tatjana Stykel, Valeria Simoncini
    Abstract:

    We consider the numerical solution of projected Lyapunov equations using Krylov Subspace iterative methods. Such equations play a fundamental role in balanced truncation model reduction of descriptor systems. We present generalizations of the extended block and global Arnoldi methods to projected Lyapunov equations and compare these methods with the alternating direction implicit method with respect to performance on different examples. A deflation strategy is also proposed to overcome possible breakdown in the recurrence.

  • convergence analysis of the extended Krylov Subspace method for the lyapunov equation
    Numerische Mathematik, 2011
    Co-Authors: Leonid Knizhnerman, Valeria Simoncini
    Abstract:

    The extended Krylov Subspace method has recently arisen as a competitive method for solving large-scale Lyapunov equations. Using the theoretical framework of orthogonal rational functions, in this paper we provide a general a priori error estimate when the known term has rank-one. Special cases, such as symmetric coefficient matrix, are also treated. Numerical experiments confirm the proved theoretical assertions.

  • recent computational developments in Krylov Subspace methods for linear systems
    Numerical Linear Algebra With Applications, 2007
    Co-Authors: Valeria Simoncini, Daniel B Szyld
    Abstract:

    Many advances in the development of Krylov Subspace methods for the iterative solution of linear systems during the last decade and a half are reviewed. These new developments include different versions of restarted, augmented, deflated, flexible, nested, and inexact methods. Also reviewed are methods specifically tailored to systems with special properties such as special forms of symmetry and those depending on one or more parameters. Copyright © 2006 John Wiley & Sons, Ltd.

  • on the occurrence of superlinear convergence of exact and inexact Krylov Subspace methods
    Siam Review, 2005
    Co-Authors: Valeria Simoncini, Daniel B Szyld
    Abstract:

    Krylov Subspace methods often exhibit superlinear convergence. We present a general analytic model which describes this superlinear convergence, when it occurs. We take an invariant Subspace approach, so that our results apply also to inexact methods, and to nondiagonalizable matrices. Thus, we provide a unified treatment of the superlinear convergence of GMRES, conjugate gradients, block versions of these, and inexact Subspace methods. Numerical experiments illustrate the bounds obtained.

Erin Carson - One of the best experts on this subject based on the ideXlab platform.

  • communication avoiding Krylov Subspace methods in theory and practice
    2015
    Co-Authors: Erin Carson
    Abstract:

    Advancements in the field of high-performance scientific computing are necessary to address the most important challenges we face in the 21st century. From physical modeling to large-scale data analysis, engineering efficient code at the extreme scale requires a critical focus on reducing communication -- the movement of data between levels of memory hierarchy or between processors over a network -- which is the most expensive operation in terms of both time and energy at all scales of computing. Achieving scalable performance thus requires a dramatic shift in the field of algorithm design, with a key area of innovation being the development of communication-avoiding algorithms.Solvers for sparse linear algebra problems, ubiquitous throughout scientific and mathematical applications, often limit application performance due to a low computation/communication ratio. Among iterative methods, Krylov Subspace methods are the most general and widely-used. To alleviate performance bottlenecks, much prior work has focused on the development of communication-avoiding Krylov Subspace methods, which can offer asymptotic performance improvements over a set number of iterations. In finite precision, the convergence and stability properties of classical Krylov methods are not necessarily maintained by communication-avoiding Krylov methods. Depending on the parameters used and the numerical properties of the problem, these communication-avoiding variants can exhibit slower convergence and decreased accuracy compared to their classical counterparts, making it unclear when communication-avoiding Krylov Subspace methods are suitable for use in practice. Until now, the literature on communication-avoiding Krylov methods lacked a detailed numerical stability analysis, as well as both theoretical and practical comparisons with the stability and convergence properties of standard implementations. In this thesis, we address this major challenge to the practical use of communication-avoiding Krylov Subspace methods. We extend a number of theoretical results and algorithmic techniques developed for classical Krylov Subspace methods to communication-avoiding Krylov Subspace methods and identify constraints under which these methods are competitive in terms of both achieving asymptotic speedups and meeting application-specific numerical requirements.

  • s-Step Krylov Subspace Methods as Bottom Solvers for Geometric Multigrid
    2014 IEEE 28th International Parallel and Distributed Processing Symposium, 2014
    Co-Authors: Samuel Williams, Mike Lijewski, Ann Almgren, Brian Van Straalen, Erin Carson, Nicholas Knight, James Demmel
    Abstract:

    Geometric multigrid solvers within adaptive mesh refinement (AMR) applications often reach a point where further coarsening of the grid becomes impractical as individual sub domain sizes approach unity. At this point the most common solution is to use a bottom solver, such as BiCGStab, to reduce the residual by a fixed factor at the coarsest level. Each iteration of BiCGStab requires multiple global reductions (MPI collectives). As the number of BiCGStab iterations required for convergence grows with problem size, and the time for each collective operation increases with machine scale, bottom solves in large-scale applications can constitute a significant fraction of the overall multigrid solve time. In this paper, we implement, evaluate, and optimize a communication-avoiding s-step formulation of BiCGStab (CABiCGStab for short) as a high-performance, distributed-memory bottom solver for geometric multigrid solvers. This is the first time s-step Krylov Subspace methods have been leveraged to improve multigrid bottom solver performance. We use a synthetic benchmark for detailed analysis and integrate the best implementation into BoxLib in order to evaluate the benefit of a s-step Krylov Subspace method on the multigrid solves found in the applications LMC and Nyx on up to 32,768 cores on the Cray XE6 at NERSC. Overall, we see bottom solver improvements of up to 4.2x on synthetic problems and up to 2.7x in real applications. This results in as much as a 1.5x improvement in solver performance in real applications.

  • a residual replacement strategy for improving the maximum attainable accuracy of s step Krylov Subspace methods
    SIAM Journal on Matrix Analysis and Applications, 2014
    Co-Authors: Erin Carson, James Demmel
    Abstract:

    Krylov Subspace methods are a popular class of iterative methods for solving linear systems with large, sparse matrices. On modern computer architectures, both sequential and parallel performance of classical Krylov methods is limited by costly data movement, or communication, required to update the approximate solution in each iteration. These motivated communication-avoiding Krylov methods, based on $s$-step formulations, reduce data movement by a factor of $O(s)$ by reordering the computations in classical Krylov methods to exploit locality. Studies on the finite precision behavior of communication-avoiding Krylov methods in the literature have thus far been empirical in nature; in this work, we provide the first quantitative analysis of the maximum attainable accuracy of communication-avoiding Krylov Subspace methods in finite precision. Following the analysis for classical Krylov methods, we derive a bound on the deviation of the true and updated residuals in communication-avoiding conjugate gradient...

  • avoiding communication in nonsymmetric lanczos based Krylov Subspace methods
    SIAM Journal on Scientific Computing, 2013
    Co-Authors: Erin Carson, Nicholas Knight, James Demmel
    Abstract:

    Krylov Subspace methods are iterative methods for solving large, sparse linear systems and eigenvalue problems in a variety of scientific domains. On modern computer architectures, communication, or movement of data, takes much longer than the equivalent amount of computation. Classical formulations of Krylov Subspace methods require data movement in each iteration, creating a performance bottleneck, and thus increasing runtime. This motivated $s$-step, or communication-avoiding, Krylov Subspace methods, which only perform data movement every $O(s)$ iterations. We present new communication-avoiding Krylov Subspace methods, CA-BICG and CA-BICGSTAB. We are the first to provide derivations of these methods. For both sequential and parallel implementations, our methods reduce data movement by a factor of $O(s)$ versus the classical algorithms. We implement various polynomial bases and perform convergence experiments to enable comparison with the classical algorithm. We discuss recent results in improving both...

Leonid Knizhnerman - One of the best experts on this subject based on the ideXlab platform.

  • art adaptive residual time restarting for Krylov Subspace matrix exponential evaluations
    Journal of Computational and Applied Mathematics, 2020
    Co-Authors: Mikhail Aleksandrovich Botchev, Leonid Knizhnerman
    Abstract:

    Abstract In this paper a new restarting method for Krylov Subspace matrix exponential evaluations is proposed. Since our restarting technique essentially employs the residual, some convergence results for the residual are given. We also discuss how the restart length can be adjusted after each restart cycle, which leads to an adaptive restarting procedure. Numerical tests arepresented to compare our restarting with three other restarting methods. Some of the algorithms described in this paper are a part of the Octave/Matlab package expmARPACK available at http://team.kiam.ru/botchev/expm/ .

  • art adaptive residual time restarting for Krylov Subspace matrix exponential evaluations
    arXiv: Numerical Analysis, 2018
    Co-Authors: Mikhail Aleksandrovich Botchev, Leonid Knizhnerman
    Abstract:

    In this paper a new restarting method for Krylov Subspace matrix exponential evaluations is proposed. Since our restarting technique essentially employs the residual, some convergence results for the residual are given. We also discuss how the restart length can be adjusted after each restart cycle, which leads to an adaptive restarting procedure. Numerical tests are presented to compare our restarting with three other restarting methods. Some of the algorithms described in this paper are a part of the Octave/Matlab package expmARPACK available at this http URL

  • convergence analysis of the extended Krylov Subspace method for the lyapunov equation
    Numerische Mathematik, 2011
    Co-Authors: Leonid Knizhnerman, Valeria Simoncini
    Abstract:

    The extended Krylov Subspace method has recently arisen as a competitive method for solving large-scale Lyapunov equations. Using the theoretical framework of orthogonal rational functions, in this paper we provide a general a priori error estimate when the known term has rank-one. Special cases, such as symmetric coefficient matrix, are also treated. Numerical experiments confirm the proved theoretical assertions.

  • Krylov Subspace approximation of eigenpairs and matrix functions in exact and computer arithmetic
    Numerical Linear Algebra With Applications, 1995
    Co-Authors: Vladimir Druskin, Leonid Knizhnerman
    Abstract:

    Many researchers are now working on computing the product of a matrix function and a vector,using approximations in a Krylov Subspace. We review our results on the analysis of one implemen-tation of that approach for symmetric matrices, which we call the Spectral Lanczos DecompositionMethod (SLDM).We have proved a general convergence estimate, relating SLDM error bounds to those obtainedthrough approximation of the matrix function by a part of its Chebyshev series. Thus, we arrivedat e ective estimates for matrix functions arising when solving parabolic, hyperbolic and ellipticpartial di erential equations. We concentrate on the parabolic case, where we obtain estimatesthat indicate superconvergence of SLDM. For this case a combination of SLDM and splittingmethods is also considered and some numerical results are presented.We implement our general estimates to obtain convergence bounds of Lanczos approximationsto eigenvalues in the internal part of the spectrum. Unlike Kaniel-Saad estimates, our estimatesare independent of the set of eigenvalues between the required one and the nearest spectrumbound.We consider an extension of our general estimate to the case of the simple Lanczos method(without reorthogonalization) in nite computer arithmetic which shows that for a moderatedimension of the Krylov Subspace the results, proved for the exact arithmetic, are stable up toroundo .

Related terms

Krylov Subspace - 15 days free trial to Access Experts & Abstracts