The Experts below are selected from a list of 288 Experts worldwide ranked by ideXlab platform
Manjunath Gorentla Venkata - One of the best experts on this subject based on the ideXlab platform.
-
SBAC-PAD - Accelerating OpenSHMEM Collectives Using In-Network Computing Approach
2019 31st International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), 2019Co-Authors: Manjunath Gorentla Venkata, Gil Bloch, Gilad Shainer, Richard GrahamAbstract:OpenSHMEM is one of the key programming models for High Performance Computing (HPC) applications with irregular communication patterns. Particularly, it is useful for problems that cannot be decomposed easily such as graph partitioning. The programming model supports Remote Memory Access (RMA), atomics, and Collective Operations. In this paper, we explore and evaluate the In-network Computing approach for accelerating the OpenSHMEM Collective Operations, particularly barrier, broadcast, and reduction Operations. To achieve acceleration, In-network Computing leverages hardware engines on the networking elements and effective software that can efficiently use these capabilities. We explore the value of this approach for Collective Operations on the InfiniBand Host Channel Adapters (HCAs) and switches. Particularly, we focus on the recently introduced Collective offload feature provided by the Mellanox Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) TM capability, which accelerates the barriers and reduction Operations; the multicast capability accelerates the broadcast Collective Operation. To leverage the hardware capabilities, we complement it with an effective software stack that includes Hierarchical Collectives (HCOLL) library, and SHARP layer. Our evaluation on Oak Ridge National Laboratory (ORNL)'s Summit system, which is the fastest supercomputer on the June 2019 Top 500 list, show that the hardware and software acceleration in the In-network Computing approach is key for achieving the performance and scalability required for Collectives and applications. For a 5120 process OpenSHMEM job, our results show that the barrier Operation is 710% faster, broadcast is 370% faster, and reduction Operation is 10 times faster when compared with the implementation of Collective Operations with no acceleration. Further, experiments with a 2D-Heat kernel show that the In-network Computing approach is very effective for realworld applications.
-
Accelerating OpenSHMEM Collectives Using In-Network Computing Approach
2019 31st International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), 2019Co-Authors: Manjunath Gorentla Venkata, Gil Bloch, Gilad Shainer, Richard GrahamAbstract:OpenSHMEM is one of the key programming models for High Performance Computing (HPC) applications with irregular communication patterns. Particularly, it is useful for problems that cannot be decomposed easily such as graph partitioning. The programming model supports Remote Memory Access (RMA), atomics, and Collective Operations. In this paper, we explore and evaluate the In-network Computing approach for accelerating the OpenSHMEM Collective Operations, particularly barrier, broadcast, and reduction Operations. To achieve acceleration, In-network Computing leverages hardware engines on the networking elements and effective software that can efficiently use these capabilities. We explore the value of this approach for Collective Operations on the InfiniBand Host Channel Adapters (HCAs) and switches. Particularly, we focus on the recently introduced Collective offload feature provided by the Mellanox Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) TM capability, which accelerates the barriers and reduction Operations; the multicast capability accelerates the broadcast Collective Operation. To leverage the hardware capabilities, we complement it with an effective software stack that includes Hierarchical Collectives (HCOLL) library, and SHARP layer. Our evaluation on Oak Ridge National Laboratory (ORNL)'s Summit system, which is the fastest supercomputer on the June 2019 Top 500 list, show that the hardware and software acceleration in the In-network Computing approach is key for achieving the performance and scalability required for Collectives and applications. For a 5120 process OpenSHMEM job, our results show that the barrier Operation is 710% faster, broadcast is 370% faster, and reduction Operation is 10 times faster when compared with the implementation of Collective Operations with no acceleration. Further, experiments with a 2D-Heat kernel show that the In-network Computing approach is very effective for realworld applications.
-
OpenSHMEM - An Evaluation of OpenSHMEM Interfaces for the Variable-Length Alltoallv Collective Operation
Lecture Notes in Computer Science, 2015Co-Authors: M. Graham Lopez, Pavel Shamis, Manjunath Gorentla VenkataAbstract:Alltoallv is a Collective Operation which allows all processes to exchange variable amounts of data with all other processes in the communication group. This means that Alltoallv requires not only $$ON^2$$ communications, but typically also additional exchanges of the data lengths that will be transmitted in the eventual Alltoallv call. This pre-exchange is used to calculate the proper offsets for the receiving buffers on the target processes. However, we propose two new candidate interfaces for Alltoallv that would mitigate the need for the user to set up this extra exchange of information at the possible cost of memory efficiency. We explain the new interface variants and show how a single call can be used in place of the traditional Alltoall/ Alltoallv pair. We then discuss the performance tradeoffs for overall communication and memory costs, as well as both software and hardware-based optimizations and their applicability to the various proposed interfaces.
-
an evaluation of openshmem interfaces for the variable length alltoallv Collective Operation
OpenSHMEM 2015 Revised Selected Papers of the Second Workshop on OpenSHMEM and Related Technologies. Experiences Implementations and Technologies - Vo, 2015Co-Authors: Graham M Lopez, Pavel Shamis, Manjunath Gorentla VenkataAbstract:Alltoallv is a Collective Operation which allows all processes to exchange variable amounts of data with all other processes in the communication group. This means that Alltoallv requires not only $$ON^2$$ communications, but typically also additional exchanges of the data lengths that will be transmitted in the eventual Alltoallv call. This pre-exchange is used to calculate the proper offsets for the receiving buffers on the target processes. However, we propose two new candidate interfaces for Alltoallv that would mitigate the need for the user to set up this extra exchange of information at the possible cost of memory efficiency. We explain the new interface variants and show how a single call can be used in place of the traditional Alltoall/ Alltoallv pair. We then discuss the performance tradeoffs for overall communication and memory costs, as well as both software and hardware-based optimizations and their applicability to the various proposed interfaces.
-
SLOAVx: Scalable LOgarithmic AlltoallV Algorithm for Hierarchical Multicore Systems
2013 13th IEEE ACM International Symposium on Cluster Cloud and Grid Computing, 2013Co-Authors: Cong Xu, Manjunath Gorentla Venkata, Richard L. Graham, Yandong Wang, Weikuan YuAbstract:Scientific applications use Collective communication Operations in Message Passing Interface (MPI) for global synchronization and data exchanges. Alltoall and AlltoallV are two important Collective Operations. They are used by MPI jobs to exchange messages among all of MPI processes. AlltoallV is a generalization of Alltoall, supporting messages of varying sizes. However, the existing MPI AlltoallV implementation has linear complexity, i.e., each process has to send messages to all other processes in the job. Such linear complexity can result in sub optimal scalability of MPI applications when they are deployed on millions of cores. To address above challenge, in this paper, we introduce a new Scalable LOgarithmic AlltoallV algorithm, named SLOAV, for MPI AlltoallV Collective Operation. SLOAV aims to achieve global exchange of small messages of different sizes in a logarithmic number of rounds. Furthermore, given the prevalence of multicore systems with shared memory, we design a hierarchical AlltoallV algorithm based on SLOAV by leveraging the advantages of shared memory, which is referred to as SLOAVx. Compared to SLOAV, SLOAVx significantly reduces the inter-node communication, thus improving the entire system performance and mitigating the impact of message latency. We have implemented and embedded both algorithms in Open MPI. Our evaluation on large-scale computer systems shows that for the 8-byte and 1024-process MPI Alltoallv Operation, the SLOAV can reduce the latency by as much as 86.4%, when compared to the state-of-the-art, and SLOAVx can further optimize the SLOAV by up to 83.1% in terms of message latency on multicore systems. In addition, experiments with NAS Parallel Benchmark (NPB) demonstrate that our algorithms are very effective for real-world applications.
Pavel Shamis - One of the best experts on this subject based on the ideXlab platform.
-
ConnectX2 In niBand Management Queues: New support for Network Of oaded
2020Co-Authors: Richard L. Graham, Pavel Shamis, Gil Bloch, Noam Bloch, Michael Kagan, Ishai Rabinovitz, Stephen W. Poole, Gilad ShainerAbstract:This paper introduces the newly developed InfiniBand (IB) Management Queue capability, used by the Host Channel Adapter (HCA) to manage network task data flow dependancies, and progress the communications associated with such flows. These tasks include sends, receives, and the newly supported wait task, and are scheduled by the HCA based on a data dependency description provided by the user. This functionality is supported by the ConnectX-2 HCA, and provides the means for delegating Collective communication management and progress to the HCA, also known as Collective communication offload. This provides a means for overlapping Collective communications managed by the HCA and computation on the Central Processing Unit (CPU), thus making it possible to reduce the impact of system noise on parallel applications using Collective Operations. This paper further describes how this new capability can be used to implement scalable Message Passing Interface (MPI) Collective Operations, describing the high level details of how this new capability is used to implement the MPI Barrier Collective Operation, focusing on the latency sensitive performance aspects of this new capability. This paper concludes with small scale benchmark experiments comparing implementations of the barrier Collective Operation, using the new network offload capabilities, with established point-to-point basedmore » implementations of these same algorithms, which manage the data flow using the central processing unit. These early results demonstrate the promise this new capability provides to improve the scalability of high-performance applications using Collective communications. The latency of the HCA based implementation of the barrier is similar to that of the best performing point-to-point based implementation managed by the central processing unit, starting to outperform these as the number of processes involved in the Collective Operation increases.« less
-
OpenSHMEM - An Evaluation of OpenSHMEM Interfaces for the Variable-Length Alltoallv Collective Operation
Lecture Notes in Computer Science, 2015Co-Authors: M. Graham Lopez, Pavel Shamis, Manjunath Gorentla VenkataAbstract:Alltoallv is a Collective Operation which allows all processes to exchange variable amounts of data with all other processes in the communication group. This means that Alltoallv requires not only $$ON^2$$ communications, but typically also additional exchanges of the data lengths that will be transmitted in the eventual Alltoallv call. This pre-exchange is used to calculate the proper offsets for the receiving buffers on the target processes. However, we propose two new candidate interfaces for Alltoallv that would mitigate the need for the user to set up this extra exchange of information at the possible cost of memory efficiency. We explain the new interface variants and show how a single call can be used in place of the traditional Alltoall/ Alltoallv pair. We then discuss the performance tradeoffs for overall communication and memory costs, as well as both software and hardware-based optimizations and their applicability to the various proposed interfaces.
-
an evaluation of openshmem interfaces for the variable length alltoallv Collective Operation
OpenSHMEM 2015 Revised Selected Papers of the Second Workshop on OpenSHMEM and Related Technologies. Experiences Implementations and Technologies - Vo, 2015Co-Authors: Graham M Lopez, Pavel Shamis, Manjunath Gorentla VenkataAbstract:Alltoallv is a Collective Operation which allows all processes to exchange variable amounts of data with all other processes in the communication group. This means that Alltoallv requires not only $$ON^2$$ communications, but typically also additional exchanges of the data lengths that will be transmitted in the eventual Alltoallv call. This pre-exchange is used to calculate the proper offsets for the receiving buffers on the target processes. However, we propose two new candidate interfaces for Alltoallv that would mitigate the need for the user to set up this extra exchange of information at the possible cost of memory efficiency. We explain the new interface variants and show how a single call can be used in place of the traditional Alltoall/ Alltoallv pair. We then discuss the performance tradeoffs for overall communication and memory costs, as well as both software and hardware-based optimizations and their applicability to the various proposed interfaces.
-
Exploring the All-to-All Collective Optimization Space with ConnectX CORE-Direct
2012 41st International Conference on Parallel Processing, 2012Co-Authors: Manjunath Gorentla Venkata, Richard L. Graham, Joshua Ladd, Pavel ShamisAbstract:The all-to-all Collective communication Operation is used by many scientific applications, and is one of the most time consuming and challenging Collective Operation to optimize. The algorithms for all-to-all Operations typically fall into two classes, logarithmic and linear scaling algorithms, with Bruck's algorithm, a logarithmic scaling algorithm, used in many small-data all-to-all implementations. The recent addition of InfiniBand CORE-Direct support for network management of Collective communications offers new opportunities for optimizing all-to-all Operation as well as supporting truly asynchronous implementations of these Operations. This paper presents several new enhancements to the Bruck small-data algorithm that leverage CORE-Direct and other InfiniBand network capabilities to produce efficient implementations of this Collective Operation. These include RDMA, SR-RNR, and SR-RTR algorithms. In addition, nonblocking implementations of these Collective Operations are also presented. Benchmark results show that the RDMA algorithm, which uses CORE-Direct capabilities to offload Collective communication management to the Host Channel Adapter (HCA), hardware gather support for sending non-continuous data, and low-latency RDMA semantics, performs the best. For a 64 processes and 128 byte-per-process all-to-all, the RDMA algorithm performs 27% better than Bruck's algorithm implementation in Open MPI and 136% better than the SR-RTR algorithm. In addition, the nonblocking versions of these algorithms have the same performance characteristics as the blocking algorithms. Finally, measurements of computation/communication overlap capacity show that all offloaded algorithms achieve about 98% overlap for large data all-to-all, whereas implementations using host-based progress achieve only about 9.5% overlap.
-
ConnectX-2 CORE-Direct Enabled Asynchronous Broadcast Collective Communications
2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum, 2011Co-Authors: Manjunath Gorentla Venkata, Pavel Shamis, Richard L. Graham, Ishai Rabinovitz, Joshua S. Ladd, Vasily Filipov, Gilad ShainerAbstract:This paper describes the design and implementation of InfiniBand (IB) CORE-Direct based blocking and nonblocking broadcast Operations within the Cheetah Collective Operation framework. It describes a novel approach that fully offloads Collective Operations and employs only user-supplied buffers. For a 64 rank communicator, the latency of CORE-Direct based hierarchical algorithm is better than production grade Message Passing Interface (MPI) implementations, 150% better than the default Open MPI algorithm and 115% better than the shared memory optimized MVAPICH implementation for a one kilo-byte (KB) message, and for eight mega-bytes (MB) it is 48% and 64% better, respectively. Flat-topology broadcast achieves 99.9% overlap in a polling based communication-computation test, and 95.1% overlap for a wait based test, compared with 92.4% and 17.0%, respectively, for a similar Central Processing Unit (CPU) based implementation.
D.k. Panda - One of the best experts on this subject based on the ideXlab platform.
-
Designing Non-blocking Broadcast with Collective Offload on InfiniBand Clusters: A Case Study with HPL
2011 IEEE 19th Annual Symposium on High Performance Interconnects, 2011Co-Authors: K. Kandalla, H. Subramoni, J. Vienne, Pai S. Raikar, K. Tomko, D.k. PandaAbstract:The upcoming MPI-3.0 standard is expected to include non-blocking Collective Operations. Non-blocking Collectives offer a new MPI interface, using which an application can decouple the initiation and completion of Collective Operations. However, to be effective, the MPI library should provide a high performance and scalable implementation. One of the major challenges in designing an effective non-blocking Collective Operation is to ensure progress of the Operation while processors are busy in application-level computation. The recently introduced Mellanox ConnectX-2 InfiniBand adapters offer a task offload interface (CORE-Direct) that enables communication progress without requiring CPU cycles. In this paper, we present the design of a non-blocking broadcast Operation (MPI Ibcast) using the CORE-Direct offload interface. Our experimental evaluations show that our implementation delivers near perfect overlap, without penalizing the latency of the MPI Ibcast Operation. Since existing MPI implementations do not provide non-blocking Collective communication, scientific applications have been modified to implement Collectives on top of MPI point-to-point Operations to achieve overlap. HPL is an example of an application use case scenario for non-blocking Collectives. We have explored the benefits of our proposed network offload based MPI Ibcast implementation with HPL and we observe that HPL can achieve its peak throughput with significantly smaller problem sizes, which also leads to an improvement in its run-time by up to 78%, with 512 processors. We also observe that our proposed designs can minimize the impact of system noise on applications.
-
Scaling alltoall Collective on multi-core systems
2008 IEEE International Symposium on Parallel and Distributed Processing, 2008Co-Authors: Rahul Kumar, Amith Mamidala, D.k. PandaAbstract:MPI_Alltoall is one of the most communication intense Collective Operation used in many parallel applications. Recently, the supercomputing arena has witnessed phenomenal growth of commodity clusters built using InfiniBand and multi-core systems. In this context, it is important to optimize this Operation for these emerging clusters to allow for good application scaling. However, optimizing MPI_Alltoall on these emerging systems is not a trivial task. InfiniBand architecture allows for varying implementations of the network protocol stack. For example, the protocol can be totally on-loaded to a host processing core or it can be off-loaded onto the NIC or can use any combination of the two. Understanding the characteristics of these different implementations is critical in optimizing a communication intense Operation such as MPI_Alltoall. In this paper, we systematically study these different architectures and propose new schemes for MPI_Alltoall tailored to these architectures. Specifically, we demonstrate that we cannot use one common scheme which performs optimally on each of these varying architectures. For example, on-loaded implementations can exploit multiple cores to achieve better network utilization, and in offload interfaces aggregation can be used to avoid congestion on multi-core systems. We employ shared memory aggregation techniques in these schemes and elucidate the impact of these schemes on multi-core systems. The proposed design achieves a reduction in MPI_Alltoall time by 55% for 512 Byte messages and speeds up the CPMD application by 33%.
-
Efficient SMP-aware MPI-level broadcast over InfiniBand's hardware multicast
Proceedings 20th IEEE International Parallel & Distributed Processing Symposium, 2006Co-Authors: A.r. Mamidala, Lei Chai, D.k. PandaAbstract:Most of the high-end computing clusters found today feature multi-way SMP nodes interconnected by an ultra-low latency and high bandwidth network. InfiniBand is emerging as a high-speed network for such systems. InfiniBand provides a scalable and efficient hardware multicast primitive to efficiently implement many MPI Collective Operations. However, employing hardware multicast as the communication method may not perform well in all cases. This is true especially when more than one process is running per node. In this context, shared memory channel becomes the desired communication medium within the node as it delivers latencies which are of an order of magnitude lower than the inter-node message latencies. Thus, to deliver optimal Collective performance, coupling hardware multicast with shared memory channel becomes necessary. In this paper we propose mechanisms to address this issue. On a 16-node 2-way SMP cluster, the Leader-based scheme proposed in this paper improves the performance of the MPI_Bcast Operation by a factor of as much as 2.3 and 1.8 when compared to the point-to-point and original solution employing only hardware multicast. We have also evaluated our designs on NUMA based system and obtained a performance improvement of 1.7 using our designs on 2-node 4-way system. We also propose a dynamic attach policy as an enhancement to this scheme to mitigate the impact of process skew on the performance of the Collective Operation
-
Efficient and scalable all-to-all personalized exchange for InfiniBand-based clusters
International Conference on Parallel Processing 2004. ICPP 2004., 2004Co-Authors: D.k. PandaAbstract:The all-to-all personalized exchange is the most dense Collective communication function offered by the MPI specification. The Operation involves every process sending a different message to all other participating processes. This Collective Operation is essential for many parallel scientific applications. With increasing system and message sizes, it becomes challenging to offer a fast, scalable and efficient implementation of this Operation. InfiniBand is an emerging modern interconnect. It offers very low latency, high bandwidth and one-sided Operations like RDMA write. Its advanced features like RDMA write gather allow us to design and implement all-to-all algorithms much more efficiently than in the past. Our aim in This work is to design efficient and scalable implementations of traditional personalized exchange algorithms. We present two novel approaches towards designing all-to-all algorithms for short and long messages respectively. The hypercube RDMA write gather and direct eager schemes effectively leverage the RDMA and RDMA with write gather mechanisms offered by InfiniBand. Performance evaluation of our design and implementation reveals that it is able to reduce the all-to-all communication time by upto a factor of 3.07 for 32 byte messages on a 16 node InfiniBand cluster. Our analytical models suggest that the proposed designs perform 64% better on InfiniBand clusters with 1024 nodes for 4k message size.
-
High performance and reliable NIC-based multicast over Myrinet/GM-2
2003 International Conference on Parallel Processing 2003. Proceedings., 2003Co-Authors: W. Yu, D. Buntinas, D.k. PandaAbstract:Multicast is an important Collective Operation for parallel programs. Some network interface cards (NICs), such as Myrinet, have programmable processors that can be programmed to support multicast. We propose a high performance and reliable NIC-based multicast scheme, in which a NIC-based multisend mechanism is used to send multiple replicas of a message to different destinations, and a NIC-based forwarding mechanism to forward the received packets without intermediate host involvement. We have explored different design alternatives and implemented the proposed scheme with the set of best alternatives over Myrinet/GM-2. MPICH-GM has also been modified to take advantage of this scheme. At the GM-level, the NIC-based multicast improves the multicast latency by a factor up to 1.48 for messages les 512 bytes, and a factor up to 1.86 for 16KB messages over 16 nodes compared to the traditional host-based multicast. Similar improvements are also achieved at the MPI level. In addition, it is demonstrated that NIC-based multicast is tolerant to process skew and has significant benefits for large systems
G. Iannello - One of the best experts on this subject based on the ideXlab platform.
-
Efficient implementation of reduce-scatter in MPI
Proceedings 10th Euromicro Workshop on Parallel Distributed and Network-based Processing, 2002Co-Authors: M. Bernaschi, G. Iannello, M. LauriaAbstract:We discuss the efficient implementation of the MPI Collective Operation called reduce-scatter. We describe the implementation issues and the performance characterization of two algorithms for the reduce-scatter that have been proven to be highly efficient in theory under the assumption of fully connected parallel system. A performance comparison with existing mainstream implementations of the Operation is presented which confirms the practical advantage of the new algorithms. Experiments show that the two algorithms have different characteristics which make them complementary in providing a performance gain over standard algorithms.
-
Efficient algorithms for the reduce-scatter Operation in LogGP
IEEE Transactions on Parallel and Distributed Systems, 1997Co-Authors: G. IannelloAbstract:We consider the problem of efficiently performing a reduce-scatter Operation in a message passing system. Reduce-scatter is the composition of an element-wise reduction on vectors of n elements initially held by n processors, with a scatter of the resulting vector among the processors. In this paper, we present two algorithms for the reduce-scatter Operation, designed in LogGP. The first algorithm assumes an associative and commutative reduction operator and it is optimal in LogGP within a small constant factor. The second algorithm allows the reduction operator to be noncommutative, and it is asymptotically optimal when values to be combined are large arrays. To achieve these results, we developed a complete analysis of both algorithms in LogGP, including the derivation of lower bounds for the reduce-scatter Operation, and the study of the m-item version of the problem, i.e., the case when the initial elements are vectors themselves. Reduce-scatter has been included as a Collective Operation in the MPI standard message passing library, and can be used, for instance, in parallel matrix-vector multiply when the matrix is decomposed by columns. To model a message passing system, we adopted the LogGP model, an extension of LogP that allows the modeling of messages of different length. While this choice makes the analysis somewhat more complex, it leads to more realistic results in the case of gather/scatter algorithms.
Gilad Shainer - One of the best experts on this subject based on the ideXlab platform.
-
ConnectX2 In niBand Management Queues: New support for Network Of oaded
2020Co-Authors: Richard L. Graham, Pavel Shamis, Gil Bloch, Noam Bloch, Michael Kagan, Ishai Rabinovitz, Stephen W. Poole, Gilad ShainerAbstract:This paper introduces the newly developed InfiniBand (IB) Management Queue capability, used by the Host Channel Adapter (HCA) to manage network task data flow dependancies, and progress the communications associated with such flows. These tasks include sends, receives, and the newly supported wait task, and are scheduled by the HCA based on a data dependency description provided by the user. This functionality is supported by the ConnectX-2 HCA, and provides the means for delegating Collective communication management and progress to the HCA, also known as Collective communication offload. This provides a means for overlapping Collective communications managed by the HCA and computation on the Central Processing Unit (CPU), thus making it possible to reduce the impact of system noise on parallel applications using Collective Operations. This paper further describes how this new capability can be used to implement scalable Message Passing Interface (MPI) Collective Operations, describing the high level details of how this new capability is used to implement the MPI Barrier Collective Operation, focusing on the latency sensitive performance aspects of this new capability. This paper concludes with small scale benchmark experiments comparing implementations of the barrier Collective Operation, using the new network offload capabilities, with established point-to-point basedmore » implementations of these same algorithms, which manage the data flow using the central processing unit. These early results demonstrate the promise this new capability provides to improve the scalability of high-performance applications using Collective communications. The latency of the HCA based implementation of the barrier is similar to that of the best performing point-to-point based implementation managed by the central processing unit, starting to outperform these as the number of processes involved in the Collective Operation increases.« less
-
SBAC-PAD - Accelerating OpenSHMEM Collectives Using In-Network Computing Approach
2019 31st International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), 2019Co-Authors: Manjunath Gorentla Venkata, Gil Bloch, Gilad Shainer, Richard GrahamAbstract:OpenSHMEM is one of the key programming models for High Performance Computing (HPC) applications with irregular communication patterns. Particularly, it is useful for problems that cannot be decomposed easily such as graph partitioning. The programming model supports Remote Memory Access (RMA), atomics, and Collective Operations. In this paper, we explore and evaluate the In-network Computing approach for accelerating the OpenSHMEM Collective Operations, particularly barrier, broadcast, and reduction Operations. To achieve acceleration, In-network Computing leverages hardware engines on the networking elements and effective software that can efficiently use these capabilities. We explore the value of this approach for Collective Operations on the InfiniBand Host Channel Adapters (HCAs) and switches. Particularly, we focus on the recently introduced Collective offload feature provided by the Mellanox Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) TM capability, which accelerates the barriers and reduction Operations; the multicast capability accelerates the broadcast Collective Operation. To leverage the hardware capabilities, we complement it with an effective software stack that includes Hierarchical Collectives (HCOLL) library, and SHARP layer. Our evaluation on Oak Ridge National Laboratory (ORNL)'s Summit system, which is the fastest supercomputer on the June 2019 Top 500 list, show that the hardware and software acceleration in the In-network Computing approach is key for achieving the performance and scalability required for Collectives and applications. For a 5120 process OpenSHMEM job, our results show that the barrier Operation is 710% faster, broadcast is 370% faster, and reduction Operation is 10 times faster when compared with the implementation of Collective Operations with no acceleration. Further, experiments with a 2D-Heat kernel show that the In-network Computing approach is very effective for realworld applications.
-
Accelerating OpenSHMEM Collectives Using In-Network Computing Approach
2019 31st International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), 2019Co-Authors: Manjunath Gorentla Venkata, Gil Bloch, Gilad Shainer, Richard GrahamAbstract:OpenSHMEM is one of the key programming models for High Performance Computing (HPC) applications with irregular communication patterns. Particularly, it is useful for problems that cannot be decomposed easily such as graph partitioning. The programming model supports Remote Memory Access (RMA), atomics, and Collective Operations. In this paper, we explore and evaluate the In-network Computing approach for accelerating the OpenSHMEM Collective Operations, particularly barrier, broadcast, and reduction Operations. To achieve acceleration, In-network Computing leverages hardware engines on the networking elements and effective software that can efficiently use these capabilities. We explore the value of this approach for Collective Operations on the InfiniBand Host Channel Adapters (HCAs) and switches. Particularly, we focus on the recently introduced Collective offload feature provided by the Mellanox Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) TM capability, which accelerates the barriers and reduction Operations; the multicast capability accelerates the broadcast Collective Operation. To leverage the hardware capabilities, we complement it with an effective software stack that includes Hierarchical Collectives (HCOLL) library, and SHARP layer. Our evaluation on Oak Ridge National Laboratory (ORNL)'s Summit system, which is the fastest supercomputer on the June 2019 Top 500 list, show that the hardware and software acceleration in the In-network Computing approach is key for achieving the performance and scalability required for Collectives and applications. For a 5120 process OpenSHMEM job, our results show that the barrier Operation is 710% faster, broadcast is 370% faster, and reduction Operation is 10 times faster when compared with the implementation of Collective Operations with no acceleration. Further, experiments with a 2D-Heat kernel show that the In-network Computing approach is very effective for realworld applications.
-
Scalable Hierarchical Aggregation Protocol (SHArP): A Hardware Architecture for Efficient Data Reduction
2016 First International Workshop on Communication Optimizations in HPC (COMHPC), 2016Co-Authors: Richard L. Graham, Gil Bloch, Gilad Shainer, Devendar Bureddy, Hal Rosenstock, Dror Goldenerg, Mike Dubman, Sasha Kotchubievsky, Vladimir Koushnir, Lion LeviAbstract:Increased system size and a greater reliance on utilizing system parallelism to achieve computational needs, requires innovative system architectures to meet the simulation challenges. As a step towards a new network class of co-processors - intelligent network devices, which manipulate data traversing the data-center network, this paper describes the SHArP technology designed to offload Collective Operation processing to the network. This is implemented in Mellanox's SwitchIB-2 ASIC, using innetwork trees to reduce data from a group of sources, and to distribute the result. Multiple parallel jobs with several partially overlapping groups are supported each with several reduction Operations in-flight. Large performance enhancements are obtained, with an improvement of a factor of 2.1 for an eight byte MPI_Allreduce() Operation on 128 hosts, going from 6.01 to 2.83 microseconds. Pipelining is used for an improvement of a factor of 3.24 in the latency of a 4096 byte MPI_Allreduce() Operations, declining from 46.93 to 14.48 microseconds.
-
ConnectX-2 CORE-Direct Enabled Asynchronous Broadcast Collective Communications
2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum, 2011Co-Authors: Manjunath Gorentla Venkata, Pavel Shamis, Richard L. Graham, Ishai Rabinovitz, Joshua S. Ladd, Vasily Filipov, Gilad ShainerAbstract:This paper describes the design and implementation of InfiniBand (IB) CORE-Direct based blocking and nonblocking broadcast Operations within the Cheetah Collective Operation framework. It describes a novel approach that fully offloads Collective Operations and employs only user-supplied buffers. For a 64 rank communicator, the latency of CORE-Direct based hierarchical algorithm is better than production grade Message Passing Interface (MPI) implementations, 150% better than the default Open MPI algorithm and 115% better than the shared memory optimized MVAPICH implementation for a one kilo-byte (KB) message, and for eight mega-bytes (MB) it is 48% and 64% better, respectively. Flat-topology broadcast achieves 99.9% overlap in a polling based communication-computation test, and 95.1% overlap for a wait based test, compared with 92.4% and 17.0%, respectively, for a similar Central Processing Unit (CPU) based implementation.