GASNet--A High-Performance Portable Communication System for Global Address Space Languages

Dan Bonachea, Christian Bell1, and Mike Welcome2
(Professor Katherine A. Yelick)
Department of Energy Office of Science, National Science Foundation, and NSA

GASNet [1] is a network-independent and language-independent high-performance communication interface intended for use in implementing the runtime system for global address space languages, such as Unified Parallel C (UPC) [2] or Titanium [3]. GASNet stands for "Global-Address Space Networking."

Goals of this research include:

  • Language-independence: compatibility with several global-address space languages and compilers
    -UPC, Titanium, Co-array Fortran, and possibly others
    -Hide language- or compiler-specific details, such as shared-pointer representation
  • Hardware independence: a variety of parallel architectures and operating systems
    -SMP: Linux/UNIX SMPs, Origin 2000, etc.
    -Clusters of uniprocessors or SMPs: IBM SP, Compaq AlphaServer, Linux/UNIX clusters, etc.
  • Explicit support for multi-threading/multi-process for supporting clusters of SMPs
  • Support many high-performance networks: MPI, Myrinet/GM, Quadrics/elan, IBM/LAPI, Infiniband
  • Ease of implementation on new hardware
    -Allow quick prototype implementations
    -Production implementations can leverage performance features of hardware
  • Provide both portability and high performance

    Figure 1 shows the organization of the GASNet communication layer and its relation to the parallel language system.

    The GASNet interface is divided into two distinct layers:

    GASNet Core API

  • Provides the most basic required network primitives
  • Implemented directly on each platform
    -Minimal set of network functions needed to support a working implementation
    -General enough to implement everything else
  • Based heavily on active messages paradigm
    -Provides powerful extensibility mechanism

    GASNet Extended API

  • Wider interface that includes more complicated operations
    -Put, get, barrier, collectives, etc.
    -Flexible and expressive non-blocking operations and synchronization mechanisms
    -Ideal compilation target for aggressive communication optimizations
  • We provide a reference implementation of the extended API in terms of the core API
  • Implementors can choose to directly implement any subset for performance leverage hardware support for higher-level operations

    GASNet has already been implemented on Myrinet/GM, Quadrics/elan, IBM/LAPI, and MPI (a fully portable implementation), and implementations are on the way for Infiniband, Gigabit Ethernet and other high-performance networks. This research effort includes carefully evaluating the performance characteristics for the target high-performance networks and interfaces [4], and developing strategies for best utilizing the network capabilities to implement low-latency, low-overhead, one-sided small put/gets and zero-copy high-bandwidth bulk transfers [5].

    Figures 2 and 3 demonstrate the small-put/get "round-trip" latency and large-put/get bandwidth achieved by the Myrinet and Quadrics implementations of GASNet in various configurations and on several machines. The leftmost bar for each machine shows the performance of the fully portable GASNet-MPI implementation on that system (MPI-based core API, reference implementation of extended API). The middle bar for the Quadrics machines shows the performance of using the reference extended API with a natively-implemented GASNet-Quadrics core API. The rightmost bar for each machine shows the performance of a fully-native implementation of the GASNet core and extended API on the underlying network. We find this performance to be very competitive with the best performance achievable on these networks, therefore justifying the claim that GASNet is indeed a very lightweight, high-performance interface.

    Figures 4 and 5 show the small-put/get "round-trip" latency for the fully-native GASNet-Quadrics and GASNet-Myrinet implementations over varying message sizes.

    This work is done in conjunction with the Berkeley UPC compiler group, a joint effort between Lawrence Berkeley National Laboratory and UC Berkeley. GASNet is already being used as a communication system for the Berkeley UPC compiler and the UC Berkeley Titanium compiler, and several other global address space language research efforts are considering adopting GASNet.


    Figure 1: High-level system organization for a GAS language using GASNet

    Figure 2: Small put/get "round-trip" latency performance of GASNet-Myrinet and GASNet-Quadrics in several configurations and machines

    Figure 3: Large put/get bandwidth performance of GASNet-Myrinet and GASNet-Quadrics in several configurations and machines

    Figure 4: Put/get "round-trip" latency performance of GASNet-Quadrics over varying message size

    Figure 5: Put/get "round-trip" latency performance of GASNet-Myrinet over varying message size

    [1]
    D. Bonachea, GASNet Specification, v1.1, UC Berkeley Computer Science Division, Report No. UCB/CSD 02/1207, October 2002.
    [2]
    T. El-Ghazawi, W. Carlson, and J. Draper, UPC Language Specification, v1.0, George Washington University, February 2001.
    [3]
    P. Hilfinger, D. Bonachea, D. Gay, S. Graham, B. Libeit, G. Pike, and K. Yelick, Titanium Language Reference Manual, UC Berkeley Computer Science Division, Report No. UCB/CSD 01/1163, November 2001.
    [4]
    C. Bell, D. Bonachea, Y. Cote, J. Duell, P. Hargrove, P. Husbands, C. Iancu, M. Welcome, and K. Yelick, "An Evaluation of Current High-Performance Networks," Int. Parallel and Distributed Processing Symp., 2003 (submitted).
    [5]
    D. Bonachea and C. Bell, "Efficient DMA Registration Strategies for Pinning-based High Performance Networks," Workshop in Communication Architecture for Clusters, 2003 (submitted).
    1Visiting Researcher, LBNL
    2Visiting Researcher, LBNL

    More information (http://upc.nersc.gov) or

    Send mail to the author : (bonachea@eecs.berkeley.edu)

    Edit this abstract