About Quadrics
Customers
Presentations
Contact Details
Employment
Overview
QsNet
QsTenG
RMS
Linux Software
QsEm for Sky-X & Sky-Y
HPC Services&Solutions
Overview
Software Download
Documentation
Platform Compliance
3rd Party Tools and Applications
External Resources
Gnats
FAQ
Case Studies
Performance Results
Benchmarks
Features & Benefits
Documents Library
Overview
QsNetII E-Series Pricing System
Partners
Channel Partners
Configurations
 
Home   Screensaver   Legal   Login
 
   
 
About Us Products Support Performance Buy Quadrics
 
 

Quadrics Linux Software


As the dominant interconnect technology in the world's top 10 supercomputers, QsNet is recognized as being the most capable high performance network. In order to consolidate and extend this position, and to promote the development of high-performance commodity clusters, Quadrics has made the core components of the QsNet software release for Linux under the GNU Public License. OEM customers and end-users are able to use these sources to support new versions of the Linux kernel; port to new architectures; add functionality and install modified versions.


Quadrics Linux Open Source Release

The following components form the Quadrics open source release for Linux:

  • QsNet kernel device modules (including IP).

  • QsNet user libraries.

  • QsNet MPI libraries.

  • QsNet diagnostic utilities.

  • QsNet co-processor development kit.

    This software is distributed via the Quadrics Web site www.quadrics.com, sources are provided under the GNU public license. Quadrics provides support of this software to OEM customers and end users.


    • QsNet kernel device modules

    The kernel device drivers and modules to support the QsNet hardware are provided as pre-built kernel kits based on the current Red Hat distributions for each of the supported architectures (Pentium®, Itanium®, and Alpha) and the SuSe distribution for Opteron™. Sources are provided for the benefit of sites using other releases of Linux or wishing to port to new architectures. The following modules are provided:

    elan3
    The Elan3 driver. Manages device configuration, interrupts and network error correction protocols. Provides support for direct user access to QsNet - OS bypass.

    elan4
    The Elan4 driver. Manages device configuration, interrupts and network error correction protocols. Provides support for direct user access to QsNet - OS bypass.

    elan
    Device independent Elan operations including Elan capability management and stats collection.

    ep
    Elan kernel communications. Provides kernel message passing and cluster membership services for QsNet. Includes support for multiple network rails.

    eip
    Provides IP over QsNet using Elan kernel communications.

    rms
    Maintains information about the processes that are part of each parallel program. Supports signal delivery and scheduling operations on parallel programs rather than individual processes.

    jtag
    Parallel port driver for the QsNet switch control interface. Required for systems with directly connected switches.

    qsnet
    Utility module containing code common to the other Quadrics modules.



    QsNet user libraries

    The qsnetlibs package provides user libraries supporting direct access to the QsNet hardware. The libraries include support for DMAs, message passing, synchronization and collectives (see the Elan Programming Manual for details). A wide range of optimisations are supported:

  • Elan DMA for remote put/get

  • Elan queuing DMA for message passing

  • Local shared memory for message passing within a node

  • Support for multiple QsNet data networks, including rail striping and optimized collectives

  • QsNet hardware broadcast and synchronization support.

    The qsnetlibs package includes an optimized implementation of Shmem for QsNet.


    • QsNet MPI libraries

    Quadrics provide an enhanced version of the ANL/MPICH MPI reference port, which has been implemented directly upon the low latency libraries provided as part of the QsNet product. MPI-2 features include one-sided communication and MPI/IO. Quadrics MPI includes support for TotalView™ message queue debugging.


    QsNet diagnostic utilities

    The qsnetdiags package contains utilities used to check the configuration and integrity of the QsNet network.


    QsNet co-processor development kit

    The QsNet PCI adaptor card (Elan) incorporates a programmable RISC based co-processor. Kernel and user level programmers can write and load new code for this unit using the GNU based development tools. Note: this kit is only required by user and kernel developers who will be recompiling the qsnetlibs libraries or writing new co-processor code. It is not needed to run parallel jobs or use/compile against MPI on a QsNet system.


    QsNet documentation

    QsNet manuals are provided in .html and .pdf format on the Quadrics website. The following manuals are available:

  • Elan Kernel Communications Manual.
  • Elan Programming Manual.
  • QsNet Installation and Diagnostics Manual.
  • RMS Reference Manual.
  • RMS User Guide.
  • Shmem Programming Manual.


    • Resource Management System

    The Quadrics Resource Management System (RMS) provides centralized management of a Linux cluster connected by QsNet or Ethernet. RMS features include scheduling of parallel jobs, scalable job startup, parallel run time (stdio forwarding, signal delivery etc) and job clean up, together with cluster-wide access control and accounting. The RMS scheduler provides priority base gang scheduling and timeslicing of parallel jobs. RMS also includes subsystems for switch network management, node status monitoring (with automatic configuring out of failed nodes) and event handling. RMS provides integration with the LSF® and OpenPBS batch systems and the TotalViewTM parallel debugger. RMS is licensed on a per-CPU basis


    Linux Software Availability

    Quadrics software is available for Pentium®, and Alpha™ systems running Red Hat Linux, Itanium® systems running Red Hat Advanced Server and for Opteron systems running SuSE Linux. Support is also provided for the vanilla Linux kernel. See www.quadrics.com/linux for up-to-date information on supported versions. Installed systems range from small clusters to some of the world's largest Linux supercomputers at Lawrence Livermore and Pacific Northwest National Labs.Quadrics provides third level software support of QsNet products to OEM customers and end-users. Additional services include system configuration, installation planning, performance tuning, second level support and custom software development. See www.quadrics.com/support for our web-based fault reporting system and for details of our support products.


    Linpack performance of QsNet Linux systems

    System CPU typeCPUsLinpack
    PNNL 1.5 GHz Itanium 2 1936 8633 GF
    LLNL -MCR 2.4 GHz Pentium 4 23047634 GF
    LLNL - ALC 2.4 GHz Pentium 4 1920 6585 GF



    "Cluster File Systems, Inc. (CFS) maintains the Lustre file system, with optimized support for the Quadrics QsNet high performance interconnect. Lustre is a scalable cluster file system with POSIX semantics, no single points of failure and high performance (see www.lustre.org). Lustre is currently being installed on clusters with QsNet of approximately 1,000 nodes. Lustre is layered on the Portals package (see www.sandiaportals.org). Portals has pluggable device support, in the form of Portals Network Abstraction Layers (NAL's) which Lustre uses to run over many different and heterogeneous networks, including QsNet. Lawrence Livermore National Laboratory and Cluster File Systems have written a Portals NAL which is used by Lustre, and included in the Lustre and Sandia portals software. Initial indications of performance are that, in optimal situations, Lustre can write up to 220MBytes/s from a single client over QsNet and that I/O throughput scales linearly when more Object Storage Targets are added."

    Peter J Braam, CEO, Cluster File Systems Inc.

    "The type of Science and Engineering calculations required by Livermore's national security mission require a cluster of this size and a very high bandwidth and low-latency interconnect that provides demonstrable and scalable performance. The 11.2 teraflop/s system will significantly expand the computing resources available to Livermore's unclassified researchers."

    Mark Seager, Asst. Dept. Head for TeraScale Systems, Lawrence Livermore National Laboratory

    QsNet performance is dependent upon the host PCI interface. Performance figures given in this document are indicative of what can be achieved, but do not represent a commitment for any particular system.
    TotalView™ is a trademark of Etnus LLC.
    LSF® is a trademark of Platform Computing Inc.
    Pentium® and Itanium® are trademarks of Intel Corporation.
    Opteron™ is a trademark of Advanced Micro Devices.
    Linux is a trademark of Linus Torvalds.
    Red Hat is a trademark of Red Hat Inc.
    All other trademarks are the property of their respective owners.


    Download Quadrics Linux Brochure

    > Legal