Running MPI jobs
================

There are several available MPI implementations on our group cluster:

- OpenMPI provided by the ``openHPC`` module, e.g. ``module load openmpi3/3.1.4``
- OpenMPI provided by the ``nvHPC`` module, e.g. ``module load nvhpc/21.3``
- OpenMPI compiled for the ``intel`` module, e.g. ``module load openmpi/3.1.4/intel``
- Intel MPI provided by the ``intel`` module, e.g. ``module load mpi/latest``

There are several ways of launching an MPI application within a SLURM
allocation, e.g. ``srun``, ``mpirun``, ``mpiexec`` and ``mpiexec.hydra``.
Unfortunately, the best way to launch your program depends on the MPI
implementation (and possibly your application), and choosing the wrong command
can severly affect the efficiency of your parallel run. Our recommendation is
the following:

Intel MPI
---------

With Intel MPI, we have found that ``mpirun`` can incorrectly distribute the
MPI ranks among the allocated nodes. We thus recommend using ``srun``::

    $ srun --mpi=pmi2 ./my_application

OpenMPI
-------

With OpenMPI, ``mpirun`` seems to be working correctly. Also, it seems that
``srun`` fails to launch your application in parallel, so here we recommend
using ``mpirun``::

    $ mpirun ./my_application