OpenMPI on SubMIT

Tags: OpenMPI

The Message Passing Interface (MPI) is a standardized and widely used communication protocol designed for parallel computing in distributed-memory systems. It enables processes running on different nodes of a cluster to exchange data efficiently, making it essential for high-performance computing (HPC) applications. OpenMPI is a popular open-source implementation of the MPI standard, offering high flexibility, scalability, and performance optimizations. Developed collaboratively by the HPC community, OpenMPI supports multiple network interfaces and integrates seamlessly with modern supercomputing environments. Its modular architecture allows users to tailor configurations for specific hardware, making it a preferred choice for researchers and engineers running large-scale simulations, numerical computations, and machine learning workloads.

Here we briefly introduce the OpenMPI installation on SubMIT and provide some examples of running your program with it.

Centralized installation of OpenMPI on SubMIT

We provide a centralized OpenMPI installation through the module system. One can load the OpenMPI module using:

module load mpi

you can check if it is successfully loaded by running:

module list
mpirun --version

Running MPI Programs in C/C++

Here is a simple example of an MPI program in C (mpi_hello.c):

#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
    MPI_Init(&argc, &argv);
    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    printf("Hello from process %d out of %d\n", rank, size);
    MPI_Finalize();
    return 0;
}

Compile the code using:

mpicc -o mpi_hello mpi_hello.c  # For C
mpicxx -o mpi_hello mpi_hello.cpp  # For C++

Run the program with:

mpirun -np 4 ./mpi_hello

Running MPI Programs in Python

To use OpenMPI with Python, one can install mpi4py or equivalent python packages. We recommended you to install it with conda using the following command:

conda install -c conda-forge mpi4py openmpi=4.1.*=external_*

The last part of the command specifies that the system-provided OpenMPI libraries will be used. Otherwise, conda will try to install its own version of OpenMPI or other MPI distributions (which usually will work as well, but there is no guarantee that they are optimized as the system-provide one).

Please also install the ucx libary (as the necessary point-to-point messaging layer):

conda install -c conda-forge ucx

Here is a simple Python MPI example (mpi_example.py):

from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

print(f"Hello from process {rank} out of {size}")

Run the script using:

mpirun -np 4 python mpi_example.py

Submitting Jobs to the Slurm

An example submission script to Slurm is:

#!/bin/bash
#SBATCH --job-name=test
#SBATCH --nodes=3
#SBATCH --ntasks=12
#SBATCH --cpus-per-task=1
#SBATCH --time=00:20:00
#SBATCH --mem-per-cpu=100

module load mpi
mpirun -np $SLURM_NTASKS ./my_program

Here we request 3 nodes with 12 tasks in total. Each task has 1 CPU, so we are only doing multi-processing here and no multi-threading.

More Testing scripts for OpenMPI

More testing scripts for OpenMPI can be found at:

https://github.com/mit-submit/submit-examples/blob/main/openmpi/

To use them, load the OpenMPI module on SubMIT and run the make file to compile the codes.