On Big Red, how do I run MPI jobs?

Username:
Password:

On this page:

MPI libraries available through `softenv`

To display the MPI libraries available on Big Red, use the softenv command, and use grep to search for mpi.

The MPICH library is installed on Big Red. Among the other keys listed, you should see packages named +mpich-mx-ibm-32 and mpich-mx-ibm-64:

janedoe@BigRed:~/> softenv | grep mpi . . . +mpich-mx-ibm-32 MPICH MX xlc/xlf 32bit +mpich-mx-ibm-64 MPICH MX xlc/xlf 64bit . . . +openmpi-1.2.6-ibm-32 Openmpi-1.2.6 Ibm compilers 32-bit +openmpi-1.2.6-ibm-64 Openmpi-1.2.6 Ibm compilers 64-bit +openmpi-1.3.3-ibm-64 Openmpi-1.3.3 Ibm compilers 64-bit +openmpi-1.4.1-ibm-32 OpenMPI v1.4.1, IBM compilers +openmpi-1.4.1-ibm-64 OpenMPI v1.4.1, IBM compilers . . .

The key mpich-mx-ibm-32, for example, indicates that it is an MPICH library compiled with the IBM XL compiler in 32-bit mode, and that it communicates with the mx protocol. The key openmpi-1.2-mx-ibm-64 indicates that it is an Open MPI library (version 1.2) that was compiled with the IBM XL compilers in 64-bit mode. For more, see Understanding MPI package names (softenv keys) on Big Red below.

Add the MPICH or Open MPI key to `.soft`

To use MPICH-MX, add the MPICH-MX key to softenv. Add +mpich-mx-ibm-32 or +mpich-mx-ibm-64 to your ~/.soft file, depending on whether you need a 32-bit or a 64-bit MPICH library. Your ~/.soft file should look similar to this:

+mpich-mx-ibm-32 ## Some more keys you may have @bigred

To use Open MPI, add the Open MPI key to softenv. Add +openmpi-1.4.1-ibm-64 to your ~/.softfile. Your ~/.soft file should look similar to this:

+openmpi-1.4.1-ibm-64 ## Some more keys you may have @bigred

If you are not familiar with the softenv system, see On Big Red and Quarry at IU, how can I use SoftEnv to customize my software environment? You may also enter man softenv-intro at the prompt on Big Red.

Compile your MPI code

To compile a 32-bit parallel C program (assuming you have +mpich-mx-ibm-32 in your ~/.soft file), you would use something like:

janedoe@BigRed:~/MPI_source_code> mpicc -q32 -o myprog myprog.c -L/opt/mx/lib32

To compile a 64-bit parallel Fortran 77 program (assuming you have +mpich-mx-ibm-64 in your ~/.soft file), you would use something like:

janedoe@BigRed:~/MPI_source_code> mpif77 -q64 -o myprog myprog.f -L/opt/mx/lib64

To compile a 32-bit parallel C program using Open MPI 1.1.1 (assuming you have +openmpi-1.1.1-xlc-8.0-32 in your ~/.soft file), you would use something like:

janedoe@BigRed:~/MPI_source_code> mpicc -q32 -o myprog myprog.c

Important: UITS strongly recommends using either the -q32 or the -q64 switch, depending on whether your code is 32- or 64-bit, when you compile and link your program. For more, see the Warning about the $OBJECT_MODE environment variable below.

Submitting a parallel job

You have three options for submitting a parallel job:

Use paralleljob for simple MPI applications.
Run your job interactively for testing and debugging.
Use LoadLeveler if you need the advanced mpirun options.

Submitting parallel jobs using the `paralleljob` script

The paralleljob script provides a convenient method for submitting parallel (multiple-processor) programs to the LoadLeveler batching and queuing system. Programs must consist of just one executable file, in contrast to some master/worker programs in which the master and workers are different executable files. For more, see On Big Red at IU, how do I use the paralleljob script to submit jobs? or enter man paralleljob on Big Red. The general form of the command is:

pw@BigRed:~/MPI_source_code> paralleljob ./hello

Running your job interactively with `mpirun`

To access one of the interactive nodes, you must first log into Big Red, and from there use ssh to connect to b509, b510, b511, or b512. Using your favorite editor, create a machinefile that looks similar to:

janedoe@BigRed:~/MPI_source_code> cat mfile b509 b510 b509 b510 b509 b510 b509 b510

Then, use mpirun to run your parallel job:

janedoe@BigRed:~/MPI_source_code> mpirun -np 8 -machinefile mfile program-name

For np, use the number of processes to start. For mfile, use the name of the machinefile that you created with the editor. For program-name, substitute the name of the program to submit.

Note: The interactive nodes mentioned above are open to all users. If you get the MX error below, try again:

MX:Aborting
MX:s9c4b7:send req(already completed):req status 8:Remote endpoint is closed

If the problem persists, email High Performance Systems for help.

Submitting a parallel job with a LoadLeveler script

To submit a parallel job that runs your MPI program, edit the sample LoadLeveler script shown below (or create your own) with the correct number of nodes and tasks, and the appropriate output/error files.

The sample script, parallel_job.sh, follows:

#! /bin/bash -l ## LoadLeveler script to submit 2 node, 4 task MPI program: hello # @ job_type = MPICH # @ class = LONG # @ account_no = NONE # @ node = 2 # @ tasks_per_node = 4 # @ wall_clock_limit = 10:00:00 # @ notification = always # @ notify_user = <email_id> # @ environment=COPY_ALL; # @ output = hello.$(cluster).$(process).out # @ error = hello.$(cluster).$(process).err # @ queue ## Users should always cd into their execution directory due to ## a bug within LoadLeveler in dealing with the initialdir keyword. # cd <execution directory> cd ${HOME}/my_project ## Use mpirun to execute your MPICH program in parallel; ## $LOADL_TOTAL_TASKS and $LOADL_HOSTFILE are defined by ## LoadLeveler for jobs of type MPICH. mpirun -np $LOADL_TOTAL_TASKS -machinefile $LOADL_HOSTFILE ./hello

Use llsubmit to submit parallel_job.sh:

pw@BigRed:~/MPI_source_code> llsubmit parallel_job.sh

You will see text similar to the following, which confirms that your job has been submitted:

llsubmit: Processed command file through Submit Filter: "/home/load. . . llsubmit: The job "s10c2b5.dim.2577" has been submitted.

If you are not familiar with job submission on Big Red, see the "Submitting batch jobs to LoadLeveler" section of Getting started on Big Red.

Warning about the $OBJECT_MODE environment variable

Note: If you use the -q32 or the -q64 switch as recommended above, you may safely ignore this section.

If you do not use the -q32 or -q64 switch, be aware of the $OBJECT_MODE environment variable, because it also indicates to the IBM XL compilers whether an application you compile is 32-bit or 64-bit.

The OBJECT_MODE variable has the following effects when you compile your program:

Setting OBJECT_MODE to 64 will force the compiler to compile 64-bit object files. It is equivalent to explicitly specifying the -q64 switch to the compiler.
Not setting OBJECT_MODE or setting it to 32 will force the compiler to compile 32-bit object files. It is equivalent to explicitly specifying the -q32 switch to the compiler.

UITS recommends using the -q32 or -q64 switch, depending on whether you want to compile a 32-bit or a 64-bit application, to avoid potential problems.

The OBJECT_MODE environment variable does not affect the execution of a program.

Understanding MPI package names (`softenv` keys) on Big Red

Big Red is a 64-bit system that supports both 32-bit and 64-bit applications. However, 32-bit object files cannot be linked with 64-bit ones.

The MPICH libraries and the Open MPI libraries on Big Red are compiled with IBM's XL compiler suite in both 32- and 64-bit modes. Communication protocols you can use include mx (Myrinet Express) and TCP/IP over the Myrinet high-performance network.

Note: Applications using the mx protocol usually get better performance.

This document was developed with support from the National Science Foundation (NSF) under Grant No. 0503697 to the University of Chicago and subcontracted to Indiana University. Additional support was provided by IU through its participation in the TeraGrid, which is supported by the NSF under Grants No. 0833618, SCI451237, SCI535258, and SCI504075. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.

This is document autn in domain all.
Last modified on July 29, 2011.