On Big Red, how do I run MPI jobs?

On this page:

• MPI libraries available through softenv
• Add the MPICH or Open MPI key to .soft
• Compile your MPI code
• Submitting a parallel job
• Warning about the $OBJECT_MODE environment variable
• Understanding MPI package names (softenv keys) on Big Red

MPI libraries available through softenv

To display the MPI libraries available on Big Red, use the softenv command and use grep to search for mpi.

The MPICH library is installed on Big Red. Among the other keys listed, you should be able to see two packages named +mpich-mx-ibm-32 and mpich-mx-ibm-64:

The key mpich-mx-ibm-32, for example, indicates that it is an MPICH library compiled with the IBM XL compiler in 32-bit mode and that it communicates with the mx protocol. The key openmpi-1.2-mx-ibm-64 indicates that it is an Open MPI library (version 1.2) that was compiled with the IBM XL compilers in 64-bit mode. For more, see Understanding MPI package names (softenv keys) on Big Red below.

Back to top

Add the MPICH or Open MPI key to .soft

To use MPICH-MX, add the MPICH-MX key to softenv. Add +mpich-mx-ibm-32 or +mpich-mx-ibm-64 to your ~/.soft file, depending on whether you need a 32-bit or a 64-bit MPICH library. Your ~/.soft file should look similar to this:

To use Open MPI, add the Open MPI key to softenv. Add +openmpi-1.2-mx-ibm-64 to your ~/.softfile. Your ~/.soft file should look similar to this:

If you're a TeraGrid user, your ~/.soft file should look similar to this:

If you are not familiar with the softenv system, see On Big Red and Quarry at IU, how can I use SoftEnv to customize my software environment? You may also enter man softenv-intro at the prompt on Big Red.

Back to top

Compile your MPI code

For example, to compile a 32-bit parallel C program (assuming you have +mpich-mx-ibm-32 in your ~/.soft file), you would use something like:

To compile a 64-bit parallel Fortran 77 program (assuming you have +mpich-mx-ibm-64 in your ~/.soft file), you would use something like:

To compile a 32-bit parallel C program using Open MPI 1.1.1 (assuming you have +openmpi-1.1.1-xlc-8.0-32 in your ~/.soft file):

Important: UITS strongly recommends using either the -q32 or the -q64 switch, depending on whether your code is 32- or 64-bit, when you compile and link your program. For more information, see the Warning about the $OBJECT_MODE environment variable below.

Back to top

Submitting a parallel job

You have three options for submitting a parallel job:

• Use parallejob for simple MPI applications.
• Run your job interactively for testing and debugging.
• Use LoadLeveler if you need the advanced mpirun options.

1. Submit parallel jobs using the paralleljob script.

   The paralleljob script provides a convenient method for submitting parallel (multiple-processor) programs to the LoadLeveler batching and queuing system. Programs must consist of just one executable file, in contrast to some master/worker programs in which the master and workers are different executable files. For complete documentation, see On Big Red at IU, how do I use the paralleljob script to submit jobs? or enter man paralleljob on Big Red. The general form of the command is:

   pw@BigRed:~/MPI_source_code> paralleljob ./hello
2. Run your job interactively with mpirun.

   To access one of the interactive nodes, you must first log into Big Red and from there use ssh to connect to b509, b510, b511, or b512. Create a machinefile, using your favorite editor, that looks similar to the following:

   janedoe@BigRed:~/MPI_source_code> cat mfile b509 b510 b509 b510 b509 b510 b509 b510

   Then use mpirun to run your parallel job:

   janedoe@BigRed:~/MPI_source_code> mpirun -np 8 -machinefile mfile program-name
   • For np, use the number of processes to start.
   • For mfile, use the name of the machinefile that you created with the editor.
   • For program-name, substitute the name of the program to submit.

   Note: The interactive nodes mentioned above are open to all users. If you get the MX error below, please try again. If the problem persists, contact High Performance Systems for help.

   MX:Aborting
   MX:s9c4b7:send req(already completed):req status 8:Remote endpoint is closed

3. Submit a parallel job with a LoadLeveler script.

   To submit a parallel job that runs your MPI program, edit the sample LoadLeveler script shown below (or create your own) with the correct number of nodes and tasks, the appropriate output/error files, etc.

   Note: If you have a TeraGrid allocation on Big Red, you must edit the account_no line, replacing NONE with your TeraGrid account number.

   The sample script, parallel_job.sh, follows:

   #! /bin/bash -l ## LoadLeveler script to submit 2 node, 4 task MPI program: hello # @ job_type = MPICH # @ class = LONG # @ account_no = NONE # @ node = 2 # @ tasks_per_node = 4 # @ wall_clock_limit = 10:00:00 # @ notification = always # @ notify_user = <email_id> # @ environment=COPY_ALL; # @ output = hello.$(cluster).$(process).out # @ error = hello.$(cluster).$(process).err # @ queue ## Users should always cd into their execution directory due to ## a bug within LoadLeveler in dealing with the initialdir keyword. # cd <execution directory> cd ${HOME}/my_project ## Use mpirun to execute your MPICH program in parallel; ## $LOADL_TOTAL_TASKS and $LOADL_HOSTFILE are defined by ## LoadLeveler for jobs of type MPICH. mpirun -np $LOADL_TOTAL_TASKS -machinefile $LOADL_HOSTFILE ./hello

   Use llsubmit to submit parallel_job.sh:

   pw@BigRed:~/MPI_source_code> llsubmit parallel_job.sh

   You will see text similar to the following, which confirms that your job has been submitted:

   llsubmit: Processed command file through Submit Filter: "/home/load. . . llsubmit: The job "s10c2b5.dim.2577" has been submitted.

   If you are not familiar with job submission on Big Red, see the "Submitting batch jobs to LoadLeveler" section of Getting started on Big Red.

Back to top

Warning about the $OBJECT_MODE environment variable

Note: If you use the -q32 or the -q64 switch as recommended above, you may safely ignore this section.

If you do not use the -q32 or -q64 switch, be aware of the $OBJECT_MODE environment variable, because it also indicates to the IBM XL compilers whether an application you compile is 32-bit or 64-bit.

The OBJECT_MODE variable has the following effects when you compile your program:

• Setting OBJECT_MODE to 64 will force the compiler to compile 64-bit object files. It is equivalent to explicitly specifying the -q64 switch to the compiler.
  
  
• Not setting OBJECT_MODE or setting it to 32 will force the compiler to compile 32-bit object files. It is equivalent to explicitly specifying the -q32 switch to the compiler.

UITS recommends that you avoid potential problems by using the -q32 or -q64 switch, depending on whether you want to compile a 32-bit or a 64-bit application.

The OBJECT_MODE environment variable does not affect the execution of a program.

Back to top

Understanding MPI package names (softenv keys) on Big Red

Big Red is a 64-bit system that supports both 32-bit and 64-bit applications. However, 32-bit object files cannot be linked with 64-bit ones.

The MPICH libraries and the Open MPI libraries on Big Red are compiled with IBM's XL compiler suite in both 32- and 64-bit modes. Communication protocols you can use include mx (Myrinet Express) and TCP/IP over the Myrinet high-performance network.

Note: Applications using the mx protocol usually get better performance.

This document was developed with support from the National Science Foundation (NSF) under Grant No. 0503697 to the University of Chicago and subcontracted to Indiana University. Additional support was provided by IU through its participation in the TeraGrid, which is supported by the NSF under Grants No. 0833618, SCI451237, SCI535258, and SCI504075. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.

Back to top

Comments/Questions/Corrections