Cluster Grid

Font Size



Menu Style


MPI job submission using gcc

To run mpi programs using gcc, you must first load the right compiler and the right mpi lib. This is done by using:

module load compilers/gcc-4.1.2
module load mpi/openmpi-1.3.2_gcc-4.1.2

A list of all available modules is found by using

module avail



  1. compile program
  2. decide what queue you're runnning, what paralell environment, how many slots
  3. wait program to finish, view output

[alexandru.herisanu@fep-53-3 mpi]$ qsub -q ibm-quad.q -pe openmpi 4 -cwd
module load compilers/gcc-4.1.2
module load mpi/openmpi-1.3.2_gcc-4.1.2

mpirun -np $NSLOTS ./mpi_scatter
Your job 12 ("STDIN") has been submitted
[alexandru.herisanu@fep-53-3 mpi]$ watch qstat


Last step, view output:


[alexandru.herisanu@fep-53-3 mpi]$ cat STDIN.*12
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
rank= 1  Results: 5.000000 6.000000 7.000000 8.000000
rank= 0  Results: 1.000000 2.000000 3.000000 4.000000
rank= 3  Results: 13.000000 14.000000 15.000000 16.000000
rank= 2  Results: 9.000000 10.000000 11.000000 12.000000


Let me explain the qsub options needed:

-q selects the queue, the group of machines to be used (TBD: add link to all available queues)

ibm-quad.q, ibm-opteron.q, ibm-nehalem.q, cell-qs22.q

-pe requests multiple job slots (paralell environment). This environment can use a tight or loose integration with sge. This basically specifies what mpi flavor to use and how to schedule them. *1 means maximum one mpi process per machine. I want 4 slots using openmpi means

ex: -pe openmpi 4

hpmpi, intelmpi, intelmpi*1, openmpi, openmpi*1

-cwd change current working directory to the one from the job was submitted. This means you can use ./executable to run the program from the script, instead of using the whole absolute path


mpirun -np $NSLOTS ./mpi_scatter

- the $NSLOTS variable here is the one from qsub (-pe openmpi 4).

- we can use ./mpi_scatter because we used -cwd