MPI job submission using gcc

Details
Hits: 7902

To run mpi programs using gcc, you must first load the right compiler and the right mpi lib. This is done by using:

module load compilers/gcc-4.1.2
module load mpi/openmpi-1.3.2_gcc-4.1.2

A list of all available modules is found by using

module avail

 

Quicksteps:

  1. compile program
  2. decide what queue you're runnning, what paralell environment, how many slots
  3. wait program to finish, view output


[alexandru.herisanu@fep-53-3 mpi]$ qsub -q ibm-quad.q -pe openmpi 4 -cwd
module load compilers/gcc-4.1.2
module load mpi/openmpi-1.3.2_gcc-4.1.2
mpirun -np $NSLOTS ./mpi_scatter
Your job 12 ("STDIN") has been submitted
[alexandru.herisanu@fep-53-3 mpi]$ watch qstat

 

Last step, view output:

 

[alexandru.herisanu@fep-53-3 mpi]$ cat STDIN.*12
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
rank= 1  Results: 5.000000 6.000000 7.000000 8.000000
rank= 0  Results: 1.000000 2.000000 3.000000 4.000000
rank= 3  Results: 13.000000 14.000000 15.000000 16.000000
rank= 2  Results: 9.000000 10.000000 11.000000 12.000000

 

Let me explain the qsub options needed:

-q selects the queue, the group of machines to be used (TBD: add link to all available queues)

ibm-quad.q, ibm-opteron.q, ibm-nehalem.q, cell-qs22.q

-pe requests multiple job slots (paralell environment). This environment can use a tight or loose integration with sge. This basically specifies what mpi flavor to use and how to schedule them. *1 means maximum one mpi process per machine. I want 4 slots using openmpi means

ex: -pe openmpi 4

hpmpi, intelmpi, intelmpi*1, openmpi, openmpi*1

-cwd change current working directory to the one from the job was submitted. This means you can use ./executable to run the program from the script, instead of using the whole absolute path

 

mpirun -np $NSLOTS ./mpi_scatter

- the $NSLOTS variable here is the one from qsub (-pe openmpi 4).

- we can use ./mpi_scatter because we used -cwd