Cluster Grid

Font Size



Menu Style


MPI job submission using gcc

To run mpi programs using gcc, you must first load the right compiler and the right mpi lib. This is done by using:

module load compilers/gcc-4.1.2
module load mpi/openmpi-1.3.2_gcc-4.1.2

A list of all available modules is found by using

module avail



  1. compile program
  2. decide what queue you're runnning, what paralell environment, how many slots
  3. wait program to finish, view output

[alexandru.herisanu@fep-53-3 mpi]$ qsub -q ibm-quad.q -pe openmpi 4 -cwd
module load compilers/gcc-4.1.2
module load mpi/openmpi-1.3.2_gcc-4.1.2

mpirun -np $NSLOTS ./mpi_scatter
Your job 12 ("STDIN") has been submitted
[alexandru.herisanu@fep-53-3 mpi]$ watch qstat


Last step, view output:


[alexandru.herisanu@fep-53-3 mpi]$ cat STDIN.*12
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
rank= 1  Results: 5.000000 6.000000 7.000000 8.000000
rank= 0  Results: 1.000000 2.000000 3.000000 4.000000
rank= 3  Results: 13.000000 14.000000 15.000000 16.000000
rank= 2  Results: 9.000000 10.000000 11.000000 12.000000


Let me explain the qsub options needed:

-q selects the queue, the group of machines to be used (TBD: add link to all available queues)

ibm-quad.q, ibm-opteron.q, ibm-nehalem.q, cell-qs22.q

-pe requests multiple job slots (paralell environment). This environment can use a tight or loose integration with sge. This basically specifies what mpi flavor to use and how to schedule them. *1 means maximum one mpi process per machine. I want 4 slots using openmpi means

ex: -pe openmpi 4

hpmpi, intelmpi, intelmpi*1, openmpi, openmpi*1

-cwd change current working directory to the one from the job was submitted. This means you can use ./executable to run the program from the script, instead of using the whole absolute path


mpirun -np $NSLOTS ./mpi_scatter

- the $NSLOTS variable here is the one from qsub (-pe openmpi 4).

- we can use ./mpi_scatter because we used -cwd


Intel MPI Howto pe clusterul NCIT

In cadrul clusterului NCIT, avem la dispozitie compilatorul Intel 11.1 - 0.38 si libraria IntelMPI 3.2.1 - 009.

Pentru a folosi IntelMPI trebuie sa selectam compilatorul si libraria mpi. Acest lucru se face fie prin comanda module:


[heri@fep-53-2 ~]$ module load compilers/intel-11.1_038
[heri@fep-53-2 ~]$ module load mpi/intelmpi-3.2.1_009


sau direct, prin folosirea scripturilor de mediu venite cu compiler-ul. In bash:


[heri@fep-53-2 ~]$ . /opt/intel/Compiler/11.1/038/bin/ intel64 (pt. C/C++)
[heri@fep-53-2 ~]$ . /opt/intel/Compiler/11.1/038/bin/ intel64 (pt. Fortran)

[heri@fep-53-2 ~]$ . /opt/intel/impi/3.2.1/bin64/ (pt. 64bit arhitecture - default)

[heri@fep-53-2 ~]$ . /opt/intel/impi/3.2.1/bin/ (pt. 32biti, nesuportat pe clusterul  nostru)


In cadrul SGE, shell-ul default este csh, iar scripturile de mediu sunt:

[heri@fep-53-2 ~]$ . /opt/intel/Compiler/11.1/038/bin/iccvars.csh intel64 (pt. C/C++)
[heri@fep-53-2 ~]$ . /opt/intel/Compiler/11.1/038/bin/ifortvars.csh intel64 (pt. Fortran)

[heri@fep-53-2 ~]$ . /opt/intel/impi/3.2.1/bin64/mpivars.csh (pt. 64bit arhitecture - default)

[heri@fep-53-2 ~]$ . /opt/intel/impi/3.2.1/bin/mpivars.csh (pt. 32biti, nesuportat pe clusterul  nostru)


Environment-ul paralel pentru IntelMPI este intelmpi si intelmpi*1.



source /opt/intel/Compiler/11.1/038/bin/iccvars.csh intel64
source /opt/intel/impi/3.2.1/bin64/mpivars.csh


mpiexec -np $NSLOTS ./hello_world.exe


Acest script il salvam sub numele de si se ruleaza cu comanda


qsub -q [coada] -pe [paralell_environment] [nr_sloturi] -cwd

Ex: qsub -q ibm-quad.q -pe openmpi 4 -cwd


(Mai mult detalii despre setup in acest articol).

Job-ul se urmareste cu comanda qstat, sau dupa preferinta watch qstat, iar output-ul se afla in fisierele myscript.*


MPI job submission using SunStudio 12

The default compiler, Sun Studio 12, is installed in /opt/sun/sunstudio12.

[heri@fep ~]$ cc -V
cc: Sun C 5.9 Linux_i386 Patch 124871-01 2007/07/31
usage: cc [ options] files.  Use 'cc -flags' for details

For this example we will compile a basic hello_world example.

Read more:

Multihomed MPI communication using OpenMPI

Today we tested something funny. We have a HS21 Blade Center with two Cisco Switches as I/O. The problem we faces was that the uplink speeds between the Blade Centers were maximum 4Gb/s because each cisco switch only had 4 external ports and there was no way to make a trunk between the two I/Os.


Read more: