|
|
# How to submit a MPICH2 job with qsub
|
|
|
How to submit a MPI job with qsub
|
|
|
=========
|
|
|
|
|
|
MPICH2 on Shark
|
|
|
----
|
|
|
'**MPICH2 on Shark**'
|
|
|
|
|
|
MPICH is a freely available, portable implementation of MPI, a standard for message-passing for distributed-memory applications used in parallel computing.
|
|
|
The MPICH2 libraries are installed on all execution nodes. In order to use the MPICH2 implementation you need a parallel environment.
|
|
|
To list all parallel environments execute this command:
|
... | ... | @@ -9,25 +12,85 @@ To list all parallel environments execute this command: |
|
|
BWA
|
|
|
make
|
|
|
mpich2
|
|
|
mpich2_mpd
|
|
|
smp
|
|
|
|
|
|
The libraries are located at:
|
|
|
|
|
|
/usr/local/mpich/mpich2
|
|
|
/usr/local/mpich/mpich-3.2
|
|
|
|
|
|
You can load either one of these MPI libraries by using the module command.
|
|
|
`module load mpich2`
|
|
|
`module load mpich/3.2`
|
|
|
|
|
|
|
|
|
|
|
|
**To use the mpich2 library**
|
|
|
|
|
|
The mpich2 library uses the parallel environment mpich2_mpd.
|
|
|
First create a **.mpd.conf** file in your home directory with the following command:
|
|
|
|
|
|
`echo "MPD_SECRETWORD=set_some_passwd_or_use_this_one" >> $HOME/.mpd.conf`
|
|
|
|
|
|
Change the security settings to read write for the user with :
|
|
|
|
|
|
`chmod 600 HOME/.mpd.conf`
|
|
|
|
|
|
To submit a test helloworld with 2 slots:
|
|
|
|
|
|
`qsub -pe mpich2_mpd 2 mpich2_mpd.sh`
|
|
|
|
|
|
The mpich2_mpd.sh submit script:
|
|
|
```
|
|
|
#!/bin/bash
|
|
|
#
|
|
|
# sample mpich2 job
|
|
|
# you will need to adjust the $PATH to your mpich2 installation
|
|
|
#with module load mpich2
|
|
|
# be sure to get the correct mpiexec for mpich2_mpd!!!
|
|
|
|
|
|
. /usr/local/Modules/current/init/bash
|
|
|
|
|
|
module load mpich2
|
|
|
|
|
|
echo "MPICH2_ROOT = $MPICH2_ROOT"
|
|
|
|
|
|
export MPD_CON_EXT="sge_$JOB_ID.$SGE_TASK_ID"
|
|
|
|
|
|
echo "Got $NSLOTS slots."
|
|
|
|
|
|
mpiexec -machinefile $TMPDIR/machines -n $NSLOTS mpihello
|
|
|
|
|
|
exit 0
|
|
|
```
|
|
|
|
|
|
The mpihello.c source code:
|
|
|
```
|
|
|
#include <stdio.h>
|
|
|
#include <mpi.h>
|
|
|
#include <unistd.h>
|
|
|
|
|
|
main(int argc, char **argv)
|
|
|
{
|
|
|
int node;
|
|
|
|
|
|
For MPICH2 we need to use the parallel environment **mpich2**.
|
|
|
On shark you will find two mpi programs inc. source in /usr/local/mpich2-bin/
|
|
|
To submit a mpich2 job with qsub and 10 slots you need a shell script:
|
|
|
The option **-rmk sge** is important to use, and the environment variable QRSH_WRAPPER.
|
|
|
int i, j;
|
|
|
float f;
|
|
|
|
|
|
#!sh
|
|
|
#!/bin/bash
|
|
|
#$ -V
|
|
|
#$ -N mpich2_tst
|
|
|
#$ -cwd
|
|
|
#$ -pe mpich2 10
|
|
|
#$ -v QRSH_WRAPPER=/usr/local/OpenGridScheduler/gridengine/bin/linux-x64/qrshwrapper
|
|
|
MPI_Init(&argc,&argv);
|
|
|
MPI_Comm_rank(MPI_COMM_WORLD, &node);
|
|
|
|
|
|
printf("Hello World from Node %d.\n", node);
|
|
|
sleep(60);
|
|
|
|
|
|
mpiexec -rmk sge -np 10 /usr/local/mpich2-bin/mpi_hello_world
|
|
|
MPI_Finalize();
|
|
|
}
|
|
|
```
|
|
|
To compile this mpihello.c first load the right module:
|
|
|
`module load mpich2`
|
|
|
Now compile the code :
|
|
|
` mpicc -o mpihello mpihello.c`
|
|
|
|
|
|
}}}
|
|
|
Change #$ -pe mpich2 **10** and the -np 10 to the number of slots you want to use.
|
|
|
Note the maximum slots that can be assinged is : 120 |
|
|
\ No newline at end of file |
|
|
Now you are ready to submit your first mpi code with 2 slots(cores/CPUs) :
|
|
|
`qsub -pe mpich2_mpd 2 mpich2_mpd.sh` |