Skip to content

S
Shark CentOS Slurm User guide

Home

Last edited by pybvanvliet
Page history

Shark CentOS Slurm - User guide

HPC-Linux team

Naam Locatie Email
John Berbee D-01-133 J.A.M.Berbee@lumc.nl
Tom Brusche D-01-128 T.A.W.Brusche@lumc.nl
Michel Villerius D-01-133 M.P.Villerius@lumc.nl
Pieter van Vliet D-01-133 P.Y.B.van_Vliet@lumc.nl

Cluster overview:

Hardware overview

Hostname IP address CPU Cores Memory GPUs Purpose Type machine
res-hpc-lo01 145.88.76.243 Intel E5-2660 32 128Gb 0 Login node Dell PowerEdge M620
res-hpc-exe012 145.88.76.233 Intel E5-2690 24 384Gb 0 Execution node Dell PowerEdge M630
res-hpc-exe013 145.88.76.247 Intel E5-2670 16 128Gb 0 Execution node Dell PowerEdge M620
res-hpc-exe014 145.88.76.242 Intel E5-2697 24 384Gb 0 Execution node Dell PowerEdge M620
res-hpc-exe015 145.88.76.235 Intel E5-2660 16 96Gb 0 Execution node Dell PowerEdge M620
res-hpc-exe016 145.88.76.236 Intel E5-2660 16 128Gb 0 Execution node Dell PowerEdge M620
res-hpc-exe017 145.88.76.232 Intel E5-2660 16 64Gb 0 Execution node Dell PowerEdge M620
res-hpc-exe018 145.88.76.238 Intel E5-2660 16 64Gb 0 Execution node Dell PowerEdge M620
res-hpc-exe019 145.88.76.239 Intel E5-2660 16 128Gb 0 Execution node Dell PowerEdge M620
res-hpc-gpu01 145.88.76.237 Intel 8160 48 512Gb 3 GPU node Dell PowerEdge R740
res-hpc-gpu02 145.88.76.234 Intel 8160 48 512Gb 3 GPU node Dell PowerEdge R740
res-hpc-lkeb01 145.88.70.197 Intel E5-1620 8 64Gb 1 GPU node HP Z440 Workstation
res-hpc-lkeb02 145.88.70.196 Intel E5-1650 12 16Gb 1 GPU node HP Z420 Workstation
res-hpc-lkeb03 145.88.65.244 Intel E5-2698 40 256Gb 4 GPU node NVIDIA DGX Station
res-hpc-lkeb04 145.88.76.248 Intel E5-1650 12 256Gb 4 GPU node Asus X99-E-10G WS
res-hpc-lkeb05 145.88.76.244 Intel Xe 6134 16 256Gb 3 GPU node Dell Precision 7920
res-hpc-mem01 145.88.76.230 Intel E7-4890 60 3Tb 0 High mem Dell PowerEdge R920
res-hpc-ma01 145.88.76.246 Intel E5-2697 2 4Gb 0 Controller node 1 VM: VMware
res-hpc-ma02 145.88.76.249 Intel E5-4650 1 4Gb 0 Controller node 2 VM: VMware
res-hpc-db01 145.88.76.245 Intel E5-2697 1 4Gb 0 Slurm DB node VM: VMware

GPU overview

Hostname GPU 0 (cores/mem) GPU 1 (cores/mem) GPU 2 (cores/mem) GPU 3 (cores/mem)
res-hpc-gpu01 TITAN Xp (3840/12Gb) TITAN Xp (3840/12Gb) TITAN Xp (3840/12Gb)
res-hpc-gpu02 TITAN Xp (3840/12Gb) TITAN Xp (3840/12Gb) TITAN Xp (3840/12Gb)
res-hpc-lkeb01 Tesla K40c (2880/12Gb)
res-hpc-lkeb02 TITAN X (Pascal) (3584/12Gb)
res-hpc-lkeb03 Tesla V100-DGXS (5120/16Gb) Tesla V100-DGXS (5120/16Gb) Tesla V100-DGXS (5120/16Gb) Tesla V100-DGXS (5120/16Gb)
res-hpc-lkeb04 GeForce GTX 1080 Ti (3584/12Gb) GeForce GTX 1080 Ti (3584/12Gb) GeForce GTX 1080 Ti (3584/12Gb) GeForce GTX 1080 Ti (3584/12Gb)
res-hpc-lkeb05 Quadro RTX 6000 (4608/24Gb) Quadro RTX 6000 (4608/24Gb) Quadro RTX 6000 (4608/24Gb)

Rules

  • Always login/connect to the login node res-hpc-lo01
  • Always use the workload manager (Slurm) to run/submit jobs or use it interactive
  • Never run a job outside the workload manager (Slurm), not on the login node nor on the execution nodes

How to connect to the login node / hpc cluster

From a Linux workstation

You are free to choose your Linux distribution, but we recommend the following distributions:

Other distributions:

From the command line (ssh)

If your login user name from your workstation is the same as the username on the HPC cluster, you can use:

  • ssh res-hpc-lo01

Otherwise:

  • ssh username@res-hpc-lo01

You can make your life easier by editing the file:

vi ~/.ssh/config
Host res-hpc-lo01
	Hostname 145.88.76.243
	User user-name
        ServerAliveInterval 60

Where you adapt the user-name.

X11 forwarding

You can show graphical output when you enable X11 forwarding

  • ssh -X res-hpc-lo01

or

  • ssh -Y res-hpc-lo01

Once you are logged in, you should be able to run a graphical program, for example:

  • xterm
  • xclock
  • xeyes

alt text

alt text alt text

A remote desktop

Install the X2Go client:

  • CentOS/Fedora/Red Hat: yum install x2goclient
  • Ubuntu/Debian: apt-get install x2goclient
  • Arch Linux: pacman -S x2goclient

X2Go enables to access a graphical desktop of a computer over the network. The protocol is tunneled through the Secure Shell protocol, so it is encrypted.

Start the x2goclient: alt text

Go to Session and create a new session: alt text

For the Session type:

  • XFCE
  • ICEWM

After you have created a new session alt text

Start the new session: alt text

You can ignore this error: alt text

The XFCE desktop: alt text

How to logout: alt text alt text

The ICEWM desktop: alt text

How to logout: alt text alt text

SSH proxy server

If you are working from home or from outside the LUMC (network), you can use the SSH proxy server to connect to the cluster.

  • IP address: 145.88.35.10
  • Hostname: res-ssh-alg01.researchlumc.nl

With the X2Go client, you have to enable the following options:

  • Use Proxy server for SSH connection
  • Type: SSH
  • Same login as on X2Go Server
  • Same password as on X2Go Server
  • Host: 145.88.35.10
  • Port: 22

alt text alt text

From a Windows workstation

From the command line (ssh)

A simple terminal ssh shell is putty. Download the client from the putty homepage

Direct client download putty

Windows 64-bit MSI installer

Once you have started the putty program, you will see:

alt text

Fill in at the Host Name (or IP address): res-hpc-lo01

alt text

At the connection setting, fill in 60 at the Seconds between keepalives. If needed, enable the Enable TCP keepalives

alt text

At the X11 setting, you can enable the Enable X11 forwarding. You want this if you need graphical output, but for this to get this working, you need to install separately a X11 server for Windows. It is better and easier to install mobaXterm, which will be explained below.

Now press Open to connect to the login node.

alt text

If you connect for the first time to the login node, you will see the warning. Press Yes to continue.

alt text

Login with your user name ans password.

alt text

Now you are logged in and you can start using the cluster.

alt text

If you press on the putty symbol at the left corner of the terminal window, you have multiple options, like:

  • New Session
  • Duplicate Session
  • Change Settings
  • Copy All to Clipboard
  • Clear Scrollback
  • Reset Terminal

alt text

You can give your session a name so that you can save it and reuse it later.

Give you session a name, for example "res-hpc-lo01". Save

alt text

Later you can load your saved session. Select your saved sessions and press Load

MobaXterm

With mobaXterm, you have a buildin SSH shell/terminal and an embedded X11 server. With this, you don't have to worry about setting up a special X11 server for showing graphical output on your Windows workstation.

Go to the MobaXterm website: MobaXterm

Here is the direct download link: Download

Choose the MobaXterm Home Editon v12.4 and install it on your Windows workstation. Once you have installed it, you can start it and you have to create a SSH session:

alt text

Choose for the session type: SSH and press OK

alt text

Fill in the Remote host and Specify username

alt text

All the default settings should be ok, but here you can check your Advanced SSH settings

alt text

The Terminal settings

alt text

Network settings

Pres OK to start the connection

alt text

On the left screen you can have multiple session saved

alt text

You can also add a, for example, a "SFTP" session, so that you can easily transfer files from and to your workstation.

alt text

Right clicking on one of your session, you can, for example, edit you session settings

alt text

Also here you can set the settings for the ssh jump host, if needed

A remote desktop

For the windows version, we can use the same version as we used with Linux:

Install the X2Go client

The setup is already described with the Linux client.

Module environment

When you will be working on the cluster, you probably need to load the correct module, which will set the correct environment for your library, compiler or program.

Here below are some useful commands and examples:

  • List all available modules
module avail

----------------------------------------------------------------------------------------------- /share/modulefiles ------------------------------------------------------------------------------------------------
   container/singularity/3.5.3/gcc.8.3.1        library/cuda/9.1/gcc.8.3.1                  library/wxwidgets/3.1.3/gcc-8.3.1                        statistical/MATLAB/v97/MCR2019b
   cryogenicEM/chimera/1.14/gcc-8.3.1           library/cuda/9.2/gcc.8.3.1                  neuroImaging/Elastix/5.0.0/gcc-7.4.0                     statistical/R/3.6.2/gcc.8.3.1
   cryogenicEM/ctffind/4.1.13/gcc-8.3.1         library/cudnn/10.0/cudnn                    neuroImaging/FSLeyes/0.32.0                              system/go/1.13.7
   cryogenicEM/eman2/2.31                       library/cudnn/10.1/cudnn                    neuroImaging/SimpleElastix/0.10.0/python3.6.8            system/hwloc/1.11.13/gcc-8.3.1
   cryogenicEM/gctf/1.06                        library/cudnn/10.2/cudnn                    neuroImaging/freesurfer/stable-pub-v6.0.0.patched        system/hwloc/2.1.0/gcc-8.3.1
   cryogenicEM/imod/4.9.12                      library/cudnn/9.0/cudnn                     neuroImaging/fsl/5.0.10                                  system/knem/1.1.3/gcc-8.3.1
   cryogenicEM/motioncor2/1.31                  library/cudnn/9.2/cudnn                     neuroImaging/fsl/5.0.11                                  system/python/2.7.17
   cryogenicEM/resmap/1.1.4                     library/ftgl/2.1.3/gcc-8.3.1                neuroImaging/fsl/5.0.9                                   system/python/3.7.6
   gwas/depict/1.rel194                         library/gdal/2.4.4/gcc-8.3.1                neuroImaging/fsl/6.0.0                                   system/python/3.8.1              (D)
   gwas/plink/1.90b6.16                         library/gdal/3.0.4/gcc-8.3.1                neuroImaging/fsl/6.0.1                                   tools/bcftools/1.10.2/gcc-8.3.1
   gwas/plink/1.90p                             library/htslib/1.10.2/gcc-8.3.1             neuroImaging/fsl/6.0.2                                   tools/bedtools2/2.29.1/gcc-8.3.1
   gwas/plink/2.00a3LM                   (D)    library/java/OpenJDK-11.0.2                 neuroImaging/fsl/6.0.3                                   tools/bwa/0.7.17/gcc-8.3.1
   library/boost/1.72.0/gcc-8.3.1               library/java/OpenJDK-12.0.2                 neuroImaging/fsl/fix/1.06.12                             tools/gitlab-runner/12.8.0
   library/cuda/10.0/gcc.8.3.1                  library/java/OpenJDK-13.0.2          (D)    pharmaceutical/nonmem/7.4.4/gcc-8.3.1                    tools/miniconda/python2.7/4.7.12
   library/cuda/10.1/gcc.8.3.1                  library/mpi/mpich/3.3.2/gcc-8.3.1           pharmaceutical/pirana/2.9.7                              tools/miniconda/python3.7/4.7.12
   library/cuda/10.2/gcc.8.3.1                  library/mpi/openmpi/3.1.5/gcc-8.3.1         pharmaceutical/pirana/2.9.8                       (D)    tools/samtools/1.10/gcc-8.3.1
   library/cuda/7.5/gcc.8.3.1                   library/mpi/openmpi/4.0.2/gcc-8.3.1         statistical/MATLAB/R2018b
   library/cuda/8.0/gcc.8.3.1                   library/pmi/openpmix/2.2.3/gcc-8.3.1        statistical/MATLAB/R2019b
   library/cuda/9.0/gcc.8.3.1                   library/pmi/openpmix/3.1.4/gcc-8.3.1        statistical/MATLAB/v93/MCR2017b

-------------------------------------------------------------------------------------- /usr/share/lmod/lmod/modulefiles/Core --------------------------------------------------------------------------------------
   lmod    settarg

  Where:
   D:  Default Module

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
  • load a module
module load library/mpi/openmpi/4.0.2/gcc-8.3.1
  • show loaded modules
module li

Currently Loaded Modules:
  1) library/mpi/openmpi/4.0.2/gcc-8.3.1
  • delete one module
module del library/mpi/openmpi/4.0.2/gcc-8.3.1
  • purge all modules
module purge

Compiling programs

We are going to compile a very simple MPI (message passing interface) program, which is quite common on a cluster.

  • vi hello.c
#include <stdio.h>
#include <mpi.h>

int main (int argc, char *argv[])
{
  int id, np, i;
  char processor_name[MPI_MAX_PROCESSOR_NAME];
  int processor_name_len;

  MPI_Init (&argc, &argv);

  MPI_Comm_size (MPI_COMM_WORLD, &np);
  MPI_Comm_rank (MPI_COMM_WORLD, &id);
  MPI_Get_processor_name (processor_name, &processor_name_len);

  for (i=1; i<2; i++)
    printf ("Hello world from process %03d out of %03d, processor name %s \n", id, np, processor_name);

  MPI_Finalize ();
  return 0;
}

If you are going to compile this program without the correct loaded module(s), you would see something like this:

$ module li
No modules loaded
$ gcc hello.c -o hello
hello.c:2:10: fatal error: mpi.h: No such file or directory
 #include <mpi.h>
          ^~~~~~~
compilation terminated.

So we need to load the correct module and use the correct compiler

$ module add library/mpi/openmpi/4.0.2/gcc-8.3.1
  • mpicc hello.c -o hello

Handy reference:

Language C C++ Fortran77 Fortran90 Fortran95
Command mpicc mpiCC mpif77 mpif90 mpif95
  • ./hello
Hello world from process 000 out of 001, processor name res-hpc-lo01.researchlumc.nl

Here you can see that we ran the program only on 1 core of the cpu. (which is the same as running: mpirun -np 1 ./hello) (np = number of processes to launch)

To make use of the MPI capabilities of the program, we have to run the program with the "mpirun" which comes with the loaded module library/mpi/openmpi/4.0.2/gcc-8.3.1

  • mpirun ./hello
Hello world from process 003 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 006 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 013 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 015 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 000 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 005 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 010 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 011 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 012 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 002 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 004 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 007 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 001 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 008 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 009 out of 016, processor name res-hpc-lo01.researchlumc.nl 
Hello world from process 014 out of 016, processor name res-hpc-lo01.researchlumc.nl

Now the program is using all the cores of the local machine. (which is the same as: mpirun -np 16 ./hello)

Workload manager: Slurm

The workload manager Slurm is installed on the cluster. This is a resource manager and a scheduler all in one.

  • resource manager: are there resources free (memory, cpu, gpus, etc) on the nodes for the job?
  • scheduler: when to run the job?

Partitions (info) (in SGE called queues)

Available partitions

Partition name Nodes Default MemPerCpu DefaultTime
all res-hpc-exe[013-019] Yes 2048 1:00:00
gpu res-hpc-gpu[01-02] No 2048 1:00:00
highmem res-hpc-mem01 No 2048 1:00:00
LKEBgpu res-hpc-lkeb[01-05] No 2048 -

A job must always be submitted to a partition (queue). By default, the jobs will be submitted to the "all" partition unless you specify a different queue. The following commands are useful:

  • sinfo
  • sinfo -a
  • sinfo -l
  • sinfo -N -l
[user@res-hpc-lo01 ~]$ sinfo 
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST 
all*         up   infinite      2    mix res-hpc-exe[014,019] 
all*         up   infinite      1  alloc res-hpc-exe018 
all*         up   infinite      4   idle res-hpc-exe[013,015-017] 
gpu          up   infinite      2    mix res-hpc-gpu[01-02] 
highmem      up   infinite      1    mix res-hpc-mem01 

[user@res-hpc-lo01 ~]$ sinfo -a
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST 
all*         up   infinite      2    mix res-hpc-exe[014,019] 
all*         up   infinite      1  alloc res-hpc-exe018 
all*         up   infinite      4   idle res-hpc-exe[013,015-017] 
gpu          up   infinite      2    mix res-hpc-gpu[01-02] 
highmem      up   infinite      1    mix res-hpc-mem01 
LKEBgpu      up   infinite      1  comp* res-hpc-lkeb02 
LKEBgpu      up   infinite      1  down* res-hpc-lkeb03 
LKEBgpu      up   infinite      2    mix res-hpc-lkeb[04-05] 
LKEBgpu      up   infinite      1   idle res-hpc-lkeb01 

[user@res-hpc-lo01 ~]$ sinfo -l
Mon Mar 23 09:21:27 2020
PARTITION AVAIL  TIMELIMIT   JOB_SIZE ROOT OVERSUBS     GROUPS  NODES       STATE NODELIST 
all*         up   infinite 1-infinite   no       NO        all      2       mixed res-hpc-exe[014,019] 
all*         up   infinite 1-infinite   no       NO        all      1   allocated res-hpc-exe018 
all*         up   infinite 1-infinite   no       NO        all      4        idle res-hpc-exe[013,015-017] 
gpu          up   infinite 1-infinite   no       NO        all      2       mixed res-hpc-gpu[01-02] 
highmem      up   infinite 1-infinite   no       NO        all      1       mixed res-hpc-mem01 
[user@res-hpc-lo01 ~]$ sinfo -l -N -a
Mon Mar 23 09:34:14 2020
NODELIST        NODES PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON               
res-hpc-exe013      1      all*       mixed 16      2:8:1 128800        0      1   (null) none                 
res-hpc-exe014      1      all*       mixed 24     2:12:1 386800        0      1   (null) none                 
res-hpc-exe015      1      all*        idle 16      2:8:1  96000        0      1   (null) none                 
res-hpc-exe016      1      all*        idle 16      2:8:1 128800        0      1   (null) none                 
res-hpc-exe017      1      all*        idle 16      2:8:1  64000        0      1   (null) none                 
res-hpc-exe018      1      all*   allocated 16      2:8:1  64000        0      1   (null) none                 
res-hpc-exe019      1      all*       mixed 16      2:8:1 128800        0      1   (null) none                 
res-hpc-gpu01       1       gpu       mixed 48     2:24:1 515000        0      1   (null) none                 
res-hpc-gpu02       1       gpu       mixed 48     2:24:1 515000        0      1   (null) none                 
res-hpc-lkeb01      1   LKEBgpu        idle 8       1:4:2  63000        0      1   (null) none                 
res-hpc-lkeb02      1   LKEBgpu completing* 12      1:6:2  15000        0      1   (null) none                 
res-hpc-lkeb03      1   LKEBgpu       down* 40     1:20:2 250000        0      1   (null) Not responding       
res-hpc-lkeb04      1   LKEBgpu       mixed 12      1:6:2 257000        0      1   (null) none                 
res-hpc-lkeb05      1   LKEBgpu       mixed 16      2:8:1 256000        0      1   (null) none                 
res-hpc-mem01       1   highmem       mixed 60     4:15:1 300000        0      1   (null) none
  • idle: this node has no jobs running on it
  • alloc(ated): the whole node is allocated by 1 or more jobs
  • mix(ed): there is 1 or more jobs running on the node, but there are still cores free on this node

Jobs info

With the following command, you can get information about your running jobs and jobs from other users:

  • squeue
  • squeue -a
  • squeue -l
[user@res-hpc-lo01 mpi-benchmarks]$ squeue 
             JOBID  PARTITION      USER  ST        TIME   NODES NODELIST(REASON) 
               258        all      user   R        0:03       2 res-hpc-exe[013-014] 

[user@res-hpc-lo01 mpi-benchmarks]$ squeue -a
             JOBID  PARTITION      USER  ST        TIME   NODES NODELIST(REASON) 
               258        all      user   R        0:06       2 res-hpc-exe[013-014] 

[user@res-hpc-lo01 mpi-benchmarks]$ squeue -l
Thu Jan 23 09:14:22 2020
             JOBID  PARTITION      USER     STATE        TIME TIME_LIMIT   NODES NODELIST(REASON) 
               258        all      user   RUNNING        0:12      30:00       2 res-hpc-exe[013-014]

Jobs typically pass through several states in the course of their execution.
The typical states are PENDING, RUNNING, SUSPENDED, COMPLETING, and COMPLETED. An explanation of some state follows:

State State (full) Explanation
CA CANCELLED Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
CD COMPLETED Job has terminated all processes on all nodes with an exit code of zero.
CG COMPLETING Job is in the process of completing. Some processes on some nodes may still be active.
F FAILED Job terminated with non-zero exit code or other failure condition.
PD PENDING Job is awaiting resource allocation.
R RUNNING Job currently has an allocation.
S SUSPENDED Job has an allocation, but execution has been suspended and CPUs have been released for other jobs.

scontrol

With the slurm command scontrol you can get a more detailed overview of your running job, node hardware and partitions:

[user@res-hpc-lo01 ~]$ scontrol show job 260
JobId=260 JobName=IMB
   UserId=user(225812) GroupId=Domain Users(513) MCS_label=N/A
   Priority=35603 Nice=0 Account=dnst-ict QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:13 TimeLimit=00:30:00 TimeMin=N/A
   SubmitTime=2020-01-23T10:27:45 EligibleTime=2020-01-23T10:27:45
   AccrueTime=2020-01-23T10:27:45
   StartTime=2020-01-23T10:27:45 EndTime=2020-01-23T10:57:45 Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2020-01-23T10:27:45
   Partition=all AllocNode:Sid=res-hpc-ma01:46428
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=res-hpc-exe[013-014]
   BatchHost=res-hpc-exe013
   NumNodes=2 NumCPUs=32 NumTasks=32 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=32,mem=64G,node=2,billing=32
   Socks/Node=* NtasksPerN:B:S:C=16:0:*:* CoreSpec=*
   MinCPUsNode=16 MinMemoryCPU=2G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=/home/user/Software/imb/mpi-benchmarks/imb.slurm
   WorkDir=/home/user/Software/imb/mpi-benchmarks
   StdErr=/home/user/Software/imb/mpi-benchmarks/job.%J.err
   StdIn=/dev/null
   StdOut=/home/user/Software/imb/mpi-benchmarks/job.%J.out
   Power=
   MailUser=user@gmail.com MailType=BEGIN,END,FAIL

[user@res-hpc-lo01 ~]$ scontrol show node res-hpc-exe013
NodeName=res-hpc-exe013 Arch=x86_64 CoresPerSocket=8 
   CPUAlloc=16 CPUTot=16 CPULoad=0.00
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=res-hpc-exe013 NodeHostName=res-hpc-exe013 Version=20.02.0-0pre1
   OS=Linux 4.18.0-80.11.2.el8_0.x86_64 #1 SMP Tue Sep 24 11:32:19 UTC 2019 
   RealMemory=128800 AllocMem=32768 FreeMem=121945 Sockets=2 Boards=1
   State=ALLOCATED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=all 
   BootTime=2019-12-11T11:50:20 SlurmdStartTime=2020-01-14T15:35:41
   CfgTRES=cpu=16,mem=128800M,billing=16
   AllocTRES=cpu=16,mem=32G
   CapWatts=n/a
   CurrentWatts=0 AveWatts=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

[user@res-hpc-lo01 ~]$ scontrol show node res-hpc-exe014
NodeName=res-hpc-exe014 Arch=x86_64 CoresPerSocket=12 
   CPUAlloc=16 CPUTot=24 CPULoad=0.00
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=res-hpc-exe014 NodeHostName=res-hpc-exe014 Version=20.02.0-0pre1
   OS=Linux 4.18.0-80.11.2.el8_0.x86_64 #1 SMP Tue Sep 24 11:32:19 UTC 2019 
   RealMemory=386800 AllocMem=32768 FreeMem=380208 Sockets=2 Boards=1
   State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=all 
   BootTime=2019-12-11T11:51:40 SlurmdStartTime=2020-01-14T15:36:20
   CfgTRES=cpu=24,mem=386800M,billing=24
   AllocTRES=cpu=16,mem=32G
   CapWatts=n/a
   CurrentWatts=0 AveWatts=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

[user@res-hpc-lo01 ~]$ scontrol show partition all
PartitionName=all
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=YES QoS=N/A
   DefaultTime=01:00:00 DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED
   Nodes=res-hpc-exe[013-014]
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=OFF
   State=UP TotalCPUs=40 TotalNodes=2 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerCPU=2048 MaxMemPerNode=UNLIMITED

Interactive jobs

  • salloc

You can open an interactive session with the salloc command:

[user@res-hpc-lo01 ~]$ salloc -N1
salloc: Granted job allocation 267
salloc: Waiting for resource configuration
salloc: Nodes res-hpc-exe013 are ready for job

[user@res-hpc-exe013 ~]$ squeue 
             JOBID  PARTITION      USER  ST        TIME   NODES NODELIST(REASON) 
               267        all      user   R        0:04       1 res-hpc-exe013 
[user@res-hpc-exe013 ~]$ exit
exit
salloc: Relinquishing job allocation 267

[user@res-hpc-lo01 ~]$

In the example above, we won't run a command so we ended up in the bash environment. With exit we leave the environment and we release the node.

[user@res-hpc-lo01 ~]$ salloc -N1 mpirun ./hello1
salloc: Granted job allocation 268
salloc: Waiting for resource configuration
salloc: Nodes res-hpc-exe013 are ready for job
Hello world from process 000 out of 001, processor name res-hpc-exe013.researchlumc.nl 
salloc: Relinquishing job allocation 268
salloc: Job allocation 268 has been revoked.

Here we allocated 1 node with one core and ran the openmpi compiled "hello1" program.

Now the same with 2 nodes, 16 cores on each machine:

[user@res-hpc-lo01 ~]$ salloc -N2 --ntasks-per-node=16 mpirun ./hello1
salloc: Granted job allocation 270
salloc: Waiting for resource configuration
salloc: Nodes res-hpc-exe[013-014] are ready for job
Hello world from process 003 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 021 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 004 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 005 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 027 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 000 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 029 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 006 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 031 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 007 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 016 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 010 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 019 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 011 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 030 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 012 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 017 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 013 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 018 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 014 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 020 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 015 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 022 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 001 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 023 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 024 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 002 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 025 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 008 out of 032, processor name res-hpc-exe013.researchlumc.nl 
Hello world from process 026 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 028 out of 032, processor name res-hpc-exe014.researchlumc.nl 
Hello world from process 009 out of 032, processor name res-hpc-exe013.researchlumc.nl 
salloc: Relinquishing job allocation 270
  • srun

With the srun command you can also open an interactive session or you can run a program through the scheduler.

Interactive:

[user@res-hpc-lo01 ~]$ srun --pty bash
[user@res-hpc-exe013 ~]$ exit
exit

Running a program:

[user@res-hpc-lo01 ~]$ cat hello.sh
#!/bin/bash
#

echo "Hello from $(hostname)"
echo "It is currently $(date)"
echo ""
echo "SLURM_JOB_NAME: $SLURM_JOB_NAME"
echo "SLURM_JOBID: " $SLURM_JOBID

[user@res-hpc-lo01 ~]$ chmod +x hello.sh 

[user@res-hpc-lo01 ~]$ srun -N1 hello.sh
Hello from res-hpc-exe013.researchlumc.nl
It is currently Thu Jan 23 12:35:18 CET 2020

SLURM_JOB_NAME: hello.sh
SLURM_JOBID:  282

sbatch

The normal and correct way to submit a job is with a slurm batch file. This is a normal bash script with special directives for slurm.

In SBATCH lines, “#SBATCH” is used to submit options. The various meanings of lines starting with “#” are:

Line starts with Treated as
# Comment in shell and Slurm
#SBATCH Comment in shell, option in Slurm
# SBATCH Comment in shell and Slurm

Options, sometimes called “directives”, can be set in the job script file using this line format for each option:

#SBATCH {option} {parameter}
Directive Description Specified As
Name the job < jobname > #SBATCH -J < jobname >
Request at least < minnodes > nodes #SBATCH -N < minnodes >
Request < minnodes > to < maxnodes > nodes #SBATCH -N < minnodes >-< maxnodes >
Request at least < MB > amount of temporary disk space #SBATCH --tmp < MB >
Run the job for a time of < walltime > #SBATCH -t < walltime >
Run the job at < time > #SBATCH --begin < time >
Set the working directory to < directorypath > #SBATCH -D < directorypath >
Set error log name to < jobname.err > #SBATCH -e < jobname.err >
Set output log name to < jobname.log > #SBATCH -o < jobname.log >
Mail < user@address > #SBATCH --mail-user < user@address >
Mail on any event #SBATCH --mail-type=ALL
Mail on job end #SBATCH --mail-type=END
Run job in partition #SBATCH -p < destination >
Run job using GPU with ID < number > #SBATCH --gres=gpu:< number >

Node-Core reservation:

Short option Long option Description
-N --nodes= Request this many nodes on the cluster. Use 1 core on each node by default
-n --ntasks= Request this many tasks on the cluster. Defaults to 1 task per node
(none) --ntasks-per-node= Request this number of tasks per node

For example:

Options Description
-N2 use 2 nodes, 1 core on each node, so in total 2 cores
-N2 --ntasks-per-node=16 use 2 nodes, 16 cores on each node, so in total 32 cores
-n32 use 32 cores in total, let Slurm decide where to run (one or multiple nodes)

Submitting jobs

module purge
module add library/mpi/openmpi/4.0.2/gcc-8.3.1

git clone https://github.com/intel/mpi-benchmarks
cd mpi-benchmarks
make clean
cd src_c
make clean
make -f Makefile TARGET=MPI1

Please notice in the Makefile: CC=mpicc

ldd ./IMB-MPI1 
	linux-vdso.so.1 (0x00007fff6e9f6000)
	libmpi.so.40 => /share/software/library/mpi/openmpi/4.0.2/gcc-8.3.1/lib/libmpi.so.40 (0x00007f7c6acb7000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7c6aa97000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f7c6a6d4000)
	libopen-rte.so.40 => /share/software/library/mpi/openmpi/4.0.2/gcc-8.3.1/lib/libopen-rte.so.40 (0x00007f7c6a41e000)
	libopen-pal.so.40 => /share/software/library/mpi/openmpi/4.0.2/gcc-8.3.1/lib/libopen-pal.so.40 (0x00007f7c6a12e000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f7c69f2a000)
	libudev.so.1 => /lib64/libudev.so.1 (0x00007f7c69d03000)
	libpciaccess.so.0 => /lib64/libpciaccess.so.0 (0x00007f7c69af9000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f7c698f0000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f7c6956e000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007f7c6936a000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f7c69153000)
	libevent-2.1.so.6 => /lib64/libevent-2.1.so.6 (0x00007f7c68efa000)
	libevent_pthreads-2.1.so.6 => /lib64/libevent_pthreads-2.1.so.6 (0x00007f7c68cf7000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f7c6afda000)
	libmount.so.1 => /lib64/libmount.so.1 (0x00007f7c68a9d000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f7c68885000)
	libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f7c683a7000)
	libblkid.so.1 => /lib64/libblkid.so.1 (0x00007f7c68155000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f7c67f4d000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f7c67d22000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f7c67a9e000)
#!/bin/bash
#SBATCH -J IMB
#SBATCH -N 2
# SBATCH --ntasks-per-node=16
# SBATCH --ntasks-per-node=6
# SBATCH -n 32
# SBATCH --exclusive
#SBATCH --time=00:30:00 
#SBATCH --error=job.%J.err
#SBATCH --output=job.%J.out
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=user@lumc.nl

# Clear the environment from any previously loaded modules
module purge > /dev/null 2>&1

# Load the module environment suitable for the job
# module load library/mpi/openmpi/3.1.5/gcc-8.3.1
module load library/mpi/openmpi/4.0.2/gcc-8.3.1

echo "Starting at `date`"

echo "Running on hosts: $SLURM_JOB_NODELIST"
echo "Running on $SLURM_JOB_NUM_NODES nodes."
echo "Running $SLURM_NTASKS tasks."
echo "Account: $SLURM_JOB_ACCOUNT"
echo "Job ID: $SLURM_JOB_ID"
echo "Job name: $SLURM_JOB_NAME"
echo "Node running script: $SLURMD_NODENAME"
echo "Submit host: $SLURM_SUBMIT_HOST"

echo "Current working directory is `pwd`"

mpirun ./IMB-MPI1 
echo "Program finished with exit code $? at: `date`"

scancel

With the scancel command you can cancel your running job or your scheduled job:

  • scancel jobid

where the jobid is your job identifier.

scontrol update

While the scontrol show a powerful command is to show info about your job, with the scontrol update command, you can change certain settings as long as your job is on hold or pending. First put your job on hold, update the settings and then release your job:

  • scontrol hold jobid
  • scontrol update job jobid NumNodes=2-2 NumTasks=2 Features=intel16
  • scontrol release jobid

See the man page for the scontrol command.

X11 forwarding

You can enable X11 forwarding with the "--x11" parameter, for example:

  • srun -n1 --pty --x11 xclock

Using GPU's

You can use a GPU with the --gres parameter, for example:

--partition=gpu
--gres=gpu:1
--ntasks=1
--cpus-per-task=1

Syntax:

  • --gres=gpu:[type gpu]:[number of gpus]

Normally you don't have to specify the type of GPU. But if there are different kind of GPUs in a single machine or you want to run on a certain type of GPU you have to specify on which GPU you want to run, for example:

--partition=LKEBgpu
--gres=gpu:1080Ti:1
Hostname Partition Type GPU and number
res-hpc-lkeb01 LKEBgpu Gres=gpu:K40C:1
res-hpc-lkeb02 LKEBgpu Gres=gpu:TitanX:1
res-hpc-lkeb03 LKEBgpu Gres=gpu:V100:4
res-hpc-lkeb04 LKEBgpu Gres=gpu:1080Ti:4
res-hpc-lkeb05 LKEBgpu Gres=gpu:RTX6000:3
res-hpc-gpu[01-02] gpu Gres=gpu:TitanXp:3

Slurm Environment Variables

Available environment variables include:

Variable Meaning
SLURM_CPUS_ON_NODE processors available to the job on this node
SLURM_JOB_ID job ID of executing job
SLURM_LAUNCH_NODE_IPADDR IP address of node where job launched
SLURM_NNODES total number of nodes
SLURM_NODEID relative node ID of current node
SLURM_NODELIST list of nodes allocated to job
SLURM_NTASKS total number of processes in current job
SLURM_PROCID MPI rank (or relative process ID) of the current process
SLURM_SUBMIT_DIR directory from with job was launched
SLURM_TASK_PID process ID of task started
SLURM_TASKS_PER_NODE number of task to be run on each node
CUDA_VISIBLE_DEVICES which GPUs are available for use

More Slurm info

Sview

sview is a graphical frontend for slurm which can be handy sometimes. It only gives you some minimal functionality.

Don't forget to enable X11 forwarding.

alt text alt text alt text alt text