GPU resources¶
SubMIT provides access to several GPUs. This section outlines how to utilize these GPUs in your workflow. Access to GPUs is available via Slurm on the submit-gpu partition through slurm and Jupyterhub. Direct SSH access is not permitted to submit-gpu. This ensures a controlled and secure environment for utilizing GPU capabilities. Keep in mind that these are shared resources so use these machines responsibly.
Slurm with GPUs¶
Interactive access (salloc)¶
SubMIT allows interactive login access to GPUs through the salloc command. This allows users to interactively test their GPU applications.
Accessing the submit-gpu partition:
salloc --partition=submit-gpu --cpus-per-gpu=1 --gres=gpu:1
To request more than one GPU, adjust the --gres=gpu:<number> option.
salloc --partition=submit-gpu --cpus-per-gpu=1 --gres=gpu:4
Batch jobs and script execution¶
The GPUs resources are also available through batch scripts.
Example with the submit-gpu partition, using GTX 1080 GPUs:
#!/bin/bash
#
#SBATCH --job-name=test_gpu
#SBATCH --output=res_%j.txt
#SBATCH --error=err_%j.txt
#
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=100
#SBATCH --partition=submit-gpu
#SBATCH --constraint=nvidia_gtx1080
#SBATCH --gres=gpu:2
#SBATCH --cpus-per-gpu=4
srun hostname
nvidia-smi
Some more examples are available in the SubMIT examples Github repository.
Selecting GPU type¶
To specify a particular GPU type from Slurm, you can use the --constraint option, selecting one of the following:
nvidia_a30: NVIDIA A30 GPUsTesla_v100: Tesla V100 GPUsnvidia_gtx1080: NVIDIA GTX1080 GPUs
For example, to select the NVIDIA A30s GPUs only,
sbatch job.sh --constraint=nvidia_a30 --partition=submit
CUDA¶
Compute Unified Device Architecture (CUDA) is a parallel computing platform and application programming interface (API) that allows software to use certain types of graphics processing unit (GPU) for general purpose processing. CUDA is available on the submit-gpu machines inherently, but you need to properly set the path. Open your .bashrc file in your /home directory and add:
export CUDA_ROOT=/usr/local/cuda
export LD_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/cuda/lib:/usr/local/cuda/lib:
export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:
Once you source your bashrc file, you should be able to use CUDA.
source ~/.bashrc
In order to check which CUDA version is installed you can use the command below; note that this command will not work if you are not on a GPU (e.g. access a GPU through salloc). Make sure this version fits your workflow.
nvcc --version
Jupyterhub¶
You can also get to the submit-gpu machines through Jupyterhub. On the main JupyterHub page, select submit-gpu in the dropdown menu.
Condor with GPUs¶
The MIT T2 and T3 both have access to GPU machines through Condor. An example of how to access these resources through Condor is shown below.
If you wish to submit jobs to GPU machines in T3/T2, you need to add additional line in the script (only through glidein submission):
RequestGPus=1
If you wish to submit jobs to GPU machines in CMS global pool, you need to add additional line in the script:
RequestGPus=1
+RequiresGPU=1
Some example scripts to run GPUs with condor can be found here: condor gpu.
pytorch example¶
Lets create a simple python code to test pytorch and name it condor_torch.py:
#!/usr/bin/python
import torch
print(torch.cuda.device_count())
# Your pytorch code
# ...
In order to execute this we will make an executable that calls a cvmfs setup and then run the python file above. Name the file exec.sh
echo `hostname`
whoami
id
source /cvmfs/sft-nightlies.cern.ch/lcg/views/dev4cuda/latest/x86_64-centos7-gcc8-opt/setup.sh
python condor_torch.py
echo ">>>>>>>>>>\n"
echo ""
ls -a
echo "<<<<<<<<<<\n"
echo ""
We can then make a condor submission file to run this. As usual, name this condor.sub:
universe = vanilla
request_disk = 1024
executable = exec.sh
arguments = $(ProcId)
should_transfer_files = YES
output = $(ClusterId).$(ProcId).out
error = $(ClusterId).$(ProcId).err
log = $(ClusterId).$(ProcId).log
when_to_transfer_output = ON_EXIT
RequestGPus=1
+DESIRED_Sites = "mit_tier3,mit_tier2"
queue 1
You can then submit this test with the following similar to what was shown in the Batch tutorial:
condor_submit condor.sub
