GPU resources
-------------

.. tags:: Slurm, Condor, JupyterHub, GPU

SubMIT provides access to several GPUs. This section outlines how to utilize these GPUs in your workflow. Access to GPUs is available via Slurm on the ``submit-gpu`` partition through **slurm and Jupyterhub**. **Direct SSH access is not permitted** to ``submit-gpu``. This ensures a controlled and secure environment for utilizing GPU capabilities. Keep in mind that these are shared resources so use these machines responsibly.

Slurm with GPUs
~~~~~~~~~~~~~~~

Interactive access (salloc)
...........................

SubMIT allows interactive login access to GPUs through the ``salloc`` command. This allows users to interactively test their GPU applications. 

Accessing the ``submit-gpu`` partition:

.. code-block:: sh

      salloc --partition=submit-gpu --cpus-per-gpu=1 --gres=gpu:1

To request more than one GPU, adjust the ``--gres=gpu:<number>`` option.

.. code-block:: sh

      salloc --partition=submit-gpu --cpus-per-gpu=1 --gres=gpu:4

Batch jobs and script execution
...............................

The GPUs resources are also available through batch scripts.

Example with the ``submit-gpu`` partition, using GTX 1080 GPUs:

.. code-block:: sh

      #!/bin/bash
      #
      #SBATCH --job-name=test_gpu
      #SBATCH --output=res_%j.txt
      #SBATCH --error=err_%j.txt
      #
      #SBATCH --time=10:00
      #SBATCH --mem-per-cpu=100
      #SBATCH --partition=submit-gpu
      #SBATCH --constraint=nvidia_gtx1080
      #SBATCH --gres=gpu:2  
      #SBATCH --cpus-per-gpu=4
      
      srun hostname
      nvidia-smi

Some more examples are available in the `SubMIT examples <https://github.com/mit-submit/submit-examples/tree/main/gpu>`_ Github repository.

Selecting GPU type
..................

To specify a particular GPU type from Slurm, you can use the ``--constraint`` option, selecting one of the following:

- ``nvidia_a30``: NVIDIA A30 GPUs
- ``Tesla_v100``: Tesla V100 GPUs
- ``nvidia_gtx1080``: NVIDIA GTX1080 GPUs

For example, to select the NVIDIA A30s GPUs only,

.. code-block:: sh

    sbatch job.sh --constraint=nvidia_a30 --partition=submit

CUDA
~~~~

Compute Unified Device Architecture (CUDA) is a parallel computing platform and application programming interface (API) that allows software to use certain types of graphics processing unit (GPU) for general purpose processing. CUDA is available on the submit-gpu machines inherently, but you need to properly set the path. Open your ``.bashrc`` file in your /home directory and add:

.. code-block:: sh

      export CUDA_ROOT=/usr/local/cuda
      export LD_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/cuda/lib:/usr/local/cuda/lib:
      export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:

Once you source your bashrc file, you should be able to use CUDA.

.. code-block:: sh

      source ~/.bashrc

In order to check which CUDA version is installed you can use the command below; note that this command will not work if you are not on a GPU (e.g. access a GPU through `salloc <https://submit.mit.edu/submit-users-guide/gpu.html#interactive-access-salloc>`_). Make sure this version fits your workflow.

.. code-block:: sh

      nvcc --version


Jupyterhub
~~~~~~~~~~

You can also get to the submit-gpu machines through Jupyterhub. On the main `JupyterHub <http://submit.mit.edu/jupyter>`_ page, select submit-gpu in the dropdown menu.


Condor with GPUs
~~~~~~~~~~~~~~~~

The MIT T2 and T3 both have access to GPU machines through Condor. An example of how to access these resources through Condor is shown below.

If you wish to submit jobs to GPU machines in T3/T2, you need to add additional line in the script (only through glidein submission):

.. code-block:: sh

       RequestGPus=1

If you wish to submit jobs to GPU machines in CMS global pool, you need to add additional line in the script:

.. code-block:: sh

       RequestGPus=1
       +RequiresGPU=1

Some example scripts to run GPUs with condor can be found here:
`condor gpu <https://github.com/mit-submit/submit-examples/tree/main/gpu/condor_gpu>`_.


pytorch example
...............

Lets create a simple python code to test pytorch and name it condor_torch.py:

.. code-block:: sh

       #!/usr/bin/python

       import torch
       print(torch.cuda.device_count())

       # Your pytorch code
       # ...

In order to execute this we will make an executable that calls a cvmfs setup and then run the python file above. Name the file exec.sh

.. code-block:: sh

       echo `hostname`
       whoami
       id
       source /cvmfs/sft-nightlies.cern.ch/lcg/views/dev4cuda/latest/x86_64-centos7-gcc8-opt/setup.sh
       python condor_torch.py
       echo ">>>>>>>>>>\n"
       echo ""
       ls -a
       echo "<<<<<<<<<<\n"
       echo ""

We can then make a condor submission file to run this. As usual, name this condor.sub:

.. code-block:: sh

       universe              = vanilla
       request_disk          = 1024
       executable            = exec.sh
       arguments             = $(ProcId)
       should_transfer_files = YES
       output                = $(ClusterId).$(ProcId).out
       error                 = $(ClusterId).$(ProcId).err
       log                   = $(ClusterId).$(ProcId).log
       when_to_transfer_output = ON_EXIT
       RequestGPus=1
       +DESIRED_Sites = "mit_tier3,mit_tier2"
       queue 1

You can then submit this test with the following similar to what was shown in the Batch tutorial:

.. code-block:: sh

       condor_submit condor.sub