Redtail User Guide
On this page
The table of contents requires JavaScript to load.
Access to Redtail
| Access to Redtail requires an active project on the system. Opportunities to use Redtail are posted to the Redtail overview page. |
Redtail consists of two tenants: an HPC tenant that uses SLURM for scheduling of traditional HPC jobs and a Kubernetes tenant used for persistent services using Kubernetes containers.
We are still establishing policies for the Kubernetes tenant, which will be available at later time. Information below pertains the SLURM/HPC tenant.
Allocations
Allocations apportion time on computational resources such as Redtail by merit and require an application.
On the SLURM/HPC tenant, the limit per quarter is 15,000 GPU hours and/or 100,000 CPU core hours. Despite having only GPU nodes, Redtail will accept CPU-only allocation requests; each node has 104 CPU cores and 64 cores are assigned to the GPUs, thus 40 cores remain for CPU jobs. Please, note that the limits may change based on the number of active projects, and number of GPU nodes in the SLURM tenant.
Redtail allocation requests follow the same quarterly schedule as other CHPC allocations. The submission deadline is on the first of the month before the start of the quarter. For example, for the winter quarter starting January 1st, the deadline is December 1st.
NOTE - Redtail allocations are awarded per PROJECT, not per PI. Therefore the process is different than other CHPC allocation requests. First one has to create a Redtail project, and then request the allocation for this project. It is possible to have more projects per research group, and it is also possible to re-request and allocation for an existing project. PI chooses which group members will be in each project. This allows the PI to more "fine tune" access of their group members to the group's allocations. It also gives better resource usage tracking capabilities to both CHPC and the PI.
New allocation requests can be submitted via the CHPC Portal.
For University of Utah affiliates (those with uNID), the process as follows:
- Become a CHPC PI, if not a PI already.
- Click "Request a new research group/project" on the group creation page, indicating that this resource group is for Redtail. Wait till CHPC staff approves this project, which is manual but should be very quick during the working hours.
- Once the project is approved, you will get notified by e-mail. Navigate to the new project on the CHPC Portal and click "Request additional allocation and resources," followed by "Request a compute allocation".
- Fill in the allocation request form and submit.
For applicants without University of Utah credentials, the process is different, as CHPC and CHPC Portal use uNIDs for user management.
- Log into CHPC Portal via CI Logon either choosing your institution credentials (Portal's CI Logon accepts Utah higher education institutions at this point), or with your Google credentials. Portal will create a temporary user account for you based on these credentials.
- Click "Request a new research group/project" on the group creation page, indicating that this resource group is for Redtail. Wait till CHPC staff approves this project, which is manual but should be very quick during the working hours.
- Once the project is approved, you will get notified by e-mail. Navigate to the group on the CHPC Portal and click "Request additional allocation and resources," followed by "Request a compute allocation".
- Fill in the allocation request form and submit.
- CHPC staff will review your submission and if it's appropriate, contact you to create University of Utah affiliate ID, and once that's done, create your CHPC account and move this application from the temporary account to your permanent CHPC account.
All allocation applications will be reviewed by the Redtail Allocation Committee, and granted allocation amount based on the review. Historically, review committees focus on well written resource justification. In order to get the amount requested, enumerate the reasons for what you ask, by providing test run timings and extrapolations from those, scaling analyses, etc.
We haven't established the allocation process for the Kubernetes tenant yet. Once that's done, we will post it here.
Logging in
The HPC tenant is accessible via SSH terminal, using CHPC account credentials. That
is, ssh uNID@redtail-hpc1.chpc.utah.edu. Note that there is no graphical interface on Redtail at this point; we are planning
to set up Open OnDemand in the future.
Hardware overview
| Details about Redtail's architecture and hardware specifications are available on the overview page. |
The HPC tenant has currently allocated 27 GPU nodes, each with 8x Nvidia H200 GPUs. The login nodes are small virtual machines with no GPU; therefore, any software setup or builds should be done using an interactive job. The also don't mount the local WEKA storage. This is different from CHPC clusters which have physical login nodes with large enough resources to do data pre/post processing, code builds, etc. Please, do these via interactive jobs instead.
The Kubernetes tenant currently has 6 GPU nodes.
Storage and data transfer
Redtail includes about 1 PB of fast WEKA storage, with similar performance to CHPC VAST. The WEKA is divided between the HPC and Kubernetes tenant. The HPC tenant has about 730 TB. While the space is persistent, we encourage researchers to consider it as scratch and stage their data in and out as needed, as we are planning to limit the storage quota to 30 TB per project.
Data to the WEKA can be transferred using CHPC external data transfer nodes (DTNs),
dtn0[5-8].chpc.utah.edu, which mount the space as /scratch/redtail/weka/slurm. Each challenge project has its own space at this location named after the project,
e.g. redtails20. The DTNs mount group spaces at their common location, /uufs/chpc.utah.edu/common/home, the scratches, e.g. /scratch/general/vast, and SCI file servers at /uufs/sci.utah.edu. To transfer data from CHPC file servers to Redtail WEKA, use rsync, though make sure to tar data sets that contain small files for better performance.
CHPC home directories are mounted on both the HPC tenant interactive and compute nodes,
as well as the "sys" branch (/uufs/chpc.utah.edu/sys), which contains CHPC installed applications. Note, however, that this is a remote
network mount and, as such, has higher latencies than typical network mounts on CHPC
systems.
Usage - HPC tenant
Applications
CHPC modules are available on Redtail; however, since Redtail runs a different OS
than CHPC, some modules may not work. We have run an (AI-generated) test script to
remove modules that don't load or start, but it is possible that something was missed.
That said, many modules have OS-oblivious binaries, or containers, and they will work.
For example, the deeplearning module. We recommend trying your existing modules, and if things don't work, let
our helpdesk know.
SLURM accounts and running jobs
Each project on Redtail has its own SLURM account, named after the project name. Since
there is only one partition, it does not need to be specified. Thus we only need to
include #SBATCH --account=redtails20 for project named redtails20.
Similarly, because there is only one GPU type, we can ask for the GPU with #SBATCH --gres=gpu:1. To ask for all eight GPUs on the node, it's easier to use #SBATCH --gpus-per-node=8.
Choose the CPU core count and memory size judiciously to leave room for potential
CPU-only jobs. For example, use options such as --mem, --ntasks-per-node, and --cpus-per-task that are appropriate for the workload you run.
Running MPI jobs
MPI jobs should use the Nvidia HPC-X software stack, which includes OpenMPI optimized for the InfiniBand network on Redtail.
Below is an example of a multi-node job script that uses all 8 GPUs per node on 2
nodes to run the HPL benchmark from the Nvidia hpc-benchmarks container. Note that
we use all the CPUs and memory on the nodes (--exclusive --mem=0), this may not be the best option for other applications.
#!/bin/bash
#SBATCH --exclusive
#SBATCH --mem=0
#SBATCH --nodes=2
#SBATCH --gpus-per-node=8
ml use /uufs/chpc.utah.edu/sys/installdir/u24/hpcx/2.51/modulefiles/
ml hpcx
srun -N $SLURM_NNODES --ntasks-per-node=8 --distribution=block:block:block --cpu-bind=none \
--mpi=pmi2 \
apptainer run --nv -B $(pwd) \
/uufs/chpc.utah.edu/common/home/u0101881/nvidia/hpc-benchmarks/hpc-benchmarks:25.04.sif \
/uufs/chpc.utah.edu/common/home/u0101881/nvidia/hpc-benchmarks/redtail/bindpcie.sh \
/workspace/hpl.sh --dat \
/workspace/hpl-linux-x86_64/sample-dat/HPL-dgx-${SLURM_NNODES}N.dat
Notice that we are using srun to launch the job. The OpenMPI supports the process manager that the SLURM was built
with. bindpcie.sh ensures the tasks get pinned to the correct CPU cores and GPUs.
#!/bin/bash
my_rank=${OMPI_COMM_WORLD_LOCAL_RANK:-$SLURM_LOCALID}
cpu_list=('0-47' '0-47' '0-47' '0-47' '56-111' '56-111' '56-111' '56-111')
nic_list=('mlx5_0' 'mlx5_1' 'mlx5_2' 'mlx5_3' 'mlx5_4' 'mlx5_5' 'mlx5_6' 'mlx5_7')
export UCX_NET_DEVICES="${nic_list[$my_rank]}:1"
exec numactl -C "${cpu_list[$my_rank]}" -l "$@"
Running PyTorch jobs
For jobs that use one node (up to 8 GPUs), the deeplearning module works well, as the GPUs within a node communicate over NVLink. For multi-node
jobs, however, the deeplearning module's PyTorch only communicates over Ethernet, not InfiniBand, which makes multi-node
training slow. For multi-node jobs, use the PyTorch container from Nvidia (NGC), whose
NCCL library uses the InfiniBand network.
Single-node job
Below is an example that trains a model using all 8 GPUs of a single node based on
a train.py script which needs to be writtend for distributed training. torchrun starts one process per allocated GPU.
#!/bin/bash
#SBATCH --account=redtails<xx>
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --gpus-per-node=8
#SBATCH --exclusive
#SBATCH --mem=0
#SBATCH --time=01:00:00
# laod pytorch module. deep/2025.4 module could also be used for single-node jobs
module load pytorch/2.14.0-ngc26.09
# run training job with one process per gpu node
torchrun --standalone --nproc_per_node=$SLURM_GPUS_ON_NODE train.py
Multi-node job
Below is an example that trains on 2 nodes with 8 GPUs each. We launch one task per
node with srun, and each task runs torchrun inside the container, which starts one
process per GPU. The first node serves as the rendezvous point (MASTER_ADDR).
#!/bin/bash
#SBATCH --account=redtails20
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=16
#SBATCH --exclusive
#SBATCH --mem=0
#SBATCH --time=01:00:00
# load PyTorch module
module load pytorch/2.14.0-ngc26.09
# set MASTER_ADDR variable to first node to coordinate processes. set MASTER_PORT to torch's 29500 default
export MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)
export MASTER_PORT=29500
# Use the ethernet ("bond0", IPv4) only for NCCL's initial setup communication between nodes.
export NCCL_SOCKET_IFNAME='=bond0'
export NCCL_SOCKET_FAMILY=AF_INET
# Ensure NCCL can use InfiniBand for the GPU-to-GPU data transfer between nodes.
export NCCL_IB_DISABLE=0
# Print output from NCCL concerning network used
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,NET
srun bash -c 'torchrun \
--nnodes=$SLURM_NNODES --nproc-per-node=$SLURM_GPUS_ON_NODE \
--node-rank=$SLURM_PROCID \
--master-addr=$MASTER_ADDR --master-port=$MASTER_PORT \
train.py'
Note:
The containers print the message "NVIDIA peer memory driver not detected. Multi-node
communication performance may be reduced." once per GPU. This message is a false alarm
as the driver is loaded and the multi-node communication works as expected.
Important differences relative to other CHPC clusters
SLURM/HPC tenant
- Redtail is running Ubuntu 24.04 OS, not Rocky Linux 8/9. As such, some CHPC module will work, but some will not. We have auto-checked the modules and hid those that don't work, but we may have missed some. If you notice an error while using CHPC module, please open a ticket, helpdesk@chpc.utah.edu.
- The base OS has a limited number of OS installed packages. CHPC OS images include many more. This may require to install/build more dependencies than is common at on prem CHPC systems.
- The login nodes are very small virtual machines - 8 CPUs and 24 GB RAM. They also don't mount the WEKA storage. Do all pre/post processing, installations, etc using interactive jobs.
- There are no graphics capabilities on the HPC tenant, and no graphical environment (X). Therefore, there is no X forwarding and no FastX. We plan to set up OnDemand web portal in the future.
- The home directories and sys branch which houses CHPC installed applications are mounted remotely, which means they have higher access latencies. This presents as slowdown when one logs in (Lmod modules get initialized), or when a module is loaded. If you notice very slow interaction with you home directory files or CHPC installed applications, please, let us know.
- The local WEKA storage has limited capacity. Each project has a quota. If you need more storage than the quota, treat the WEKA as persistent scratch and keep the files in CHPC storage (e.g. group, RAI, etc).
- To copy cata in/out of WEKA, use the CHPC External Data Transfer Nodes, dtn0[5-8].chpc.utah.edu.
- The SLURM partition structure is very simple, just one GPU type, one partition. Thus one does not need to specify the partition or GPU type in the job request.
Kubernetes tenant
We will update once the Kubernetes tenant is available to users.