Debugging

DDT

Allinea Distributed Debugging Tool (DDT) is a multi-process, multi-thread graphical debugger for C, C++, Fortran, assembler and mixed source codes with GPU and accelerator support. Its feature set is similar to Totalview, although the user interface is different and more modern. 

More information

Allinea was purchased by ARM in late 2016, which has committed to continuous development and support of Allinea tools. For more information on DDT, see the Allinea DDT information.

For details on every aspect of DDT use see the DDT User Guide. 

Allinea also has a few introductory videos.

Differences from Totalview

Among the advantages to Totalview are more modern GUI with syntax highlighting with program checking, built in editor and version control.

The basic use of DDT is fairly similar to Totalview, though variable examination is different, especially with arrays. See Viewing Variables and Data from DDT user's guide for details.

DDT is installed in /uufs/chpc.utah.edu/sys/installdir/allinea-forge. DDT supports most available compilers, including GNU, PGI, and Intel.

In order to compile your code and run it in DDT, follow these steps:

Serial Code

Serial code can be debugged on personal desktops or on the interactive nodes, provided it does not generate high load that can affect responsiveness to other users.

Follow these steps:

1. Set up the environment for the appropriate compiler and DDT:
module load ddt


2. Compile with -g:
For GNU compilers: g77 -g source_name.f -o executable_name
For Intel compilers: ifort -g source_name.f -o executable_name
For PGI compilers: pgf77 -g source_name.f -o executable_name

3. Run the program in Totalview:
ddt ./executable_name

Parallel MPI/OpenMP Code

Parallel code can be debugged on personal desktops or on the interactive nodes using MPICH. On the interactive nodes, be considerate to other users and don't run too many processes at the same time. In case of need for more CPUs in the debugging session (our DDTlicense has 32 CPU limit), debug on the compute nodes.

Parallel debugging on personal desktops or on cluster interactive nodes

To debug in parallel on interactive nodes, we recommend to use MPICH MPI implementation due to ease of set up, although other MPI implementations should also work. 
Also, since interactive nodes have limited number of processors (12 to 20) and memory (24 to 64 GB), only parallel programs that need less processors and memory should be debugged on interactive nodes. Still, most bugs show up on any number of processors so we recommend debugging on the interactive nodes whenever possible since it's much easier than debugging on compute nodes. Note that you can debug more MPI processes than there are physical CPUcores, since debugging in general puts less load on the CPUs and as such more than one process can be executed at a single core at the time.

The debugging process on interactive nodes is as follows:

1. Load the DDT and MPICH module and check that MPICH is the default compiler
   

   module load gcc mpich ddt
which mpicc

   should return /uufs/chpc.utah.edu/sys/installdir/mpich/###/bin/mpicc, where ### is version number with a letter suffix. For more details on MPI implementations on HPC clusters, see CHPC's MPI Documentation.
2. Compile with -g:
   

   For GNU compilers: g77 -g source_name.f -o executable_name
   For Intel compilers: ifort -g source_name.f -o executable_name
   For PGI compilers: pgf77 -g source_name.f -o executable_name

3. Start DDT with the program name:
   

   ddt ./executable_name

4. DDT opens Run window. In this window, choose the MPI, click Details to expand the MPI menu choose number of MPI processes and pick the appropriate MPI implementation. If the code also uses OpenMP, choose the OpenMP option as well.
5. Click Run button to load the program into DDT. 
6. Set breakpoints and click the green arrow button (Play/Continue - F9) to run the program in the debugger.

Parallel debugging on compute nodes

In order to debug on compute nodes, one has to run a job through the queue. This should be done only in extreme cases when the bug does not show up during a parallel MPI run on a single node or if it takes a very long time to reach the point of the bug. If the queue wait time is too long (there are no nodes available), please, contact CHPC at helpdesk@chpc.utah.edu to set up a reservation for you so that you can get the nodes you need right away.

The process is the same as in debugging on the desktop or interactive node, except that we need to submit an interactive job through the queue, and then load the DDT module and run DDT inside of the job.

Python code

DDT supports Python debugging including MPI (e.g. mpi4py), and Python libraries written in C/C++, with a few caveats.

To debug a Python code, start DDT as follows:
ddt python3 %allinea_python_debug% my_python_code.py

Totalview

TotalView is a full-featured, source-level, multi-process, multi-thread graphical debugger f or C, C++, Fortran, assembler, and mixed source/assembler codes.

NOTE: As of 2023, we have retired the license server that served the discontinued Totalview license, and due to this Totalview is not available anymore. However, Totalview still offers free licenses for students, which we encourage those who are eligible to try. We are leaving the documentation below for legacy purposes.

NOTE: As of 2017 we have discontinued the maintenance payment for Totalview and switched to DDT. We will keep supporting the latest 2016 release of Totalview but will not update to newer versions.

More information

RogueWave, formerly TotalviewTech, manufacturer of Totalview Rogue Wave Software
Debugging with Totalview tutorial presentation (requires a video player like QuickTime, download may take some time). Video Tutorial on Totalview

Totalview is installed in /uufs/chpc.utah.edu/sys/installdir/totalview/std. This version runs on any Linux workstation with 64 bit OS. Both serial and parallel debugging (MPI, OpenMP) are supported.

Totalview supports most available compilers, including GNU, PGI, and Intel.

In order to compile your code and run it in Totalview, follow these steps:

Serial Code

Serial code can be debugged on personal desktops or on the interactive nodes, provided it does not generate high load that can affect responsiveness to other users.

Follow these steps:

1. Set up the environment for the appropriate compiler and Totalview. With modules, e.g.:
module load intel totalview
or with the phased out explicit environment sourcing:
source /uufs/chpc.utah.edu/sys/pkg/intel/ics/bin/compilervars.csh intel64
source /uufs/chpc.utah.edu/sys/installdir/totalview/std/etc/totalview.csh

2. Compile with -g:
For GNU compilers: g77 -g source_name.f -o executable_name
For Intel compilers: ifort -g source_name.f -o executable_name
For PGI compilers: pgf77 -g source_name.f -o executable_name

3. Run the program in Totalview:
totalview ./executable_name

Parallel MPI/Open MP code

Parallel code can be debugged on personal desktops or on the interactive nodes using MPICH. On the interactive nodes, be considerate to other users and don't run too many processes at the same time. In case of need for more CPUs in the debugging session (our Totalview license has 32 CPU limit), debug on the compute nodes.

Parallel debugging on personal desktops or on cluster interactive nodes

To debug in parallel on interactive nodes, we recommend to use MPICH MPI implementation due to ease of set up, although other MPI implementations should also work. 
Also, since interactive nodes have limited number of processors (12 to 20) and memory (24 to 64 GB), only parallel programs that need less processors and memory should be debugged on interactive nodes. Still, most bugs show up on any number of processors so we recommend debugging on the interactive nodes whenever possible since it's much easier than debugging on compute nodes. Note that you can debug more MPI processes than there are physical CPUcores, since debugging in general puts less load on the CPUs and as such more than one process can be executed at a single core at the time.

The debugging process on interactive nodes is as follows:

1. Check that MPICH is the default compiler
   which mpicc
   should return /uufs/chpc.utah.edu/sys/installdir/mpich/###/bin/mpicc, where ### is version number with a letter suffix. For more details on MPI implementations on HPC clusters, see CHPC's MPI Documentation.
2. Compile with -g:
     mpif77 -g source_name.f -o executable_name
3. Start Totalview:
     totalview &
   Totalview opens New Program window. In this window, fill in the following:
   * In tab Program, field Program, put in the executable program name
   * In tab Parallel, field Parallel System, choose MPICH3, and for Tasks set the number of processors you want to run on
   * Click OK to load the program into Totalview. After a little while, you will get a dialog box saying: Process executable.exe is a parallel job. Do you want to stop now? Click Yes, set some breakpoints and start debugging.

Parallel debugging on compute nodes

In order to debug on compute nodes, one has to run a job through the queue. This should be done only in extreme cases when the bug does not show up during a parallel MPI run on a single node or if it takes a very long time to reach the point of the bug. If the queue wait time is too long (there are no nodes available), please, contact CHPC at helpdesk@chpc.utah.edu to set up a reservation for you so that you can get the nodes you need right away.

1. Compile with -g:
   mpicc -g source_name.c -o executable_name
   mpif90 -g source_name.f -o executable_name
   For parallel OpenMP application using PGI compilers, include -mp to invoke OpenMP directives support, -fopenmp for GNU compilers or -openmp for Intel compilers.
2. Start interactive PBS session, e.g.:
   sbatch -N 2 -n 24 -A account -p partition -t 2:00:00 --tty /bin/tcsh -l
3. Start Totalview:
   totalview &
   Totalview opens New Program window. In this window, fill in the following:
   In tab Program, field Program, put in the executable program name
   In tab Parallel, field Parallel System, choose MPICH3, and for Tasks set the number of processors you want to run on
   Click OK to load the program into Totalview. After a little while, you will get a dialog box saying: Process executable.exe is a parallel job. Do you want to stop now? Click Yes, set some breakpoints and start debugging.

If you run Totalview for the first time, you may have to also specify a full path to the TV Server, tdvsvr. To do that, modify Launch Strings dialog in Menu - Preferences to:
  /uufs/chpc.utah.edu/sys/installdir/totalview/std/bin/tdvsvr

instead of just plain
  tdvsvr
Click go to run the code. After a little while, you will get a dialog box saying: Process executable.exe is a parallel job. Do you want to stop now? Click Yes, set some breakpoints and start debugging.

Enabling ReplayEngine

CHPC also owns 16 CPU license for the ReplayEngine plugin for Totalview, which enables one to step backwards in their debugging session. In order to use ReplayEngine, in the New Program dialog after starting Totalview, check the Enable ReplayEngine checkbox.

Enabling memory debugging

Totalview has built in memory debugging options. To enable memory debugging, in the New Program dialog after starting Totalview, check the Enable memory debugging checkbox. This option checks for array index mismatches, allocation/deallocation problems and many other common memory bugs and is useful when the program crashes for no apparent reason or when the program exits with segmentation fault error.