Debugging is a crucial part of the code development and tuning cycle - either if bugs are introduced by optimization or by new code pieces.

Parallel debugging with ARM DDT

In addition to command-line debuggers like gdb the graphical debugger DDT can be used that is especially suited for scalar, multi-threaded and large-scale parallel applications written in C, C++ and Fortran.

The lecture of DDT Users Guide is recommended.

Slides of the 2017 ARM Forge tutorial at DKRZ can be found here

Getting started

  • Load the arm-forge module

    module load arm-forge
  • Recompile your application with -g and low optimization -O0, for the Intel compiler you might also have to deactivate inlining and IP optimizations, i.e. -fnoinline -no-ip

  • If you are not using shared libraries, you will have to explicitly relink against the memory debugging library in order to use the Memory Debugging feature in DDT: -L$(DDT_ROOT)/lib/64 -ldmalloc -Wl,--allow-multiple-definition

  • Start the DDT debugger and choose what kind of debugging you want to use:

    • run and debug a program (on current node/allocation or via job submission) - see below

    • wait for incoming client connections - see below

    • attach to a running program - for advanced users; if you encounter your already running job is misbehaving

    • open core file - if your previous job crashed and wrote a core (set ulimit -c unlimited before)

The two most often used approaches for interactive debugging are briefly described below.

Running a Program

Click the RUN option at the DDT welcome screen, specify the application binary to be started and the options to be used. You should not start debugging your application on one of the login nodes directly but use one of the following options instead:

  • Submit to Queue: the DDT GUI allows you to set any options for the SLURM batch script template. After submitting your debugging job (click Submit button) it will be queued as a normal job for the chosen partition. Once the job is scheduled, the DDT GUI will connect to the compute nodes and you can start debugging interactively

  • Use a SLURM allocation: if you already got an allocation via salloc and started the DDT GUI within, you have to change the MPI Implementation to ‘SLURM (generic)’ in order to use srun for launching your application and deselect the ‘Submit to Queue’ option. This way allows you to start as many debugging runs as needed without repeated job submission to the queue.

Starting DDT from a job script

If your job script is too complicated to be replaced by the template version of the DDT GUI (see above), you might also start the DDT client itself from your existing job script:

  • Start the GUI and configure DDT to use the correct MPI for your job (e.g. IntelMPI or bullxMPI)

  • Modify the job script to prefix the srun command:

    module load arm-forge
    ddt --connect srun ...
  • Start the GUI by entering ddt, afterwards submit the job. Once the job gets scheduled, it connects to the GUI and you will have to accept the incoming reverse connection request.

  • Afterwards you can change debug options like ‘Memory Debugging’ or the srun command to start the run

  • To end the debugging session use the menu bar “File” -> “End Session”; do not scancel the job while the DDT GUI is still connected since this might cause zombie processes on the compute nodes and jobs remaining in closing state

  • The SLURM job should be finalized automatically then by DDT (see srun terminate message in the GUI)

Tips & Tricks

Offline debugging

In some situations interactive work with the DDT debugger is not needed - especially for memory debugging of long running applications one might launch DDT in offline mode. You just need to modify your batch script to prefix the srun call and specify the file where HTML output of DDT should be written to. In addition you might enable memory debugging and set breakpoint where the stack traces are recorded

ddt --offline=job.html --mem-debug=thorough --break-at=<file>:<line> srun

The application will run to completion, or to the end of the job. When errors occur, for example an application crash, the stack back trace of crashing processes will be recorded to the offline output file.

Remote connection

Instead of running the DDT GUI directly on mistral, you might also use the ARM Forge GUI to connect to mistral using SSH for you such that you can run the user interface on your local machine without the need for X forwarding. Remote clients are available for Windows, Mac OS X and Linux - download (choose version 19.1). No licence file is required by a remote client. The licence of the remote system will be used once connected. Use the following remote launch settings for mistral:

  • Connection name: An optional name for this connection

  • Host name: <username>

  • Remote installation directory: needs to be set to the used version of Allinea Forge on Mistral - like /sw/rhel6-x64/analysis-tools/arm-forge-19.1.4/bin

  • Remote script: An optional script that will be run before starting the remote daemon on mistral. You may use this script to load the required modules for arm-forge, the desired MPI and compiler. This script file should be created on the remote system and the full path to the file entered in the Remote Script field box.