Debugging your code

Introduction

In general you should compile your application using the compiler's -g option. This allows you to debug using symbol names (i.e., variables, functions, …) instead of just addresses. The performance impact tends to be negligible. Reducing the optimization level to -O0 makes the tracing mirror the source code but it may also make bugs disappear.

On our HPC clusters we support the following debugging tools:

• The gdb GNU Debugger can be used to debug single- and multi-threaded programs.
• Valgrind useful for detecting memory errors|leaks in programs. It is providid by the new/valgrind module.
• Intel inspector is a memory and thread debugger for parallel programs. It supports command line and graphical user interface (GUI), which requires X11-forwarding to be configured.

To use gdb to localize an error in a non-interactive job you can use its batch options; instead of

sbatch --wrap="./MY_PROGRAM"

you can use

sbatch --wrap="gdb --batch --eval-command=run --eval-command=bt --nw --args ./MY_PROGRAM"

Debugging threaded programs

To use Intel inspector you need to first have an interactive job running on a compute node. This will be used to run the program to be debugged.

• CLI: on the compute node of the interactive job

[sfux@eu-login-01 ~]$ inspxe-cl <-action> program

• GUI: on a shell on any login node with X11-forwarding:

[sfux@eu-login-01 ~]$ inspxe-gui

and the GUI of intel inspector will start.

gdb can also be used to debug threaded programs. See the short tutorial for examples.

Debugging MPI programs

Debugging MPI programs can be a complex task. The Open MPI library has a page on parallel debugging using Open MPI.

gdb, valgrind and intel inspector can also be used to debug programs parallelized with MPI. For all tools, mpirun is used to launch the program to be debugged]. This will launch one instance per process. HPC team provides a wrapper that does a stack backtrace when the program exits and prints it in a rank-specific file.

• gdb:

mpirun gdb ./my_program. 

or using the wrapper.

DBG=true mpirun /cluster/apps/local/mpi-redir.sh ./my_program

The example can then be extended for more advanced gdb usage.

• valgrind:

mpirun /cluster/apps/local/mpi-redir.sh valgrind ./my_program

• intel inspector:

mpirun inspxe-cl <action> -r RESULT_DIRECTORY ./my_program

To visualize the results of the intel inspector analysis, please use its GUI. On the GUI use menu (File->Open->Result,) or specify the result directory on the command line when calling the GUI.