Skip to content

Intel Trace Analyzer and Collector (ITAC) - Correctness Checker#

Intel Trace Analyzer and Collector (ITAC) contains a correctness checker for MPI applications.

Requirements#

  • For ITAC's correctness checker to work the application must use Intel MPI. It might also work with other MPICH-based MPI libraries.
  • The ITAC module must be loaded via module load itac.
  • Optional: compile your code with debugging symbols enabled, i.e. -g (GCC, Clang, Intel oneAPI/Classic) and -ggdb3 (GCC, Clang) to see source locations.

Usage#

With Intel MPI and mpirun/mpiexec as launcher#

With Intel MPI's mpiexec and mpirun correctness checking can be activated by specifying the flag -check-mpi, like in the following example:

module load intelmpi itac
mpiexec -check-mpi ./a.out

With Intel MPI and srun as launcher#

When Intel MPI is used with srun as launcher several libraries must be preloaded to activate the correctness check:

module load intelmpi itac
LD_PRELOAD=$VT_ROOT/slib/libVTmc.so:$I_MPI_ROOT/lib/release/libmpi.so srun ./a.out

See for more details:

Use Valgrind together with correctness checking#

ITAC's correctness check can be combined with Valgrind. Furthermore Intel MPI provides a suppression file, located under $VT_ROOT/lib/impi.supp, for Valgrind to reduce the output of false positives.

A command line can then look like:

mpirun -check_mpi -l -n <num procs> valgrind --suppressions=$VT_ROOT/lib/impi.supp ./a.out

The Valgrind option --trace-children=yes is required when Valgrind only calls a wrapper script.

See the documentation for details.

Write correctness checking results to a trace file#

When correctness checking results should be stored inside a trace file, this has to be activated by setting environment variable VT_CHECK_TRACING=on, like

export VT_CHECK_TRACING=on
mpiexec -check-mpi ./a.out

See the documentation for details.

Configuration#

Correctness checking can be influenced by environment variables, some important variables include:

  • VT_DEADLOCK_TIMEOUT=<duration>(m|s): assume a deadlock occurred if no progress for <duration> occurred, default is 1m.
  • VT_CHECK_TRACING=on|off: write events and errors found to .stf trace file, default is off.
  • VT_CHECK_MAX_ERRORS=<number>: stop after detecting <number> errors, default is 1.
  • VT_PCTRACE=on|off: include source locations in .stf trace file.

Environment variables can be set in two ways:

  • export environment variables before invoking the launcher: 
    export VT_DEADLOCK_TIMEOUT=10m
export VT_CHECK_MAX_ERROS=10
mpirun -check-mpi ./a.out
  • specify environment variables to be set for the MPI application through the -genv flag: 
    mpirun -check_mpi -genv VT_DEADLOCK_TIMEOUT 10m -genv VT_CHECK_MAX_ERROS 10 ./a.out

More options can be found in the documentation.