Compiling and Linking#

This page describes how to build software on Levante, i.e. to generate executable files from source code files typically written in C, C++ or Fortran and parallelised using MPI and/or OpenMP.

Compilers#

Different release versions of the following compiler suites are available on Levante:

This variety of compilers supports different use cases, such as generating highly optimised binaries for CPUs or GPUs or debugging complex Fortran codes.

Important

Compilers are not available by default in your shell environment; you must load the appropriate module file to access them. We recommend specifying the version number explicitly. Otherwise, either the default version (if one is defined for that module) or the lexicographically highest available version will be loaded.

Intel Compilers#

For most applications we recommend to use the Intel compilers, which are part of the Intel oneAPI Toolkit. The compiler version can be selected by loading the corresponding module file, for example:

# Use a specific version of Intel compiler
$ module load intel-oneapi-compilers/2022.0.1-gcc-11.2.0
$ module load intel-oneapi-compilers/2025.3.2-gcc-13.4.0

The specific compiler names are summarized in the table below. The Intel classic compilers are no longer available in newer versions of the Intel OneAPI Toolkits distributions.

Classic

OneAPI

Purpose
icc
icx

for C source code
ifort
ifx

for Fortran source code
icpc
icpx

for C++ source code

Compiler options that are commonly used for the Intel compilers are:

Option

Descripion
-qopenmp

Generates multi-threaded code based on the OpenMP directives
-g

Creates debugging information in the object files. This is necessary if you want to debug your program
-O[0-3]

Sets the optimization level
-I/path/to/include

Adds directories to search for include files
-L/path/to/lib

A path can be given in which the linker searches for libraries
-Wl,-rpath,/path/to/lib

Pass -rpath option to the linker, which adds the specified directory to the runtime library search path
-D

Defines a CPP macro
-U

Undefines a CPP macro
-ipo

Inter-procedural optimization
-march=core-avx2

Indicates the processor for which code is generated
-mtune=core-avx2

Indicates the processor for which optimizations are performed
-sox

Stores information like compiler version, options used etc. in the executable file
-help

Displays available compiler options

For further information on optimisation flags and other Intel oneAPI compiler options, please refer to the manual pages of the respective compiler, e.g.:

$ man ifx
$ man icx
$ man icpx

or the comprehensive documentation on Intel website.

Note

Using the compiler option -march=core-avx2 forces the Intel compiler to use full AVX2 support/vectorization (with FMA instructions) which might results in binaries that do not produce MPI decomposition independent results. Adding the option -no-fma should solve this issue but could result in slightly longer runtime.

GNU Compiler Collection (GCC)#

GCC is a suite of compilers for C (gcc), C++ (g++), Fortran (gfortran), D (gcd) and some other programming languages. You need to load an environment module for gcc to access a more recent version of the GNU compiler suite. The use of the older system gcc located in /usr/bin and provided as part of the base Linux operating system is generally inadvisable. Example for loading a specific version of gcc:

# Use a specific version of gcc compiler
$ module load gcc/11.2.0-gcc-11.2.0

nAG#

nAG Fortran Compiler (nagfor) has proved to be very useful for debugging und checking if the source code is standard conforming. It is not appropriate to create model binaries for production runs. To make nAG compilers available in your shell environment, you need to load an nag environment module, e.g.:

# Use a specific version of nag compiler
$ module load nag/7.2-gcc-11.2.0

NVIDIA HPC SDK#

The NVIDIA HPC Software development Kit (SDK) contains a compiler suite that is of special interest for users and developers of GPU-ready codes. It supports GPU offloading using OpenMP, OpenACC and CUDA. Please refer to the section on GPU Programming for more information. Since 2020 NVIDIA HPC compilers replace the PGI compilers.

Compiling and Linking MPI programs#

MPI Libraries#

Currently, two implementations of Message Passing Interface (MPI) library are available on Levante:

No MPI libraries are available by default. Similar to compilers, you have to explicitly load an appropriate environment module for a certain MPI implementation.

Important

Since Fortran module files (.mod) are compiler-specific, you must use an MPI installation built with the same compiler that you use to build your code. The compiler used for each MPI installation is indicated by suffixes in the MPI modulefile names, such as intel-2021.5.0, oneapi-2025.3.2, gcc-11.2.0 or nvhpc-24.7 etc.

MPI Compiler Wrappers#

It is highly advisabe to use MPI compiler wrappers to compile and link MPI parallel programs. Such wrappers are provided with each MPI library implementation. They automatically build up the MPI environment (i.e. set paths to MPI include files and MPI libraries) to facilitate the compilation and linking steps. The following table shows the names of the Open MPI and Intel MPI compiler wrappers:

Language

Open MPI wrapper

Intel MPI wrapper

Fortran

mpifort

mpiifort

C++

mpic++, mpicxx, mpiCC

mpiicpc

C

mpicc

mpiicc

Examples#

  • Compile a MPI parallel program in Fortran using Intel oneAPI Fortran compiler and Open MPI:

$ module load intel-oneapi-compilers/2025.3.2-gcc-13.4.0
$ module load openmpi/5.0.10-oneapi-2025.3.2
$ mpifort -O2 -march=core-avx2 -fp-model source -o myprog myprog.f90

  • Compile a hybrid parallel (MPI + OpenMP) program in Fortran using Intel oneAPI Fortran compiler and Open MPI:

$ module load intel-oneapi-compilers/2025.3.2-gcc-13.4.0
$ module load openmpi/5.0.10-oneapi-2025.3.2
$ mpifort -qopenmp -O2 -march=core-avx2 -fp-model source -o myprog myprog.f90

  • Compile a hybrid MPI/OpenMP program in Fortran using GCC Fortran compiler and Open MPI:

$ module load gcc/11.2.0-gcc-11.2.0
$ module load openmpi/4.1.2-gcc-11.2.0
$ mpifort -fopenmp -O2 -march=native -mpc64 -o myprog myprog.f90

  • Compile a MPI program in Fortran using Intel Fortran compiler and Intel MPI:

$ module load intel-oneapi-compilers/2022.0.1-gcc-11.2.0
$ module load intel-oneapi-mpi/2021.5.0-intel-2021.5.0
$ mpiifort -O2 -march=core-avx2 -fp-model source -o myprog myprog.f90

Note

The computational performance and scalability of MPI applications on Levante can be considerably improved by an optimal choice of the runtime parameters provided by MPI libraries. The appropriate MPI run time settings strongly depend on the type of application and MPI library used. For most MPI versions installed on Levante, we provide some recommendations for MPI environment settings that proved to be beneficial for different model codes commonly used at DKRZ.

Libraries from the software tree#

Many commonly used libraries are available from the central software tree located at /sw/spack-levante. They are all installed in individual directories so we can provide different versions and configurations of the same library. This means that you should carefully chose which library to link and run your model with.

For many libraries, you can find a module file which tells you where to find the library for compiling and linking. More detailed information can be inquired with the spack command for all installed libraries.

How to build software with netCDF#

NetCDF libraries are commonly used in climate models for data input and output. If only the Fortran interface is directly accessed in your program, it is sufficient to know the installation path of the netcdf-fortran library and to use this information for building your software. Netcdf-fortran installations available on Levante can be inquired with the module avail command:

$ module avail netcdf-fortran

Based on the compiler and MPI you want to use, you need to select one of the available libraries that is compatible and meets your requirements. The installation path of that library can then be inferred with the module show command. For example, if you use Intel oneAPI compiler and Open MPI, and you find that netcdf-fortran/4.6.2-openmpi-5.0.10-oneapi-2025.3.2 matches your setup, the following module show command:

$ module show netcdf-fortran/4.6.2-openmpi-5.0.10-oneapi-2025.3.2

-------------------------------------------------------------------
/sw/spack-levante/spack/modules/netcdf-fortran/4.6.2-openmpi-5.0.10-oneapi-2025.3.2:

module-whatis   {NetCDF (network Common Data Form) is a set of software libraries...}
conflict           netcdf-fortran
prepend-path    --delim : PATH /sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc/bin
prepend-path    --delim : MANPATH /sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc/share/man
prepend-path    --delim : PKG_CONFIG_PATH /sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc/lib/pkgconfig
prepend-path    --delim : CMAKE_PREFIX_PATH /sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc/.
append-path     --delim : MANPATH {}
-------------------------------------------------------------------

provides the installation path of the library:

/sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc

The directory containing include files is then:

/sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc/include

and the path to the library files is:

/sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc/lib

In a usual setting, you can instruct your compiler and linker to use the netCDF library from this path both for compiling and when running your program using the following options:

$ mpifort -I/sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc/include \
          -L/sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc/lib -lnetcdff \
          -Wl,-rpath,/sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc/lib \
          -o myprog myprog.f90

The -I option tells the Intel compiler where to find include files and Fortran module description files (.mod) for netcdf-fortran. The -L option tells the link editor where to find the netcdf-fortran library (libnetcdff) itself.

Since the dynamic version of the library is linked by default, you must also encode its runtime search path in the generated binary using the -Wl,-rpath, option. This is necessary because netcdf-fortran is not installed in a standard directory, such as /usr/lib64, that is automatically searched by the dynamic linker. Otherwise, you will encounter the following error at runtime:

myprog: error while loading shared libraries: libnetcdff.so.7:
cannot open shared object file: No such file or directory

In general, you will have to add options corresponding to the above for each library you are going to use.

On Levante, libraries in the software tree already include link information indicating where their respective dependencies can be found. For example, netcdf-fortran contains the correct paths to netcdf-c, hdf5, libaec, and other dependencies. You do not need to provide this information again unless your program also calls functions from those libraries directly. In that case, we recommend using the spack command as follows to obtain information about the dependency libraries:

$ spack find -dp netcdf-fortran %oneapi@2025.3.2  ^openmpi@5.0.10

The above command will list the installation of the netcdf-fortran library built with the specified version of the Intel oneAPI compiler and depending on the chosen Open MPI installation and subsequently all of its dependencies plus the corresponding installation paths:

-- linux-rhel8-zen3 / oneapi@2025.3.2 ---------------------------
netcdf-fortran@4.6.2               /sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc
  netcdf-c@4.10.0                  /sw/spack-levante/netcdf-c-4.10.0-wplcog
     ...
     hdf5@1.14.6                   /sw/spack-levante/hdf5-1.14.6-cz2y6n
         cmake@3.31.11             /sw/spack-levante/cmake-3.31.11-3zz4ku
             ncurses@6.4           /sw/spack-levante/ncurses-6.4-vpcjt5
         gmake@4.4.1               /sw/spack-levante/gmake-4.4.1-zwphlx
     libaec@1.1.3                  /sw/spack-levante/libaec-1.1.3-qhqvrk
     libxml2@2.9.7                 /usr
     parallel-netcdf@1.14.1        /sw/spack-levante/parallel-netcdf-1.14.1-wshell
         m4@1.4.18                 /usr
     zlib@1.2.11                   /usr
     zstd@1.5.7                    /sw/spack-levante/zstd-1.5.7-3trffm
  openmpi@5.0.10                   /sw/spack-levante/openmpi-5.0.10-pjjvwl
     hcoll@4.7.3199                /opt/mellanox/hcoll
     ...

==> 1 installed package

You can alternatively use the spack hash (which is also part of the package installation path) to infer the above information about dependencies and paths:

$ spack find -dp /yuo6bc

These paths can then be used to fill in additional information needed by build systems for packages that also directly interact with one or more of these other libraries (or just require that information because it is needed on other systems where libraries are not installed completely linked), for example:

$ mpifort -I/sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc/include \
          -L /sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc/lib \
          -L/sw/spack-levante/netcdf-c-4.10.0-wplcog/lib -lnetcdff -lnetcdf \
          -Wl,-rpath,/sw/spack-levante/netcdf-fortran-4.6.2-yuo6bc/lib \
          -Wl,-rpath,/sw/spack-levante/netcdf-c-4.10.0-wplcoglib \
          -o myprog myprog.f90 util.o

Many libraries provide utilities to facilitate compiling and linking, e.g. netcdf-c contains the nc-config command to query several bits of information. You can use

$ module load netcdf-c/4.10.0-openmpi-5.0.10-oneapi-2025.3.2
$ nc-config --all

to see all details. A similar tool is also available for the netcdf-fortran library:

$ module load netcdf-fortran/4.6.2-openmpi-5.0.10-oneapi-2025.3.2
$ nf-config --all

It is advisable to use the ldd program to verify that your program has been correctly linked. Check that all library paths match the dependency information and, in particular, that no library is reported as “not found”:

$ LD_LIBRARY_PATH= ldd myprog
On this page