Compiling and Linking#

This page briefly describes how to build software on Levante, i.e. to generate executable files from source code files (typically written in C/C++ or Fortran).

Compilers#

As listed below, we provide a selection of high quality compilers on Levante. Compilers are not loaded by default. You have to use the module environment to access them. We recommend to specify the module version number explicitly otherwise the lexicographically highest version is loaded by default, which might not be latest or desired version.

Intel Compilers#

For most applications we recommend to use the Intel compilers. The compiler version can be selected by loading the corresponding module file, for example:

# Use the "latest" versions of Intel compiler
$ module load intel-oneapi-compilers

# Use a specific version of Intel compiler
$ module load intel-oneapi-compilers/2022.0.1-gcc-11.2.0

Intel compilers are now part of the Intel® oneAPI Toolkits, which explains the use of the oneapi suffix in the module name.

The specific compiler names are:

icc - for C source code
ifort - for Fortran source code
icpc - for C++ source code

The table below lists some useful compiler options that are commonly used for the Intel compilers.

Option	Descripion
`-qopenmp`	Generates multi-threaded code based on the OpenMP directives
`-g`	Creates debugging information in the object files. This is necessary if you want to debug your program
`-O[0-3]`	Sets the optimization level
`-L/path/to/lib`	A path can be given in which the linker searches for libraries
`-Wl,-rpath,/path/to/lib`	Pass `-rpath` option to the linker, which adds a directory to the runtime library search path
`-D`	Defines a CPP macro
`-U`	Undefines a CPP macro
`-I/path/to/include`	Adds directories to search for include files
`-sox`	Stores information like compiler version, options used etc. in the executable file
`-ipo`	Inter-procedural optimization
`-march=core-avx2`	Indicates the processor for which code is generated
`-mtune=core-avx2`	Indicates the processor for which optimizations are performed
`-help`	Displays available compiler options

For further information, please refer to the man pages of the respective compiler

$ man ifort
$ man icc
$ man icpc

or the comprehensive documentation on Intel website.

Note

Using the compiler option -march=core-avx2 forces the Intel compiler to use full AVX2 support/vectorization (with FMA instructions) which might results in binaries that do not produce MPI decomposition independent results. Adding the option -no-fma should solve this issue but could result in slightly longer runtime.

GNU Compiler Collection (GCC)#

GCC is a suite of compilers for C (gcc), C++ (g++) Fortran (gfortran), and D (gcd) programming languages. You need to load an environment module for gcc to access a recent version of the GNU compiler suite. The use of the older system gcc located in /usr/bin and provided as part of the base Linux operating system is generally inadvisable. Example for loading a recent version of gcc:

# Use the "latest" versions of gcc compiler
$ module load gcc

# Use a specific version of gcc compiler
$ module load gcc/11.2.0-gcc-11.2.0

NAG#

NAG compilers have proved to be very useful for debugging und checking if the source code is standard conforming. It is not appropriate to create model binaries for production runs. To make NAG compilers available in your shell environment, you need to load an nag environment module, e.g.:

$ module load nag/7.1-gcc-11.2.0

NVIDIA HPC SDK#

The NVIDIA HPC SDK contains a compiler suite that is of special interest for users and developers of GPU-ready codes. It supports GPU offloading using OpenMP, OpenACC and CUDA. Please refer to the section on GPU Programming for more information. Since 2020 NVIDIA HPC compilers replaces the PGI compilers.

Compiling and Linking MPI programs#

MPI Libraries#

Currently, two implementations of Message Passing Interface (MPI) library are available on Levante:

OpenMPI Version 4.1.2
IntelMPI Version 2021.5.0

No MPI libraries are loaded by default. Similar to compilers, you have to explicitly load an appropriate environment module for a certain MPI implementation.

Note

Because Fortran module files are compiler specific, it is important to use the MPI installation built with the same compiler (as marked by the suffixes like intel-2021.5.0 or gcc-11.2.0 in the MPI modulefile names) as the compiler selected to build your code.

MPI Compiler Wrappers#

It is highly advisabe to use MPI compiler wrappers to compile and link MPI parallel programs. Such wrappers are provided with each MPI library implementation. They automatically build up the MPI environment (i.e. set paths to MPI include files and MPI libraries) to facilitate the compilation and linking steps. The following table shows the names of the Intel compilers as well as names of IntelMPI and OpenMPI compiler wrappers:

Language	Intel Compiler	IntelMPI wrapper	OpenMPI wrapper
Fortran 90/95/2003	`ifort`	`mpiifort`	`mpifort` (or `mpif90`, deprecated)
Fortran 77	`ifort`	`mpiifort`	`mpifort` (or `mpif77`, deprecated)
C++	`icpc`	`mpiicpc`	`mpic++`, `mpicxx`, `mpiCC`
C	`icc`	`mpiicc`	`mpicc`

Examples#

Compile a hybrid MPI/OpenMP program using Intel Fortran compiler and OpenMPI:

$ module load intel-oneapi-compilers/2022.0.1-gcc-11.2.0
$ module load openmpi/4.1.2-intel-2021.5.0
$ mpifort -qopenmp -O2 -march=core-avx2 -fp-model source -o mpi_omp_prog program.f90

Compile a MPI program in Fortran using Intel Fortran compiler and Intel MPI:

$ module load intel-oneapi-compilers/2022.0.1-gcc-11.2.0
$ module load intel-oneapi-mpi/2021.5.0-intel-2021.5.0
$ mpiifort -O2 -march=core-avx2 -fp-model source -o mpi_prog program.f90

Compile a MPI program in Fortran using GCC Fortran compiler and OpenMPI:

$ module load gcc/11.2.0-gcc-11.2.0 openmpi/4.1.2-gcc-11.2.0
$ mpifort -O2 -march=core-avx2 -fp-model source -o mpi_prog program.f90

Note

The computational performance and scalability of MPI applications on Levante can be considerably improved by an optimal choice of the runtime parameters provided by MPI libraries. The appropriate MPI run time settings strongly depend on the type of application and MPI library used. For most MPI versions installed on Levante, we provide some recommendations for MPI environment settings that proved to be beneficial for different model codes commonly used at DKRZ.

Libraries from the software tree#

Many commonly used libraries are available from our software tree located at /sw/spack-levante. They are all installed in individual directories so we can provide different versions and configurations of the same library. This means that you should carefully chose which library to link and run your model with.

For many libraries, you can find a module file which tells you where to find the library for compiling and linking. More detailed information can be inquired with the spack command for all installed libraries.

How to build software with netCDF#

NetCDF libraries are commonly used in climate models for data input and output. If only the Fortran interface is directly accessed in your program, it is sufficient to know the installation path of the netcdf-fortran library and to use this information for building your software. Netcdf-fortran installations available on Levante can be inquired with the module avail command:

$ module avail netcdf-fortran

Based on the compiler and MPI you want to use, you need to select one of the available libraries that is compatible and meets your requirements. The installation path of that library can then be inferred with the module show command. For example, if you use Intel compiler and Open MPI, and you find that netcdf-fortran/4.5.3-openmpi-4.1.2-intel-2021.5.0 matches your setup, the following command

$ module show netcdf-fortran/4.5.3-openmpi-4.1.2-intel-2021.5.0

-------------------------------------------------------------------
/sw/spack-levante/spack/modules/netcdf-fortran/4.5.3-openmpi-4.1.2-intel-2021.5.0:

module-whatis   {NetCDF (network Common Data Form) is a set of software ...}
conflict        netcdf-fortran
prepend-path    PATH /sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/bin
...

provides the installation path of the library:

/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g

Its directory containing include files is then:

/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/include

and the path to the library files is:

/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/lib

In a usual setting, you can instruct your compiler and linker to use the netCDF library from this path both for compiling and when running your program using the following options:

$ mpifort -I/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/include \
          -L/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/lib -lnetcdff \
          -Wl,-rpath,/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/lib \
          -o myprog myprog.f90

The -I option tells the Intel compiler where to find include files and Fortran module description files (.mod) for netcdf-fortran. The -L option tells the link editor where to find the netcdf-fortran library itself.

Since by default the dynamic version of the library will be linked against, you will also need to encode its run-time search path into the generated binary using the -Wl,-rpath, option. This is necessary since netcdf-fortran is not installed in a standard directory (like /usr/lib64 etc.) automatically searched by the dynamic linker. Otherwise you will get the following error message at run time:

myprog: error while loading shared libraries: libnetcdff.so.7:
cannot open shared object file: No such file or directory

In general, you will have to add options corresponding to the above for each library you are going to use.

On Levante, libraries in the software tree already have link information for where to find their respective dependencies. For example, netcdf-fortran contains the correct paths to netcdf-c, hdf5, libaec etc. You won’t need to supply this information again, unless your program also calls functions from those libraries directly. In that case, we recommend to use the spack command as follows to inquire information about the dependency libraries:

$ spack find -dp netcdf-fortran %intel@2021.5.0 ^openmpi@4.1.2

==> 1 installed package
-- linux-rhel8-zen2 / intel@2021.5.0 ----------------------------
netcdf-fortran@4.5.3                /sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g
    netcdf-c@4.8.1                  /sw/spack-levante/netcdf-c-4.8.1-2k3cmu
        curl@7.61.1                 /usr
        hdf@4.2.15                  /sw/spack-levante/hdf-4.2.15-jvpsue
            libaec@1.0.5            /sw/spack-levante/libaec-1.0.5-gij7yv
            libjpeg-turbo@1.5.3     /usr
            libtirpc@1.2.6          /sw/spack-levante/libtirpc-1.2.6-7laks4
                krb5@1.19.2         /sw/spack-levante/krb5-1.19.2-rteqqi
                    openssl@1.1.1g  /usr
            zlib@1.2.11             /usr
        hdf5@1.12.1                 /sw/spack-levante/hdf5-1.12.1-tvymb5
            numactl@2.0.14          /sw/spack-levante/numactl-2.0.14-6yawk5
            openmpi@4.1.2           /sw/spack-levante/openmpi-4.1.2-yfwe6t
        ...
        parallel-netcdf@1.12.2      /sw/spack-levante/parallel-netcdf-1.12.2-mc24h4

The above command will list the installation of the netcdf-fortran library built with the specified version of the Intel compiler and depending on the chosen Open MPI installation and subsequently all of its dependencies plus the corresponding installation paths.

You can alternatively use the spack hash (which is also part of the package installation path) to infer the above information about dependencies and paths:

$ spack find -dp /k6xq5g

These paths can then be used to fill in additional information needed by build systems for packages that also directly interact with one or more of these other libraries (or just require that information because it is needed on other systems where libraries are not installed completely linked), for example

$ mpifort -I/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/include \
          -L/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/lib \
          -L/sw/spack-levante/netcdf-c-4.8.1-2k3cmu/lib -lnetcdff -lnetcdf \
          -Wl,-rpath,/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/lib \
          -Wl,-rpath,/sw/spack-levante/netcdf-c-4.8.1-2k3cmu/lib \
          -o myprog myprog.f90 util.o

Many libraries provide utilities to facilitate compiling and linking, e.g. netcdf-fortran contains the nf-config command to query several bits of information. You can use

$ nf-config --all

to see all details.

It is advisable to use the ldd program to test whether you succeeded to fully link your program correctly by inspecting if all paths match what the dependency information tells you and especially if no library is reported as being “not found”:

$ LD_LIBRARY_PATH= ldd myprog