Compiling and Linking#
This page briefly describes how to build software on Levante, i.e. to generate executable files from source code files (typically written in C/C++ or Fortran).
Compilers#
As listed below, we provide a selection of high quality compilers on Levante. Compilers are not loaded by default. You have to use the module environment to access them. We recommend to specify the module version number explicitly otherwise the lexicographically highest version is loaded by default, which might not be latest or desired version.
Intel Compilers#
For most applications we recommend to use the Intel compilers. The compiler version can be selected by loading the corresponding module file, for example:
# Use the "latest" versions of Intel compiler
$ module load intel-oneapi-compilers
# Use a specific version of Intel compiler
$ module load intel-oneapi-compilers/2022.0.1-gcc-11.2.0
Intel compilers are now part of the Intel® oneAPI
Toolkits, which explains the use of the oneapi
suffix in the module name.
The specific compiler names are:
icc
- for C source codeifort
- for Fortran source codeicpc
- for C++ source code
The table below lists some useful compiler options that are commonly used for the Intel compilers.
Option |
Descripion |
---|---|
-qopenmp |
Generates multi-threaded code based on the OpenMP directives |
-g |
Creates debugging information in the object files. This is necessary if you want to debug your program |
-O[0-3] |
Sets the optimization level |
-L/path/to/lib |
A path can be given in which the linker searches for libraries |
-Wl,-rpath,/path/to/lib |
Pass |
-D |
Defines a CPP macro |
-U |
Undefines a CPP macro |
-I/path/to/include |
Adds directories to search for include files |
-sox |
Stores information like compiler version, options used etc. in the executable file |
-ipo |
Inter-procedural optimization |
-march=core-avx2 |
Indicates the processor for which code is generated |
-mtune=core-avx2 |
Indicates the processor for which optimizations are performed |
-help |
Displays available compiler options |
For further information, please refer to the man pages of the respective compiler
$ man ifort
$ man icc
$ man icpc
or the comprehensive documentation on Intel website.
Note
Using the compiler option -march=core-avx2
forces the Intel
compiler to use full AVX2 support/vectorization (with FMA
instructions) which might results in binaries that do not produce
MPI decomposition independent results. Adding the option
-no-fma
should solve this issue but could result in slightly
longer runtime.
GNU Compiler Collection (GCC)#
GCC is a suite of compilers for C (gcc
),
C++ (g++
) Fortran (gfortran
), and D (gcd
) programming
languages. You need to load an environment module for gcc to access a
recent version of the GNU compiler suite. The use of the older system
gcc
located in /usr/bin
and provided as part of the base Linux
operating system is generally inadvisable. Example for loading a
recent version of gcc
:
# Use the "latest" versions of gcc compiler
$ module load gcc
# Use a specific version of gcc compiler
$ module load gcc/11.2.0-gcc-11.2.0
NAG#
NAG compilers have proved to
be very useful for debugging und checking if the source code is
standard conforming. It is not appropriate to create model binaries
for production runs. To make NAG compilers available in your shell
environment, you need to load an nag
environment module, e.g.:
$ module load nag/7.1-gcc-11.2.0
NVIDIA HPC SDK#
The NVIDIA HPC SDK contains a compiler suite that is of special interest for users and developers of GPU-ready codes. It supports GPU offloading using OpenMP, OpenACC and CUDA. Please refer to the section on GPU Programming for more information. Since 2020 NVIDIA HPC compilers replaces the PGI compilers.
Compiling and Linking MPI programs#
MPI Libraries#
Currently, two implementations of Message Passing Interface (MPI) library are available on Levante:
No MPI libraries are loaded by default. Similar to compilers, you have to explicitly load an appropriate environment module for a certain MPI implementation.
Note
Because Fortran module files are compiler specific, it is important
to use the MPI installation built with the same compiler (as marked
by the suffixes like intel-2021.5.0
or gcc-11.2.0
in the
MPI modulefile names) as the compiler selected to build your code.
MPI Compiler Wrappers#
It is highly advisabe to use MPI compiler wrappers to compile and link MPI parallel programs. Such wrappers are provided with each MPI library implementation. They automatically build up the MPI environment (i.e. set paths to MPI include files and MPI libraries) to facilitate the compilation and linking steps. The following table shows the names of the Intel compilers as well as names of IntelMPI and OpenMPI compiler wrappers:
Language |
Intel Compiler |
IntelMPI wrapper |
OpenMPI wrapper |
---|---|---|---|
Fortran 90/95/2003 |
|
|
|
Fortran 77 |
|
|
|
C++ |
|
|
|
C |
|
|
|
Examples#
Compile a hybrid MPI/OpenMP program using Intel Fortran compiler and OpenMPI:
$ module load intel-oneapi-compilers/2022.0.1-gcc-11.2.0
$ module load openmpi/4.1.2-intel-2021.5.0
$ mpifort -qopenmp -O2 -march=core-avx2 -fp-model source -o mpi_omp_prog program.f90
Compile a MPI program in Fortran using Intel Fortran compiler and Intel MPI:
$ module load intel-oneapi-compilers/2022.0.1-gcc-11.2.0
$ module load intel-oneapi-mpi/2021.5.0-intel-2021.5.0
$ mpiifort -O2 -march=core-avx2 -fp-model source -o mpi_prog program.f90
Compile a MPI program in Fortran using GCC Fortran compiler and OpenMPI:
$ module load gcc/11.2.0-gcc-11.2.0 openmpi/4.1.2-gcc-11.2.0
$ mpifort -O2 -march=core-avx2 -fp-model source -o mpi_prog program.f90
Note
The computational performance and scalability of MPI applications on Levante can be considerably improved by an optimal choice of the runtime parameters provided by MPI libraries. The appropriate MPI run time settings strongly depend on the type of application and MPI library used. For most MPI versions installed on Levante, we provide some recommendations for MPI environment settings that proved to be beneficial for different model codes commonly used at DKRZ.
Libraries from the software tree#
Many commonly used libraries are available from our software tree
located at /sw/spack-levante
. They are all installed in individual
directories so we can provide different versions and configurations of
the same library. This means that you should carefully chose which
library to link and run your model with.
For many libraries, you can find a module file which tells you where to find the library for compiling and linking. More detailed information can be inquired with the spack command for all installed libraries.
How to build software with netCDF#
NetCDF libraries
are commonly used in climate models for data input and output. If only
the Fortran interface is directly accessed in your program, it is
sufficient to know the installation path of the netcdf-fortran
library and to use this information for building your
software. Netcdf-fortran
installations available on Levante can be
inquired with the module avail
command:
$ module avail netcdf-fortran
Based on the compiler and MPI you want to use, you need to select one
of the available libraries that is compatible and meets your
requirements. The installation path of that library can then be
inferred with the module show
command. For example, if you use
Intel compiler and Open MPI, and you find that
netcdf-fortran/4.5.3-openmpi-4.1.2-intel-2021.5.0
matches your
setup, the following command
$ module show netcdf-fortran/4.5.3-openmpi-4.1.2-intel-2021.5.0
-------------------------------------------------------------------
/sw/spack-levante/spack/modules/netcdf-fortran/4.5.3-openmpi-4.1.2-intel-2021.5.0:
module-whatis {NetCDF (network Common Data Form) is a set of software ...}
conflict netcdf-fortran
prepend-path PATH /sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/bin
...
provides the installation path of the library:
/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g
Its directory containing include files is then:
/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/include
and the path to the library files is:
/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/lib
In a usual setting, you can instruct your compiler and linker to use the netCDF library from this path both for compiling and when running your program using the following options:
$ mpifort -I/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/include \
-L/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/lib -lnetcdff \
-Wl,-rpath,/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/lib \
-o myprog myprog.f90
The -I
option tells the Intel compiler where to find include files
and Fortran module description files (.mod) for
netcdf-fortran
. The -L
option tells the link editor where to
find the netcdf-fortran
library itself.
Since by default the dynamic version of the library will be linked
against, you will also need to encode its run-time search path into
the generated binary using the -Wl,-rpath,
option. This is
necessary since netcdf-fortran
is not installed in a standard
directory (like /usr/lib64
etc.) automatically searched by the
dynamic linker. Otherwise you will get the following error message at
run time:
myprog: error while loading shared libraries: libnetcdff.so.7:
cannot open shared object file: No such file or directory
In general, you will have to add options corresponding to the above for each library you are going to use.
On Levante, libraries in the software tree already have link
information for where to find their respective dependencies. For
example, netcdf-fortran
contains the correct paths to
netcdf-c
, hdf5
, libaec
etc. You won’t need to supply this
information again, unless your program also calls functions from those
libraries directly. In that case, we recommend to use the spack
command as follows to inquire information about the dependency
libraries:
$ spack find -dp netcdf-fortran %intel@2021.5.0 ^openmpi@4.1.2
==> 1 installed package
-- linux-rhel8-zen2 / intel@2021.5.0 ----------------------------
netcdf-fortran@4.5.3 /sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g
netcdf-c@4.8.1 /sw/spack-levante/netcdf-c-4.8.1-2k3cmu
curl@7.61.1 /usr
hdf@4.2.15 /sw/spack-levante/hdf-4.2.15-jvpsue
libaec@1.0.5 /sw/spack-levante/libaec-1.0.5-gij7yv
libjpeg-turbo@1.5.3 /usr
libtirpc@1.2.6 /sw/spack-levante/libtirpc-1.2.6-7laks4
krb5@1.19.2 /sw/spack-levante/krb5-1.19.2-rteqqi
openssl@1.1.1g /usr
zlib@1.2.11 /usr
hdf5@1.12.1 /sw/spack-levante/hdf5-1.12.1-tvymb5
numactl@2.0.14 /sw/spack-levante/numactl-2.0.14-6yawk5
openmpi@4.1.2 /sw/spack-levante/openmpi-4.1.2-yfwe6t
...
parallel-netcdf@1.12.2 /sw/spack-levante/parallel-netcdf-1.12.2-mc24h4
The above command will list the installation of the netcdf-fortran
library built with the specified version of the Intel compiler and
depending on the chosen Open MPI installation and subsequently all of
its dependencies plus the corresponding installation paths.
You can alternatively use the spack hash (which is also part of the package installation path) to infer the above information about dependencies and paths:
$ spack find -dp /k6xq5g
These paths can then be used to fill in additional information needed by build systems for packages that also directly interact with one or more of these other libraries (or just require that information because it is needed on other systems where libraries are not installed completely linked), for example
$ mpifort -I/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/include \
-L/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/lib \
-L/sw/spack-levante/netcdf-c-4.8.1-2k3cmu/lib -lnetcdff -lnetcdf \
-Wl,-rpath,/sw/spack-levante/netcdf-fortran-4.5.3-k6xq5g/lib \
-Wl,-rpath,/sw/spack-levante/netcdf-c-4.8.1-2k3cmu/lib \
-o myprog myprog.f90 util.o
Many libraries provide utilities to facilitate compiling and linking,
e.g. netcdf-fortran
contains the nf-config
command to query several
bits of information. You can use
$ nf-config --all
to see all details.
It is advisable to use the ldd
program to test whether you
succeeded to fully link your program correctly by inspecting if all
paths match what the dependency information tells you and especially
if no library is reported as being “not found”:
$ LD_LIBRARY_PATH= ldd myprog