- 29 November 2021
2D Climate data can be sampled using different grid types and topologies, which might make a difference when it comes to data analysis and visualization. As the grid lines of regular or rectilinear grids are aligned with the axes of the geopgraphical lat-lon coordinate system, these model grids are relatively easy to deal with. A common, but more complex case is that of a curvilinear or a rotated (regional) grid. In this blog article we want to illuminate this case a bit; we describe how to identify a curvilinear grid, and we demonstrate how to visualize the data using the “normal” cylindric equidistant map projection.
Data can not only be stored in different file formats (e.g. netCDF, GRIB), but also in different data structures. Besides its spatial dimension (e.g. 1D, 2D, 3D), we need to have a closer look at the grid and the topology used. As the time dependency of the data is encoded as the time dimension, a variable might be called a 3D variable although the spatial grid is only 2D.
- 25 November 2021
including the new ICON-ESM-LR model primarily published at DKRZ.
A first ensemble set of simulations from the ESM ICON-ESM-LR for the DECK experiments is available including the experiments
- 02 July 2021
We proudly 🥳 announce that the CDP is extended by new sets of CMIP6 data primarily published at DKRZ. We also published new versions of corrected variables for the MPI-ESM1-2 Earth System Models.
The ensemble set of simulations from the ESM MPI-ESM1-2-HR for the dcppA-hindcast experiment is completed by another 5 realizations (8.5TB). In total, this set consists of about 10 realizations for 60 initialization years in the interval from 1960-2019 resulting in 595 realizations and 31 TB. For each realization, about 100 variables are available for a simulation time of about 10 years.
- 23 June 2021
We proudly announce new publications of model simulations when we publish them at our DKRZ ESGF node. We also keep you updated about the status and the services around the CMIP Data Pool. Find extensive documentions under this link.
- 29 March 2021
- 03 March 2021
In this post we want to answer a few questions which may arise for project administrators and principal investigators at DKRZ. Some of the dates for requesting new resource allocations will be different in 2021. From 2022 on we will return to the usual schedule.
Your project will be automatically extended with the same resources as for the current allocation period. After July 1, 2021, you can continue working on Mistral as you did before.
- 05 January 2021
Slurm config on Mistral has been updated to fix an issue related to memory use.
Prior the update, some Slurm jobs continue consuming the available
memory (and even swap) of the allocated node and exceed the allocated
memory (set in
srun). If this occurs, it also affect
- 16 November 2020
According to the official Web site,
Dask jobqueue can be used to
deploy deploy Dask on job queuing systems like PBS, Slurm, MOAB, SGE,
LSF, and HTCondor. Since the queuing system on Mistral is Slurm, we are
going to show how to start a Dask cluster there. The idea is simple as
described here. The difference is that the workers can be distributed
through multiple nodes from the same partition. Using Dask jobqueue will
Dask cluster as a Slurm jobs.
In this case, Jupyterhub will often play an interface role and the Dask can use more than the allocated resources to your jupyterhub session (profiles).
- 13 November 2020
Extensions bring additional interesting features to Jupyter*. Depending on the workflow in the notebook, users can install/enable extensions when required. Although is easy to add extensions to both Jupyter notebook an lab, the process can be sometimes annoying based on where jupyter is served from.
In general, installing and enabling extensions in your laptop or using
start-jupyter script is straightforward, especially when the
developers well describe their extensions. There should be no
restrictions or permissions issues, just follow the instructions.
- 05 November 2020
can’t use NCL (Python) as kernel in Jupyter
This tutorial won’t work
- 07 October 2020
you are using singularity containers
you need jupyter notebooks
- 07 October 2020
created your own conda env
- 05 October 2020
We introduced a new feature to the preset and advanced options form.
This is a nice feature especially for the advanced options form, which
contain many fields. You can also reset the options to their initial
values by clicking on
reset. The form options are saved in the
client’s browser every 10 seconds and are not lost if:
the browser crashes
- 02 October 2020
Recently, we deployed a new version of Singularity: 3.6.1. The old version is not available anymore due to many bugs reported by some users.
Errors like these are now fixed:
- 01 October 2020
vs code is your favorite IDE
interested to use the remote extension
- 30 September 2020
According to this link:
The Python cf package is an Earth Science data analysis library that is built on a complete implementation of the CF data model. The cf package implements the CF data model 1 for its internal data structures and so is able to process any CF-compliant dataset. It is not strict about CF-compliance, however, so that partially conformant datasets may be ingested from existing datasets and written to new datasets. This is so that datasets that are partially conformant may nonetheless be modified in memory.
- 25 September 2020
Each Jupyter notebook is running as a SLUM job on MIstral. By default,
stderr of the SLURM batch job that is spawned by
Jupyterhub is written to your
HOME directory on the HPC system. In
order to make it simple to locate the log file:
if you use the
preset options form: the log file is named
- 23 September 2020
Have you ever tried to install/use a software on Mistral and seen a message like this?
This is for example one of the reasons why PyTorch is not available
in our python3 module. Those software packages require a newer version
glibc. Unfortunately, most of Mistral nodes are based on CentOS 6
kernel. To check the version of
- 18 September 2020
There are multiple ways to create a dask cluster, the following is only an example. Please consult the official documentation. The Dask library is installed and can be found in any of the python3 kernels in jupyterhub. Of course, you can use your own python environment.
The simplest way to create a Dask cluster is to use the distributed module:
- 03 September 2020
It is our great pleasure to introduce the DKRZ Tech Talks. In this series of virtual talks we will present services of DKRZ and provide a forum for questions and answers. They will cover technical aspects of the use of our compute systems as well as procedures such as compute time applications and different teams relevant to DKRZ such as our machine learning specialists. The talks will be recorded and uploaded afterwards for further reference.
Go here for more information.