ModVis#

Overview#

In recent years, the automated monitoring of running simulations in (near-)real time has evolved to one of the important aspects of the Earth system modeling workflows. This kind of monitoring has proved extremely useful for control of the progress and status of a running simulation, for assessment of the physical experiment state (e.g. existence of drifts in global time series) and for getting a first impression of the reaction of the model to the changing external forcing (e.g. increasing greenhouse gas concentrations in the atmosphere) or model tuning parameters.

DKRZ provides a web server infrastructure to allow for easy presentation of and access to the monitoring data. The intention behind this service is to relieve the climate scientists from technical aspects of web server configuration and maintenance and to let them benefit from convenient word-wide access to monitoring data using any web browser. However, the collection and preparing of the actual monitoring data is not part of this service. This means, the graphical representation of monitoring data and generation of (HTML-formatted) webpages remains in the responsibility of the experiment supervisor. Usually, this requires the following additional steps:

Selection of parameters of interest to be monitored (e.g. global air surface temperature, radiation fluxes at the surface and TOA, total precipitation etc. for the atmospheric models; global SSTs, AMOC, sea ice extent and thickness, sea level change etc. for ocean models).
Writing of additional data processing and visualization routines. Thereby, the identified monitoring parameters can be calculated (and sometimes visualized) within a running model or by using separate post-processing routines, which are integrated into your model runtime environment and automatically called after each model simulation chunk is finished.

The experiment monitoring data can be supplemented with additional routine model evaluations (e.g. comparison with observations or other experiments) and diagnostics of climate mean state and climate variability.

Prerequisites and Usage#

The ModVis resource is available to all DKRZ users. To add this resource to your HLRE project, the project administrator needs to contact the User Support first.

Please be aware that due to hardware limitations, the project quota for file storage on ModVis is 1GB. This might be increased upon request in special cases.

After a project has been approved for ModVis, the project administrator can activate the ModVis permission for project members on a per-user basis at luv.dkrz.de:

After the approval of the user update request by DKRZ Beratung (which is in general accomplished within 1 hour on work days), the user will be added to the group modvis and can access the ModVis server using the Secure Shell (ssh):

$ ssh -X <userid>@modvis.dkrz.de

This host is only reachable from the HPC nodes. Upon first login, the user home directory will be created automatically. Instructions on how to set up password-less login to ModVis using public key iauthentication can be found here. Your HOME directory is not intended for storaging large data. Please, use it for storage of small files (for example, shell startup files ~/.bash_profile, ~/.profile, ~/.bashrc etc.) only.

For reasons of robustness, the Lustre file system of the HPC system is not mounted on the ModVis machine. Therefore, the monitoring data needs to be transferred from the HPC sytem to ModVis via user written transfer scripts employing (ssh-based) data copying tools as, for example, rsync or scp.

The created webpages and images should be stored in the project directory

/var/www/projects/<project-id>
/var/www/projects/<project-id>/<my_subdir>

where the <project-id> is a placeholder for the actual project account (e.g. eg0101) and <my_subdir> is the name of a (user) sub-directory. Usually, the data must be world-readable to allow access by the web server.

The current data policy is to store only small data files (images, HTML pages, plain text files etc.) suitable for access via web browser on the ModVis machine. Data files containing large data amounts (e.g. netCDF files with 2D or 3D data fields) will rapidly exhaust the available disk storage capacity. All data stored on ModVis is backed up on a daily basis.

The created webpages are accessible world-wide under the following generic URL https://modvis.dkrz.de/<project-id>/<my_subdir>, where <my_subdir> is a sub-directory containing the index.html file.

Password protection#

The accessibility of the webpages can be protected with a password. For this purpose you need to create .htaccess and .htpasswd files in the project directory. The Apache web-server configuration file .htaccess has the following content:

AuthType Basic
AuthName "Password Required"
AuthUserFile "/var/www/projects/<project-id>/.htpasswd"
Require valid-user

To generate the .htpasswd file containing login data (username and encrypted password) we recommend to use the htpasswd command, for example:

$ cd /var/www/projects/<project-id>
$ htpasswd -cm ./.htpasswd <username>

# You will be prompted to enter a password and to repeat it
New password:
Re-type new password:
Adding password for user <username>

where <username> can be freely selected. Please, memorize the chosen password since it will be stored in encrypted form and can not be looked up in the .htpasswd file later.

To display the usage of the htpasswd command and the meaning of the options (-c -m) used above, use the -h option:

$ htpasswd -h

Now, to view the password protected area (project directory and all sub-directories) in webrowser user will be requested to login with username and password:

Finer grained definition of access permissions (on the directory basis) is possible by creating further .htaccess configuration files that override settings in the top directory.

Examples#

Online monitoring in STORM project#

To illustrate how the ModVis web server might be used, we provide a link to the actual monitoring data from the first coupled high-resolution experiments performed within STORM project at DKRZ in 2012-2013. The horizontal model resolution in these experiments is 50 km in the atmospheric model and 10 km in the ocean model.