Gitlab#

GitLab is an open source end-to-end software development platform with built-in version control, issue tracking, code review, CI/CD, and more. General information about gitlab can be found here: http://doc.gitlab.com/ce/

DKRZ offers access to a self-hosted gitlab installation for all DKRZ users with a valid DKRZ account. The details on how to get a DKRZ account can be found here. To grant access for collaboration purposes also to external partners who are not yet part of any DKRZ project, you can use a personal project.

Getting started#

For a general overiew and all details on the usage of gitlab we recommend the official gitlab documentation. Here we only outline the essentials on how to get started with the DKRZ gitlab.

You can access the gitlab webinterface at https://gitlab.dkrz.de. With your first login your gitlab account is created and you can be added as a member to existing projects or create new projects yourself. Each user is allowed to create up to 5 git projects/repositories - please contact support@dkrz.de if you need more resources.

Projects can have different visibility settings:

  • Public - visible to everyoneplease consider creating a group whenever projects should be public - see notes below

  • Internal - visible to other DKRZ GitLab users

  • Private - visible only to project members

When you add new members to your project, you will assign them to a role. You can read about project permissions at https://gitlab.dkrz.de/help/user/permissions.

Note

Please note that our GitLab server is primarily designed for versioning of source code. Therefore, we kindly ask you to not store large files or binary code. Please keep the total amount of repositories (either private or group namespace) below 500MB. Currently, there is no explicit quota set, but we will inform you whenever usage exceeds the threshold of 500 MB in sum. In seldom cases we will have to temporarily deactivate your access if a misuse of the service is detected.

Groups and persistent (public) projects#

Gitlab allows to define groups to gather related projects together and grant members access to several projects at once. All projects within the same group will end up at the same namespace - this is also true for projects created under the users namespace, e.g.

Note

Once a users personal DKRZ account is deleted (e.g. due to the end of all participating HLRE project), all personal Gitlab projects will be eventually also deleted. Therefore, we strongly encourage users to define a group with more than one owner whenever the project and its associated repository should be public and/or persistent, so that the persistence of the project does not rely on the account of one single user.

Version control with git#

To get started with Git repositories and version control, we cross-refer to these sites:

Here we only outline the essentials on how to use git on DKRZ resources.

On levante you should use the latest Git version when working with GitLab:

$ module add git

As a new user you will want to add an SSH key to your GitLab profile. This allows you to access project repositories with a Git client without having to type your password every time. You can add an ssh key to your GitLab account at https://gitlab.dkrz.de/-/profile/keys.

Cloning an existing repository using SSH keys is done via:

$ git clone git@gitlab.dkrz.de:me/my_project.git

You have to upload your public SSH key to the gitlab server first! Alternatively, one can use HTTPS protocol to clone a repository:

$ git clone https://gitlab.dkrz.de/me/my_project.git

Using Git for large files#

Managing large files such as audio, video and graphics files has always been one of the shortcomings of Git. The general recommendation is to not have Git repositories larger than 1GB to preserve performance. At DKRZ we kindly ask our users to not exceed 100MB per Git repository.

Nevertheless, large files can be managed by Git using the Git LFS client (https://git-lfs.github.com/). It replaces large files with text pointers inside Git, while storing the file contents on a remote server. To manage less than 10 GB of large files you can use the internal lfs storage of our gitlab server. It is enabled by default in the project settings. For details on how to define which files should be managed by Git LFS, please consult the official documentation.

In case you need to manage 10 GB and more of large files DKRZ provides the option to store these files on the cloud storage. Yet this solution is subject to some limitations and is not officially supported.

Git LFS with DKRZ cloud storage#

As a prerequisite to use Git LFS with the DKRZ cloud storage user have to:

  • install Git LFS extension (to check that your git client provides git lfs, use e.g. git lfs --version)

  • prepare an empty container at DKRZ cloud storage, e.g. via Swiftbrowser accessible at https://swiftbrowser.dkrz.de

  • disable the usage of the internal LFS storage in the project settings under General/Visibility, project features, permissions

  • make sure that no single file for Git LFS is larger than 5GB, which is the current maximum of our Git LFS client

Note

Please note that we can only guarantee the persistence of the backend of our cloud storage, whereas the frontend might change over the years. This implies that the stored data will persist, but the links to the data, stored in the history of your repository, might loose their validity.

For new local repositories that use Git LFS (and also for bare repositories that have not been setup for Git LFS), one needs to initialize lfs support and specify the LFS server – in the example ‘container’ should be the name of the swift container that was prepared initially

$ cd repo
$ git lfs install
$ URL="https://git-lfs-swift.dkrz.de/container"
$ git config lfs.url $URL
$ git config lfs.$URL/.access basic
$ git config lfs.batch true
$ git config --file=.lfsconfig lfs.url $URL

For details on how to define which files should be managed by Git LFS, please consult the official documentation.

Pushing to the enabled repository will contact the git-lfs server and ask for swift credentials authentication where

  • Username is ‘swift-account;ldap-username’ (with semicolon between account and username)

  • Password is your usual LDAP password

All tracked files will now be stored at cloud storage in the specified container for specified swift-account.

Using gitlab CI#

GitLab offers a continuous integration service to automatically build, test and deploy your code. Detailed documentations on the topic can be found here.

The DKRZ welcomes and supports the use of gitlab CI to improve and ease the process of code maintenance. Yet, due to the fact, that the configuration, authorization and authentication of gitlab CI is possible only on gitlab-project level - not on user level - the use to test and run code directly on levante is limited and currently not supported.

But, DKRZ offers different runners to automatically build and publish the documentation of your code. You can find more details on the usage of these runners here.

Gitlab Pages#

With GitLab Pages, you can publish static websites directly from a repository in GitLab. Please refer to the official Gitlab Pages documentation for details.

Note

If you create a Gitlab Pages website, it is your responsibility to post a legally valid imprint and privacy statement that accurately describes how you collect, use, and share personal information and other visitor information, and how you comply with applicable data privacy laws, rules, and regulations. Please note that DKRZ may collect Technical Information from visitors to your Gitlab Pages website, including logs of visitor IP addresses, to maintain the security and integrity of the website and service.

We genrally discourage the dynamic integration of external contents (such as google fonts) in your webpage as this may conflict with the General Data Protection Regulation.

On this page