Python environment locations

Kernels are based on python environments created with conda, virtualenv or other package manager. In some cases, the size of the environment can tremendously grow depending on the installed packages. The default location for python files is the $HOME directory. In this case, it will quickly fill your quota. In order to avoid this, we suggest that you create/store python files in other directories of the filesystem on Mistral.

Possible locations

The following are two alternative locations where you can create your Python environment:

  • WORK (depends on the project) you can use this option if you are working on the same workflow and you want to share the same python env among the project members

  • SCRATCH (my favourite and theoritacally fast) for testing new packages/releases. Files are deleted 14 days since the last file access.

This link gives you detailed information on the filesystem on Mistral.

Example

In this example, we will show you how to create conda environment in different location (SCRATCH).

$ module load python3/unstable
$ conda create --prefix /scratch/[a,b,g,k,m,u]/<userid>/<new_your_conda_env>
$ source activate /scratch/[a,b,g,k,m,u]/<userid>/<new_conda_env>

That’s it, you can now install other packages.

Note

For different locations, you just need to change the prefix.

Automate the prefix

Warning

to be used with caution. Further information on .condarc can be found [here](https://docs.conda.io/projects/conda/en/latest/user-guide/configuration/use-condarc.html).

If you would like to set up the same prefix for future use, you need a .condarc on your home directory with the following content:

envs_dirs:
    - /scratch/[a,b,g,k,m,u]/<userid>/

Now, you can create a conda env with the standard command:

$ conda create -n my_conda_env

the new environment will be automatically created in /scratch/[a,b,g,k,m,u]/<userid>/.

Note

Creating kernels for jupyterhub based on these conda env is possible. The location for the packages is automatically set when you run ipykernel install …