Python virtual environments#

We describe how to create conda and venv environments. These environment have an independent set of packages.

See Python for our Python/Conda modules and JupyterHub for our JupyterHub instances.

Install or build packages inside an interactive job on the target cluster

Install or build packages using an interactive job on the target cluster (Alex, Fritz, TinyGPU, Woody) to make sure GPU support and hardware features are properly autodetected on initial installation.

Not all compute nodes have direct internet access. Configure a proxy to enable access, either in the shell:

export http_proxy=http://proxy:80
export https_proxy=http://proxy:80

or as argument, e.g. for pip:

in the shell: pip install --proxy http://proxy:80 <package-name>
inside Jupyter Notebook/JupyterHub: %pip install --proxy http://proxy:80 <package-name>

If you are not inside a conda/venv environment add the --user flag to pip.

Conda environments#

First time only initialization#

The following steps only have to be performed once on our systems. They will initialize conda and cause conda to store packages and environments under $WORK instead of $HOME in order to save space in the latter.

Summary of all the following steps for easy copy and paste:

if [ ! -f ~/.bash_profile ]; then
  echo "if [ -f ~/.bashrc ]; then . ~/.bashrc; fi" > ~/.bash_profile
fi
module add python
conda config --add pkgs_dirs $WORK/software/private/conda/pkgs
conda config --add envs_dirs $WORK/software/private/conda/envs

Detailed explanation of the steps

If not already exists, create the file ~/.bash_profile (located in your $HOME) with the following content:
```
if [ -f ~/.bashrc ]; then . ~/.bashrc; fi
```
- This ensures the file ~/.bashrc, which will later be created/changed, to be loaded.
- See the bash documentation for details.

Ensure you have a Python module loaded:

module list
# output should contain a Python module:
# Currently Loaded Modulefiles:
# 1) python/3.9-conda

Store newly installed conda packages and conda environments under $WORK to save space in $HOME by executing:

conda config --add pkgs_dirs $WORK/software/private/conda/pkgs
conda config --add envs_dirs $WORK/software/private/conda/envs

Check the configuration is used, note that the variable $WORK will be expanded to the real path:

conda info
# output should contain:
#     package cache : /apps/python/...
#                     <real path of $WORK>/software/private/conda/pkgs
#  envs directories : /apps/python/...
#                     <real path of $WORK>/software/private/conda/envs

For more options see .condarc.

Conda environments can also be used for package management (and more). You can share conda environments with co-workers by having them add your environment path to their envs_dirs entry in ~/.condarc as well.

Creating conda environments#

You can create conda environments on our systems. Ensure you have initialized conda as described above. For GPU support of your packages you must use an interactive job.

Create a conda environment by executing:
```
conda create -n <env. name> python=<py. version>
```
- This creates a conda environment named <env. name>
- The new environment uses Python of the specified version <py. version>.
Activate the environment:
```
conda activate <env. name>
```
Your prompt should now be prefixed with the name of the environment like (<env. name>).

When you install now packages via conda or pip they will end up within the conda environment, therefore no --user option is needed.

An alternative way to build a conda environment is by using an environment.yml file. This method comes in very handy, if you intend to copy your environment from a local machine to our hpc systems. Or to copy between clusters.

Create a yml-file on your system with the original conda environment activated: <conda env export --from-history --file environment.yml>
Copy the environment.yml to the target system and create a new environment with: conda env create -f environment.yml

To activate a conda environment inside a batch job use:

module add python
conda activate <env. name>

Virtual environments with `venv`#

With venv you can create lightweight virtual environments. When using pip inside an activated venv environment packages are automatically installed there.

conda does not recognize venv environments

Installing packages via conda from inside an activated venv environment will fail. Conda does not work together with such an environment. For this you need a to create a conda environment.

Preparation: ensure a Python module is loaded.
Create a venv virtual environment in <path to env>:
```
python3 -m venv <path to env>
```
For example: python3 -m venv $WORK/venvs/testing
Activate the environment:
```
source <path to env>/bin/activate
```
For example: source $WORK/venvs/testing/bin/activate
The environment is activated when your prompt is prefixed with the directory name of the virtual environment like (<directory>). In case of our example your prompt would start with (testing).

Using environments in Jupyter Notebooks and JupyterHub#

Python environments can be made available in Jupyter Notebooks and JupyterHub by installing the ipykernel package.

conda environment:
1. Install the ipykernel package
```
conda install -n <env. name> ipykernel
```
  where <env. name> denotes the conda environment you want to use.
2. Only required for Jupyter Notebooks: register environment
```
conda run -n <env. name> python3 -m ipykernel --user --name <env. name>
```
  An existing kernel with the same name as <env. name> will be overridden.
Python venv environment:
1. Activate the venv environment, here named <env. name>.
2. Install ipykernel package and register it
```
pip install ipykernel
python3 -m ipykernel install --user --name=<env. name>
```
  An existing kernel with the same name as <env. name> will be overridden.