Skip to content

Python, Anaconda#

We recommend the usage of our Python modules that provide an Anaconda installation with a wide range of preinstalled packages and the Conda package manager. See Conda environments on how to configure Conda on our systems.

We discourage the usage of the system-wide installed Python on our systems as it is rather old.

Using Python/Anaconda#

  • List available Python versions:
    module avail python

  • Load the default Python module: 

    module load python
    Note: the version of the default Python module might change over time.

  • Load a specific Python version:

    module load python/<version>

Installing packages#

Install/build packages inside an interactive job on the target cluster

It is recommended to install/build packages using an interactive job on the target cluster to make sure all hardware features and GPU support can be used properly. Make sure to load modules that might be needed by your Python code, e.g. CUDA/CUDNN for GPU support.

Some of our cluster nodes cannot directly connect to the internet and require a proxy to be configured. In your shell on the compute node enter:

export http_proxy=http://proxy:80
export https_proxy=http://proxy:80

Installing packages via pip#

pip is a package manager for Python. By default pip will install packages system wide, which will fail due to missing permissions.

Packages can be installed locally by adding --user:

pip install --user <package-name>

Inside a Jupyter notebook use:

%pip install --user --proxy http://proxy:80 <package-name>

This command also configures the usage of a proxy through the --proxy flag.

Alternatively create a conda/venv environment into which you install your packages.

Adjusting installation path of packages#

By default packages installed through pip or distutils with the --user flag are installed under the $HOME directory. As quota on $HOME is scarce and we regularly backup and snapshot $HOME it makes sense, to install packages on a filesystem without backup, like $WORK.

The installation path for pip and distutils can be changed by setting the environment variable PYTHONUSERBASE.

To install packages that do not need to be backed up into $WORK run: 

export PYTHONUSERBASE=$WORK/software/private
Future packages will be installed into $WORK/software/private.

To change the installation path permanently add the code snippet to your ~/.bashrc on the clusters.

Sharing packages and environments with coworkers#

If you intend to share the packages/environments with your coworkers consider wrapping the python package inside a module. For information on the module system see your HPC-Cafe from March 2020.

Setup and define the target folder with environment variable PYTHONUSERBASE. Install the package as above. Your module file needs to add to PYTHONPATH the site-packages folder and to PATH the bin folder, if the package comes with binaries.

For an example see the module quantumtools on woody.