Python Software Installation: Virtual Environments

Setting up a virtual environment

A Python virtual environment is an environment that is based on an existing Python installation but has an independent set of packages. Since you can create a virtual environment in any directory you can write to, setting up a virtual environment will allow you to install your own Python packages. Each virtual environment has its own set of packages, so you can create multiple to manage the dependencies of specific projects.

There are two options for setting up a virtual environment on CHPC systems. You may either use the setup script (recommended) or create a virtual environment manually. Please note that virtual environments are not portable; you can't move your venv or copy a venv to or from another computer. It is a good idea to keep notes about how you set up your environment so you can re-create it or share it with others if necessary.

Using the setup script (recommended)

The CHPC has created a setup script to help you create a virtual environment and an associated module quickly and easily. To use the setup script, simply run

module load python3  # Load a version of Python as the basis for the venv
/uufs/chpc.utah.edu/sys/installdir/python/modules/venv_setup

and the script will walk you through the setup process. If you choose to set up a module, be sure to save and use the module use and module load commands from the output of the script to load your environment.

Manual setup (for advanced users)

If you have already set up a virtual environment with the setup script, you can skip this section and proceed directly to the instructions for installing packages.

Initializing the virtual environment

The first step to setting up a custom Python environment is to make a virtual environment (venv). Throughout this page, we refer to a venv named my-environment in the my-python-venvs directory. You are welcome to choose different names that better describe your project.

module load python3  # Load a version of Python as the basis for the venv
python -m venv --system-site-packages ~/my-python-venvs/my-environment

The --system-site-packages flag will include packages from the base installation. For CHPC-installed Python versions (those loaded with module load), this includes packages like NumPy and SciPy, which have been installed by CHPC staff.

Once you have created a venv, you need to load it. For this, we recommend setting up a module.

Setting up and loading the module

The CHPC uses modules to manage software. Setting up a module for your Python environment will allow you to load it the same way you access other software on CHPC resources. It will also allow you to use your environment easily with Open OnDemand's Jupyter application (see notes about Jupyter), which is important if you use the web interface to run Python jobs.

To set up a module, you will need to create a script; as an example, we create the file ~/my-python-modules/my-environment/latest.lua (again, feel free to change the name of my-python-modules to suit your needs) with the following contents:

-- The location of your venv relative to your home directory
local venv_location = "my-python-venvs/my-environment"

--[[ You should not need to change any of the following information for most
     Python environments --]]
local path = pathJoin(os.getenv("HOME"), venv_location)
setenv("VIRTUAL_ENV", path)
prepend_path("PATH", pathJoin(path, "bin"))
help("User-installed Python environment")
family("python")

Once you have your module set up, you can load it to use your custom Python environment:

module use ~/my-python-modules
module load my-environment

Once you have loaded your module, you can use the python command to run Python from your virtual environment.

Installing packages

Now that you have a module with a custom Python environment loaded, you can use pip to install your own packages. You may need to upgrade the pip version first:

pip install --upgrade pip

Then, if you would like to install some-common-package, run

pip install some-common-package

to install it in your virtual environment. If you are running Python from your virtual environment and the package you are installing is available on the Python Package Index, your package should be installed and you should be able to access it when you run python. If you encounter issues with permissions, double-check that you are running Python from your virtual environment (the command which python will show you the location of the executable, which should be in the directory you set up earlier; if not, try setting up and loading the module again).

Using Jupyter notebooks and Open OnDemand

To use Jupyter with a custom environment in Open OnDemand, you must first install it (if it is not already present). Once you have loaded your custom Python environment as described above, you can run

pip install jupyter

to install it. Then, when launching the Jupyter app on Open OnDemand, set the "Jupyter Python version" field to "Custom (Environment Setup below)" and the "Environment Setup for Custom Python" field to

module use my-python-modules
module load my-environment

so the job uses your custom Python version.