Excerpt | ||
---|---|---|
| ||
Different versions of Python are available on HPC, both as system modules as well as system software on each compute node. Python 2 is available but is no longer supported by the Python Foundation, so we recommend you use Python 3. Python version 3 requires the python3 command or pip3 to differentiate. It is very different from Python version 2, so do not assume that Python 3 will work for you or that all older modules will work with version 3. We recommend you use the virtual environment supported by Python. One of the biggest reasons is that you can preserve and manage your own compute environment. The packages provided in the system Python are subject to change which may introduce incompatibilities at runtime. We also provide Anaconda as a software module that can be used to create and manage isolated development environments. |
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
Installation & Package PolicyWe maintain a two tiered approach to Python packages
|
Panel | |||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||
Available Python Versions
Six versions of Python are available on HPC. They are only available on compute nodes and are accessible either using a batch submission or interactive session.
|
Panel | |||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||||||||||||||||
Installing Python Packages Using virtualenv
One of the best things about Python is the number of packages provided by the user community. On a personal machine, the most popular method today for managing these packages is the use of a package manager, like pip. Unfortunately, these require root access and are not a viable solution on the clusters. There is an easy solution, however. You can use virtualenv to create a personal python environment that will persist each time you log in. There is no risk of packages being updated under you for another user. To find packages you might want to start with python.org.
Virtual Environment Instructions
|
Panel | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Using and Installing Python Packages with CondaInitializing CondaUsers have access to Example for setting up a local conda environment:
Once your environment is activated, you will be able to download and use custom packages with It should be noted that the One way to more effectively control your environment is to turn off conda's auto-activation feature. This can be done by running the command:
This will prevent Anaconda from being loaded into your environment until you manually activate it using:
If you have turned off auto-activation, you can still use Anaconda in a batch script using:
Installing Packages with CondaOnce conda has been configured following the steps above, users can often install software they need to run if it comes as a conda package.
If you wish to install and run a new conda package, please follow these steps: 1. Access an interactive session. This can be done quickly by running
You will still be able to run any software installed this way on either Puma or Ocelote. Switching to El Gato is preferable for quickly accessing a compute node. 2. Create an environment for your new software. Give the environment a title related to the software you are installing so that you can keep track. This is especially helpful if you plan on having more than one additional conda environment.
You can then check your available environments with
3. Follow the software-specific installation instructions. This may be as simple as running "conda install <my_package>", or it may involve installing a handful of dependencies. If the installation instructions ask you to create a new environment, you do not have to repeat this step. You should then be able to access your software within this environment! If you are unable to load your software, check your active environment with
and the installed packages with
4. (optional) Sometimes, installing the package from conda isn't sufficient. After you have the source code installed, you may want to clone the git repo with examples and useful scripts. Naturally, this is very much dependent on the software you are using and the resources the developers provide. |
Panel | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Jupyter Notebooks on OOD
HPC provides access to Jupyter notebooks on all three clusters through our Open OnDemand interface. For more information on using this service, see our page on Open On Demand. Custom KernelsTo use locally-installed packages in your Jupyter session, you can create a virtual environment and install your own kernel. The default version of Python available in Jupyter is 3.8.2. If you would like to create a virtual environment using a standard python module, you will need to use the default version that Jupyter uses. If you want to use your own version of python, you can use an Anaconda environment. Steps for both options are provided below:
Loading Modules in JupyterIn OnDemand Jupyter sessions, accessing HPC software modules directly from within a notebook can be challenging due to system configurations. However, it's still possible to access these modules when needed. For instance, machine learning packages like TensorFlow or PyTorch often require additional software modules such as CUDA for GPU utilization. To access software modules in your Jupyter notebooks, follow the steps below: Step 1: If you haven't already done so, create a custom kernel for your Jupyter notebook environment. Step 2: You will then need to edit your kernel configuration file
Step 3: Next, you will need to modify your kernel's configuration by editing this file. Start by opening it with a text editor, for example
The part you need to change is the section under
Replace Step 4: Save the |