THIS SITE IS DEPRECATED
We have transitioned to another service and are no longer actively updating this site.
Refer to our new documentation at: hpcdocs.hpc.arizona.edu
Shell Basics
Shortcuts
| Shortcut | Description | Example Usage |
|---|---|---|
~ | A shortcut for your home directory. | cd ~ |
. | Your working directory. | ./executable |
.. | One directory above your working directory. | cd .. |
Commands
| Command | Purpose | Example |
|---|---|---|
ls | List the contents of a directory | ls ~ |
ls -la | List all the contents of a directory (including hidden files) | ls -la ~ |
cd | Change your working directory | cd /path/to/directory |
pwd | Show the path to your working directory | pwd |
mkdir | Create a new directory | mkdir new_dir |
rm | Delete a file. Be careful, files deleted this way can't be retrieved | rm filename |
rm -r | Delete a directory. Be careful, directories deleted this way can't be retrieved | rm -r dirname |
Hidden Files and Directories
Hidden files and directories start with a dot "." and won't show up when you do a simple ls. Some of these files are used to configure your environment when you first log in. Be careful when removing or modifying these files since doing so can cause unintended consequences.
| Dot file/directory | What it does | Warnings |
|---|---|---|
| ~/.bash_profile | This file sets your working environment when you first log into HPC. This file sources your ~/.bashrc (see below). | Removing this file will result in the loss of the module command |
| ~/.bashrc | This file sets your working environment when you first log into HPC. Modifying this script | Removing this file will result in the loss of the module command |
| ~/.local | This is a hidden directory in your home where pip-installed python packages, jupyter kernels, RStudio session information, etc. goes. | If you pip-install python packages when a virtual environment is not active, they will be installed in this directory. These will then be automatically loaded for all future python sessions (version-specific), including in Singularity images. This may cause versioning issues. We recommend always using virtual environments. |
| ~/.apptainer | A hidden directory in your home where Apptainer images and cache files are stored by default. | This directory can grow large quickly and fill up your home. You can modify your ~/.bashrc to use a different location that has a larger quota. |
Environment Variables
Overview
These are important bash variables that control how your environment is configured. For example: what executables, libraries, header files, etc. are findable by default; information about your Slurm job; MPI settings; GPU configuration; etc. To see all the environment variables that are defined for your session, try running the command:
env
$PATH
The variable PATH is a :-delimited list of directories the system searches for executables. For example:
[sarawillis@cpu3 mybin]$ echo $PATH /usr/local/bin:/usr/bin
When you try executing any command (e.g. ls, cd, cat, python3), the system will look through each path in $PATH to see if it can find your command. Using the $PATH shown above as an example, if you tried executing foo, the system would first check /usr/local/bin for a file called foo. If a file called foo is not in that location, it would then check /usr/bin. If no file is found in that location, you'll get a "command not found" error.
[sarawillis@cpu3 mybin]$ foo bash: foo: command not found
If you have an executable that you would like to add as a default command for your session, you can add its directory to your PATH. For example:
[sarawillis@cpu3 mybin]$ ls foo [sarawillis@cpu3 mybin]$ pwd /home/u21/sarawillis/mybin [sarawillis@cpu3 mybin]$ export PATH=/home/u21/sarawillis/mybin:$PATH [sarawillis@cpu3 mybin]$ echo $PATH /home/u21/sarawillis/mybin:/usr/local/bin:/usr/bin [sarawillis@cpu3 mybin]$ foo My command was found!
Linux File Permissions
Overview
HPC users may find it useful or necessary to share files and/or directories with collaborators. To do this effectively, it may become necessary to familiarize yourself with the linux file permission system in order to give collaborators access to your files.
Linux file permissions involve a few properties. Users are categorized as follows and assigned permissions based on these categories.
- Owner: this is the user who created the file
- Group: this is the PI group containing a list of users who have file permissions according to the "group" setting
- Other: this is everyone else who does not fall into the above two categories
Each group is assigned a boolean permission to the following actions:
- Read: view the contents of the file or folder
- Write: edit the file, or create items within the folder
- Execute: run the file if it is an executable, or enter the folder to make it the current working directory
Viewing File Permissions
By default, all files and folders you create in your home directory are owned by you, and the group is assigned to your HPC sponsor group. If you are a PI, then this is your personal research group. Permissions and file properties are best viewed from the command line. Use the following command to view the contents of a directory, including permissions:
ls -l
Here is the output of this command for an example directory:
ejahn@junonia:~/example $ ls -l total 16 -rw-r--r--. 1 ejahn staff 0 Oct 10 11:18 test1.txt -rwxr-xr--. 1 ejahn staff 0 Oct 10 11:18 test2* drwxr-xr-x. 2 ejahn staff 0 Oct 10 11:18 testdir1/ drwxrwx---. 2 ejahn staff 0 Oct 10 11:18 testdir2/
The first character indicates whether the item is a regular file (dash) or directory (letter "d"). The following nine characters describe the permissions of this file according to the three categories of user and three actions listed above. The first group of three characters refer to the permitted actions of the owner; the middle three characters the group; the final three characters to all other users. The order of actions is the same for each category: read, write, execute. A permitted action is indicated by a letter, and a non-permitted action is indicated by a dash. Let's go through the permissions of each of the above files.
- File test1.txt has a permissions string that contains "rw-" for the owner, "r--" for the group, and "r--" for others. This means the owner can read and write to this file, but not execute. In this case, the owner does not need execute permissions because it is a text file and not an executable program. Then, group members and other users have read access, but neither write nor execute permissions. This setup is good for files containing information that you may want to share with others, but not necessarily let them edit.
- File test2* is an executable program, as indicated by the asterisk after the file name. The owner has complete permissions, group members have read and execute permissions, and others have read permissions. This means that group members could potentially run this program if they needed to. Non-group-members can view the program, but not run it.
- Directory testdir1 is a folder in which the owner can view, create, and enter. Group members can view and enter the folder, but not create or modify any files it might contain.
- Directory testdir2 is a folder in which the owner and group members can view, create, and enter. Others cannot perform any actions on this folder, including viewing its contents. This may be a useful setup for a collaborative shared folder which contains data that only should be viewed by group members. This protects data from being viewed by other HPC users.
After this string and following the number, there are two net IDs listed. In this case they are the owner "ejahn" and the group "staff". The first net ID is always the owner of the file, and the second is the group assigned to the file. These are important to note because they help determine which users have which set of permissions.
File Permissions
Additionally, file permissions can be altered by the owner of the file using the "chmod" command. The structure of the command is:
chmod <options> <file>
The options can either be given in terms of relative (additive/subtractive) actions, or absolute.
Relative actions have the structure <who><operator><action>:
- Who: one to three characters from the set (u,g,o) to specify user (owner), group, and/or other
- Operator: "+" or "-" to indicate add or remove the following action from the specified categories
- Action: one to three characters from the set (r,w,x) to specify read, write, or execute
The only user who can modify the permissions of a given file is the owner. Even if you have full permissions for a file, if it is not owned by you, then you cannot edit the permissions.
For example, if I wanted to remove read access from file test1.txt for group members and others, I could run the command
chmod go-r test1.txt
Then, when I view the file permissions, I can see:
ejahn@junonia:~/example $ ls -l total 16 -rw-------. 1 ejahn staff 0 Oct 10 11:18 test1.txt -rwxr-xr--. 1 ejahn staff 0 Oct 10 11:18 test2* drwxr-xr-x. 2 ejahn staff 0 Oct 10 11:18 testdir1/ drwxrwx---. 2 ejahn staff 0 Oct 10 11:18 testdir2/
Note that there are only dashes for group members and other users for file test1.txt, indicating that their read permissions have been removed. To add them back, I would change the "-" to a "+" in the chmod command:
chmod go+r test1.txt
Now, we can look at the absolute options. Rather than adding or removing given permissions, this methods set them to a specific setting according to "octal" permissions. Octal refers to the eight possible combinations of r,w,x, each of which is indicated by a number from 0-7 as follows:
- 0 = no permissions whatsoever; this person cannot read, write, or execute the file.
- 1 = execute only.
- 2 = write only.
- 3 = write and execute (1+2)
- 4 = read only.
- 5 = read and execute (4+1)
- 6 = read and write (4+2)
- 7 = read and write and execute (4+2+1)
Each octal option refers to the permissions assigned to the usual order of owner, group, other. All permissions are thereby indicated by three octal numbers. For example, directory testdir1 above would have octal permissions of 755. If I wanted to remove execute permissions from group and other for this directory, I could run the command:
chmod 744 testdir1
This sets group and other to read only. Alternatively,
chmod 700 testdir1
removes all permissions from group and other, but maintains full permissions for the owner.
There is no concrete advantage to using either absolute or relative permissions for the chmod command. Use whichever one makes the most sense to you!
WARNING: Be careful when setting permissions. Write permissions allow users to delete files & folders, with no way to recover them! Under no circumstance should you provide full file permissions to all categories of user unless you are (1) absolutely sure, or (2) comfortable with possible loss of data or other unintended consequences.
Compression and Archiving
Are you planning on transferring files to or from HPC? Do you have a lot of them? Then archiving is for you! Archiving files is the process of consolidating one or more files or directories into a single, compressed package or archive file. This simplifies data management, reduces storage space, and streamlines file transfer and backup operations. Transferring a single archived file to an external backup location my result in transfer speeds that are an order of magnitude faster than transferring the same data as an uncompressed directory with thousands (or sometimes millions) of files.
Some examples of using archival software such as gzip and tar are shown below.
dec
tar
Create an archive of a directory
(puma) [sarawillis@wentletrap archiving_example]$ tar czvf dir.tar.gz dir dir/ dir/subdir1/ dir/subdir1/file.txt dir/file1.txt dir/file3.txt dir/file2.txt (puma) [sarawillis@wentletrap archiving_example]$ ls dir dir.tar.gz