Shell Basics

Shortcuts

Commands

Hidden Files and Directories

Hidden files and directories start with a dot "." and won't show up when you do a simple ls. Some of these files are used to configure your environment when you first log in. Be careful when removing or modifying these files since doing so can cause unintended consequences.

Environment Variables

Overview

These are important bash variables that control how your environment is configured. For example: what executables, libraries, header files, etc. are findable by default; information about your Slurm job; MPI settings; GPU configuration; etc. To see all the environment variables that are defined for your session, try running the command:

env

$PATH

The variable PATH is a :-delimited list of directories the system searches for executables. For example:

[sarawillis@cpu3 mybin]$ echo $PATH
/usr/local/bin:/usr/bin

When you try executing any command (e.g. ls, cd, cat, python3), the system will look through each path in $PATH to see if it can find your command. Using the $PATH shown above as an example, if you tried executing foo, the system would first check /usr/local/bin for a file called foo. If a file called foo is not in that location, it would then check /usr/bin. If no file is found in that location, you'll get a "command not found" error.

[sarawillis@cpu3 mybin]$ foo
bash: foo: command not found

If you have an executable that you would like to add as a default command for your session, you can add its directory to your PATH. For example:

[sarawillis@cpu3 mybin]$ ls
foo
[sarawillis@cpu3 mybin]$ pwd
/home/u21/sarawillis/mybin
[sarawillis@cpu3 mybin]$ export PATH=/home/u21/sarawillis/mybin:$PATH
[sarawillis@cpu3 mybin]$ echo $PATH
/home/u21/sarawillis/mybin:/usr/local/bin:/usr/bin
[sarawillis@cpu3 mybin]$ foo
My command was found!

Linux File Permissions

Overview

HPC users may find it useful or necessary to share files and/or directories with collaborators. To do this effectively, it may become necessary to familiarize yourself with the linux file permission system in order to give collaborators access to your files. 

Linux file permissions involve a few properties. Users are categorized as follows and assigned permissions based on these categories.

• Owner: this is the user who created the file
• Group: this is the PI group containing a list of users who have file permissions according to the "group" setting
• Other: this is everyone else who does not fall into the above two categories

Each group is assigned a boolean permission to the following actions:

• Read: view the contents of the file or folder
• Write: edit the file, or create items within the folder
• Execute: run the file if it is an executable, or enter the folder to make it the current working directory

Viewing File Permissions

By default, all files and folders you create in your home directory are owned by you, and the group is assigned to your HPC sponsor group. If you are a PI, then this is your personal research group. Permissions and file properties are best viewed from the command line. Use the following command to view the contents of a directory, including permissions:

ls -l

Here is the output of this command for an example directory:

ejahn@junonia:~/example $ ls -l
total 16
-rw-r--r--. 1 ejahn staff 0 Oct 10 11:18 test1.txt
-rwxr-xr--. 1 ejahn staff 0 Oct 10 11:18 test2*
drwxr-xr-x. 2 ejahn staff 0 Oct 10 11:18 testdir1/
drwxrwx---. 2 ejahn staff 0 Oct 10 11:18 testdir2/

The first character indicates whether the item is a regular file (dash) or directory (letter "d"). The following nine characters describe the permissions of this file according to the three categories of user and three actions listed above. The first group of three characters refer to the permitted actions of the owner; the middle three characters the group; the final three characters to all other users. The order of actions is the same for each category: read, write, execute. A permitted action is indicated by a letter, and a non-permitted action is indicated by a dash. Let's go through the permissions of each of the above files.

• File test1.txt has a permissions string that contains "rw-" for the owner, "r--" for the group, and "r--" for others. This means the owner can read and write to this file, but not execute. In this case, the owner does not need execute permissions because it is a text file and not an executable program. Then, group members and other users have read access, but neither write nor execute permissions. This setup is good for files containing information that you may want to share with others, but not necessarily let them edit.
• File test2* is an executable program, as indicated by the asterisk after the file name. The owner has complete permissions, group members have read and execute permissions, and others have read permissions. This means that group members could potentially run this program if they needed to. Non-group-members can view the program, but not run it.
• Directory testdir1 is a folder in which the owner can view, create, and enter. Group members can view and enter the folder, but not create or modify any files it might contain.
• Directory testdir2 is a folder in which the owner and group members can view, create, and enter. Others cannot perform any actions on this folder, including viewing its contents. This may be a useful setup for a collaborative shared folder which contains data that only should be viewed by group members. This protects data from being viewed by other HPC users.

After this string and following the number, there are two net IDs listed. In this case they are the owner "ejahn" and the group "staff". The first net ID is always the owner of the file, and the second is the group assigned to the file. These are important to note because they help determine which users have which set of permissions. 

File Permissions

Additionally, file permissions can be altered by the owner of the file using the "chmod" command. The structure of the command is:

chmod <options> <file>

The options can either be given in terms of relative (additive/subtractive) actions, or absolute. 

Relative actions have the structure <who><operator><action>:

• Who: one to three characters from the set (u,g,o) to specify user (owner), group, and/or other
• Operator: "+" or "-" to indicate add or remove the following action from the specified categories
• Action: one to three characters from the set (r,w,x) to specify read, write, or execute

For example, if I wanted to remove read access from file test1.txt for group members and others, I could run the command

chmod go-r test1.txt

Then, when I view the file permissions, I can see: 

ejahn@junonia:~/example $ ls -l
total 16
-rw-------. 1 ejahn staff 0 Oct 10 11:18 test1.txt
-rwxr-xr--. 1 ejahn staff 0 Oct 10 11:18 test2*
drwxr-xr-x. 2 ejahn staff 0 Oct 10 11:18 testdir1/
drwxrwx---. 2 ejahn staff 0 Oct 10 11:18 testdir2/

Note that there are only dashes for group members and other users for file test1.txt, indicating that their read permissions have been removed. To add them back, I would change the "-" to a "+" in the chmod command:

chmod go+r test1.txt

Now, we can look at the absolute options. Rather than adding or removing given permissions, this methods set them to a specific setting according to "octal" permissions. Octal refers to the eight possible combinations of r,w,x, each of which is indicated by a number from 0-7 as follows:

• 0 = no permissions whatsoever; this person cannot read, write, or execute the file.
• 1 = execute only.
• 2 = write only.
• 3 = write and execute (1+2)
• 4 = read only.
• 5 = read and execute (4+1)
• 6 = read and write (4+2)
• 7 = read and write and execute (4+2+1)

Each octal option refers to the permissions assigned to the usual order of owner, group, other. All permissions are thereby indicated by three octal numbers. For example, directory testdir1 above would have octal permissions of 755. If I wanted to remove execute permissions from group and other for this directory, I could run the command:

chmod 744 testdir1

This sets group and other to read only. Alternatively, 

chmod 700 testdir1

removes all permissions from group and other, but maintains full permissions for the owner.

There is no concrete advantage to using either absolute or relative permissions for the chmod command. Use whichever one makes the most sense to you!

Compression and Archiving

Are you planning on transferring files to or from HPC? Do you have a lot of them? Then archiving is for you! Archiving files is the process of consolidating one or more files or directories into a single, compressed package or archive file. This simplifies data management, reduces storage space, and streamlines file transfer and backup operations. Transferring a single archived file to an external backup location my result in transfer speeds that are an order of magnitude faster than transferring the same data as an uncompressed directory with thousands (or sometimes millions) of files.

Some examples of using archival software such as gzip and tar are shown below.

dec

tar

Create an archive of a directory

(puma) [sarawillis@wentletrap archiving_example]$ tar czvf dir.tar.gz dir
dir/
dir/subdir1/
dir/subdir1/file.txt
dir/file1.txt
dir/file3.txt
dir/file2.txt
(puma) [sarawillis@wentletrap archiving_example]$ ls
dir  dir.tar.gz