Shell 101

Opening

To open a terminal, navigate to the directory you want using the File Explorer, then right-click in the file list and select “Zsh Prompt Here”.

To exit the terminal, you have several options:

1. Use the “X” button on the window frame.

2. Type exit.

3. Press <ctrl>+d.

Autocompletion

Note

When you see <key>, that means press that key. For example, ls a<tab> means you should type ls a, then press the TAB key.

ZSH’s most powerful feature is autocompletion. First, create an empty directory using the File Explorer, then open a terminal there.

To examine the contents of the current directory, use the ls command (type in ls followed by <enter>). Of course, it’s empty, so let’s fill it up with something.

Let’s grab a copy of the Fossil source code and unzip it there just using two commands (which I recommend you just copy-paste inside the terminal):

wget https://www.fossil-scm.org/index.html/zip/trunk/fossil.zip
unzip fossil.zip

The wget command is used to get stuff from the web, and unzip does what you think it does.

Now let’s try ls again. We now see two entries, fossil (the unzipped directory) and fossil.zip.

Let’s take a peek inside the fossil directory. To do that, we can call ls with an extra parameter (or “argument”), i.e.:

ls fossil

There’s a whole bunch of files. To view one of the files, we use the less command:

less fossil/Makefile.classic

You can use the arrow keys and page-up/page-down. To exit less, press q.

Typing that whole command just to display a file is annoying. Thankfully, there’s a faster way. Try:

less fo/clas<tab>

The <tab> key tells the shell to try to autocomplete what we’ve typed so far.

Sometimes, the completion isn’t unique. Try, for example:

less fo/mak<tab>

The shell displays less fossil/Makefile.. Pressing <tab> again shows us the two possible options, Makefile.classic and Makefile.in. To remove the ambiguity and autocomplete to Makefile.classic, we can either:

1. type c followed by <tab>; or

2. simply press <tab> again, then use the arrow keys to select the option we want.

Changing directory

Paths are relative to the current directory. If we’re currently in /home/user/ and we want to refer to file /home/user/hello/world.txt, we can refer to it as hello/world.txt.

Continuing our exploration of the Fossil source tree, let’s move inside the fossil subdirectory. To do that, we change directory (or cd) into it:

cd fossil

Get in the habit of running ls to see what’s in the current directory.

Helping yourself

Next, let’s move to the src subdirectory:

cd src

Try the following command:

ls -S

To find out what the -S option means and what other options are available, enter ls --help to view the command’s built-in usage guide. This is usually just a summary, though. To view the proper documentation, try the man ls command, which will display the “manual page” of ls.

To view a detailed list (including modification times) of the files in a directory, use the -l option to ls.

Change directory autocompletion

To move to the parent directory, change directory to .., i.e.:

cd ..

Let’s move into another directory, say www:

cd www

If we now want to go back to the src directory, we could either do cd .. followed by cd src (boo!), or we can use the directory history autocompletion. Try:

cd -<tab>

(That is, cd - followed by <tab>.) You can now either enter the number of the entry you want, or press <tab> again and use the arrow keys and <enter> to select the entry you want.

File management commands

You can achieve most of these things using the File Explorer, but (once you get used to it) you can probably do them faster in the terminal.

Basic commands

To create a directory, use the mkdir {path} command. To create a directory and all parent directories if they don’t exist, use the -p option to mkdir command. mkdir -p will also not complain if all the directories already exist.

To create an empty (zero length) file, you can use the touch {filename} command.

To remove a file, use rm {filename}. To recursively remove everything below a path, use rm -r {filename}. To remove an empty directory (and throw an error if it’s not empty) use rmdir.

To copy a file or directory, use cp {source}... {destination}. The exact semantics of this command depend on whether the destination already exists and whether it is a directory:

# copies `file1` to `file2`, overwriting `file2` if already exists
cp file1 file2

# copies `file1` and `file2` to existing directory `existing_dir`
cp file1 file2 existing_dir

# fails with "cp: target 'nonexisting' is not a directory"
cp file1 file2 nonexisting

# fails with "cp: -r not specified; omitting directory 'dir1'"
# in other words, to copy recursively, you need the `-r` switch
cp dir1 nonexisting

# recursive copy `dir1` to `nonexisting_dir`
cp -r dir1 nonexisting_dir

# recursive copy `dir1` to `existing_dir/dir1`
cp -r dir1 existing_dir

# recursive copy `dir1` to `existing_dir/dir1`, and `dir2` to `existing_dir/dir2`
cp -r dir1 dir2 existing_dir

To move or rename a file, use the mv command. Note that mv and cp have similar semantics:

# rename `file1` to `file2`, overwriting `file2` if it exists (and isn't a directory)
mv file1 file2

# move `file1` to `existing_dir/file1`
mv file1 existing_dir

# move `file1` and `file2` to `existing_dir/file1` and `existing_dir/file2`
mv file1 file2 existing_dir

To preserve the file modification times and attributes, use the -a switch to the cp command.

Editing text files

I’ve added a custom command, k to your “$HOME/bin/” directory. You can use it to edit text files:

k file.txt

You can have a look at its own source code and modify it as you please (so it calls your favorite text editor instead):

k ~/bin/k

Searching

To search for a piece of text, use the grep command. For example, to search for “solve_quadratic” below the path “some/directory/”, you would run:

grep -r 'solve_quadratic' some/directory/

Globbing

Let’s say you want to copy every CSV file from the “output” directory into the current directory (“.”; the current directory is “.”). Even with autocompletion, constructing the command cp output/file1.csv output/file2.csv output/file3.csv ... ./ would be quite tedious. There is a better way, which is to use the shell’s built-in pattern matching mechanism, called globbing:

cp output/*.csv .

The wildcard “*” matches any filename that does not start with “.”. There is also the double-star wildcard “**” which is like “*” except it also recurses down subdirectories. In other words,

Path	Matches “output/*.csv”?	Matches “output/**/*.csv”?	Notes
output/a.csv	yes	yes
a.csv	no	no	doesn’t start with “output/”
output/.b.csv	no	no	starts with a “.”
output/subdir/x.csv	no	yes	“**” recurses down subdirectories; “*” doesn’t

Rsync

rsync is the swiss army knife of file copying and synchronization. It has many advantages over cp and cp -r:

• It only copies the files that have actually changed.

• It can work remotely over ssh.

• When working remotely, it achieves bandwidth efficiency by only copying the parts of files that have actually changed (link).

• You can include or exclude particular paths.

Note that rsync has different semantics from cp, so be careful. In particular, note that when it is called with a single source argument, whether that source has a trailing “/” matters! See:

# creates "dir/" if it doesn't exist, then copies `file1` and `file2` into it
rsync -a file1 file2 dir/

# copies dir1/X to dir2/X
rsync -a dir1/ dir2/

# copies dir1/X to dir2/dir1/X
rsync -a dir1 dir2/

# copies everything under dir1/ to dir2/, and removes everything in dir2/ that wasn't under dir1/
# under this usage, it's basically a directory sync tool
rsync -a --delete dir1/ dir2/

# copy to remote ssh host "workstation"
rsync -a dir1/ workstation:dir2/

I personally like to edit my code in a local fossil checkout, then rsync over the code to the group server, and run it there. The command I use is something like:

rsync /path/to/devel/project/ --exclude '/out/**' --exclude '*.pyc' --exclude '__pycache__' --include project/'**' --include doc/'**' --include pint/'**' --include 'data/**' --include '*/' --exclude '*' -av server:j/

Rsync remote paths

Remote directory paths are relative to the remote home directory (typically /home/<username/). For example, if you’d like to copy local “dir1” to remote directory “$HOME/dir2” (where “$HOME” is the remote home directory):

rsync -a dir1/ foo:dir2/

You can also specify an absolute path (one that starts with “/”). For example, if you want to copy local “dir1” to remote “/tmp/dir2” (which is not inside the home directory), you can do:

rsync -a dir1/ foo:/tmp/dir2/

Warning

If you have spaces or special characters inside the remote path, you may need to add a lot of backslashes. ZSH autocompletion works for rsync remote paths, which avoids you having to type something like:

rsync -a foo:hello\\\ world.txt "hw.txt"

Secure Shell (SSH)

SSH is a secure way to interact with other UNIX/Linux computers (such as the Compute Canada clusters, or your own group’s server). Here’s how to make it super comfortable and secure using public-key authentication.

Generating a keypair

You only need to do this once. After you’ve generated a keypair you can use it on every server.

To generate a keypair (made up of a private and public part), run the command:

ssh-keygen -t rsa -b 4096 -N ''

By default, this will save your private key in ~/.ssh/id_rsa, and your public key in ~/.ssh/id_rsa.pub.

Now back up these two files somewhere safe where other people can’t get to them. Don’t just drop them as-is into Dropbox, at the very least zip them up and set a secure password on the zip file (and write that down somewhere). If an attacker gets their hands on your id_rsa file, they can login as you to any remote machine!

Setting up a new host

Basic config

Let’s say you were given a Compute Canada login on a server “foo5.compute.ca” with username “yak” and password “bop”. Edit the file ~/.ssh/config and add the following to it:

Host foo
  HostName foo5.compute.ca
  HostKeyAlias foo5.compute.ca
  User yak

(If you more info on this config file, use man ssh_config.)

Now you can ssh into it (annoyingly, it will ask for your password “bop” every time):

ssh foo

Setting up passwordless authentication

Now that you’re on the remote machine, you need to create the .ssh directory (if it doesn’t exist), then add the contents of your public key file ~/.ssh/id_rsa.pub (on your local machine) to ~/.ssh/authorized_keys (on the remote machine). If you’re lazy like I am, you can actually achieve all of that in a single command on your local machine:

ssh-copy-id foo

From now on, when you ssh foo it won’t ask for your password anymore, and you can use rsync -a local/dir/ foo:remote/dir/ to copy files effortlessly!

SSHFS

If you’re on GNU/Linux or OSX, you can access a remote directory as if it were local using SSH filesystem (sshfs). First, create/choose an empty directory mountdir (you can call it anything you want of course) where you would like the remote directory to appear (to be “mounted”). Then simply run:

sshfs foo:remote/directory/ mountdir

You can now navigate inside mountdir as if it were a local directory, using the shell or using a graphical file manager!

Warning

File management commands typically don’t care that mountdir is actually a remote directory. In particular, this includes the remove/rm command. If you use rm -r whatever and mountdir is below whatever/, that command will delete everything on the remote side corresponding to the mounted directory.

The remote directory paths are relative to the remote home directory (typically /home/<username/). If you’d like the whole remote home directory to appear at mountdir, run:

sshfs foo: mountdir

And if you’d like to see the whole filesystem on “foo”, run:

sshfs foo:/ mountdir

Terminating processes

• If it’s an interactive process, “q” usually quits it.

• Hit Ctrl-C. This is the standard way, and gives the process a chance to clean things up as it’s closing.

• If that doesn’t work, hit Ctrl-D (which closes the input channel).

• If that doesn’t work, hit Ctrl-Z (which pauses the process), followed by kill -9 % to forcibly kill the process.