Shell 101
#########

.. highlight:: sh

.. _shell101-opening:

Opening
=======

To open a terminal, navigate to the directory you want using the File
Explorer, then right-click in the file list and select "Zsh Prompt
Here".

.. image:: /_static/zsh-here.svg

To exit the terminal, you have several options:

#. Use the "X" button on the window frame.
#. Type ``exit``.
#. Press ``<ctrl>+d``.

Autocompletion
==============

.. note::

   When you see ``<key>``, that means press that key. For example,
   ``ls a<tab>`` means you should type ``ls a``, then press the TAB
   key.

ZSH's most powerful feature is autocompletion. First, create an empty
directory using the File Explorer, then :ref:`open a terminal there
<shell101-opening>`.

To examine the contents of the current directory, use the ``ls``
command (type in ``ls`` followed by ``<enter>``). Of course, it's
empty, so let's fill it up with something.

Let's grab a copy of the Fossil source code and unzip it there just
using two commands (which I recommend you just copy-paste inside the
terminal)::

    wget https://www.fossil-scm.org/index.html/zip/trunk/fossil.zip
    unzip fossil.zip

The ``wget`` command is used to **get** stuff from the **w**\ eb, and
``unzip`` does what you think it does.

Now let's try ``ls`` again. We now see two entries, ``fossil`` (the
unzipped directory) and ``fossil.zip``.

Let's take a peek inside the ``fossil`` directory. To do that, we can
call ``ls`` with an extra parameter (or "argument"), i.e.::

    ls fossil

There's a whole bunch of files. To **view** one of the files, we use
the ``less`` command::

    less fossil/Makefile.classic

You can use the arrow keys and page-up/page-down. To exit ``less``,
press ``q``.

Typing that whole command just to display a file is
annoying. Thankfully, there's a faster way. Try::

    less fo/clas<tab>

The ``<tab>`` key tells the shell to try to autocomplete what we've
typed so far.

Sometimes, the completion isn't unique. Try, for example::

    less fo/mak<tab>

The shell displays ``less fossil/Makefile.``. Pressing ``<tab>`` again
shows us the two possible options, ``Makefile.classic`` and
``Makefile.in``. To remove the ambiguity and autocomplete to
``Makefile.classic``, we can either:

#. type ``c`` followed by ``<tab>``; or
#. simply press ``<tab>`` again, then use the arrow keys to select the
   option we want.

Changing directory
------------------

Paths are relative to the current directory. If we're currently in
``/home/user/`` and we want to refer to file
``/home/user/hello/world.txt``, we can refer to it as ``hello/world.txt``.

Continuing our exploration of the Fossil source tree, let's move
inside the ``fossil`` subdirectory. To do that, we **change
directory** (or ``cd``) into it::

   cd fossil

Get in the habit of running ``ls`` to see what's in the current
directory.

Helping yourself
----------------

Next, let's move to the ``src`` subdirectory::

   cd src

Try the following command::

   ls -S

To find out what the ``-S`` option means and what other options are
available, enter ``ls --help`` to view the command's built-in usage
guide. This is usually just a summary, though. To view the proper
documentation, try the ``man ls`` command, which will display the
"**man**\ ual page" of ``ls``.

To view a detailed list (including modification times) of the files in
a directory, use the ``-l`` option to ``ls``.

Change directory autocompletion
-------------------------------

To move to the parent directory, change directory to ``..``, i.e.::

    cd ..

Let's move into another directory, say ``www``::

    cd www

If we now want to go back to the ``src`` directory, we could either do
``cd ..`` followed by ``cd src`` (boo!), or we can use the directory history autocompletion. Try::

    cd -<tab>

(That is, ``cd -`` followed by ``<tab>``.) You can now either enter
the number of the entry you want, or press ``<tab>`` again and use the
arrow keys and ``<enter>`` to select the entry you want.

File management commands
========================

You can achieve most of these things using the File Explorer, but
(once you get used to it) you can probably do them faster in the
terminal.

Basic commands
--------------

To create a directory, use the ``mkdir {path}`` command. To create a
directory and all parent directories if they don't exist, use the
``-p`` option to ``mkdir`` command. ``mkdir -p`` will also not
complain if all the directories already exist.

To create an empty (zero length) file, you can use the ``touch
{filename}`` command.

To **r**\ e\ **m**\ ove a file, use ``rm {filename}``. To recursively remove
everything below a path, use ``rm -r {filename}``. To remove an empty
directory (and throw an error if it's not empty) use ``rmdir``.

To **c**\ o\ **p**\ y a file or directory, use ``cp
{source}... {destination}``. The exact semantics of this command
depend on whether the destination already exists and whether it is a
directory::

  # copies `file1` to `file2`, overwriting `file2` if already exists
  cp file1 file2

  # copies `file1` and `file2` to existing directory `existing_dir`
  cp file1 file2 existing_dir

  # fails with "cp: target 'nonexisting' is not a directory"
  cp file1 file2 nonexisting

  # fails with "cp: -r not specified; omitting directory 'dir1'"
  # in other words, to copy recursively, you need the `-r` switch
  cp dir1 nonexisting

  # recursive copy `dir1` to `nonexisting_dir`
  cp -r dir1 nonexisting_dir

  # recursive copy `dir1` to `existing_dir/dir1`
  cp -r dir1 existing_dir

  # recursive copy `dir1` to `existing_dir/dir1`, and `dir2` to `existing_dir/dir2`
  cp -r dir1 dir2 existing_dir

To **m**\ o\ **v**\ e or rename a file, use the ``mv`` command. Note that ``mv`` and ``cp`` have similar semantics::

  # rename `file1` to `file2`, overwriting `file2` if it exists (and isn't a directory)
  mv file1 file2

  # move `file1` to `existing_dir/file1`
  mv file1 existing_dir

  # move `file1` and `file2` to `existing_dir/file1` and `existing_dir/file2`
  mv file1 file2 existing_dir

To preserve the file modification times and attributes, use the ``-a``
switch to the ``cp`` command.

Editing text files
------------------

I've added a custom command, ``k`` to your "$HOME/bin/" directory. You
can use it to edit text files::

  k file.txt

You can have a look at its own source code and modify it as you
please (so it calls your favorite text editor instead)::

  k ~/bin/k

Searching
---------

To search for a piece of text, use the ``grep`` command. For example,
to search for "solve_quadratic" below the path "some/directory/", you
would run::

  grep -r 'solve_quadratic' some/directory/

.. _shell-globbing:

Globbing
--------

Let's say you want to copy every :abbr:`CSV (Comma-Separated Values)`
file from the "output" directory into the current directory ("."; the
current directory is "."). Even with autocompletion, constructing the
command ``cp output/file1.csv output/file2.csv output/file3.csv
... ./`` would be quite tedious. There is a better way, which is to
use the shell's built-in pattern matching mechanism, called
*globbing*::

  cp output/*.csv .

The wildcard "*" matches any filename that does not start with
".". There is also the double-star wildcard "**" which is like "*"
except it also recurses down subdirectories. In other words,

+-----------------------------+------------------------------+-------------------------------+-------------------------------+
| Path                        | Matches "output/\*.csv"?     | Matches "output/\*\*/\*.csv"? | Notes                         |
+=============================+==============================+===============================+===============================+
| ``output/a.csv``            | yes                          | yes                           |                               |
+-----------------------------+------------------------------+-------------------------------+-------------------------------+
| ``a.csv``                   | no                           | no                            | doesn't start with "output/"  |
+-----------------------------+------------------------------+-------------------------------+-------------------------------+
| ``output/.b.csv``           | no                           | no                            | starts with a "."             |
+-----------------------------+------------------------------+-------------------------------+-------------------------------+
| ``output/subdir/x.csv``     | no                           | yes                           | "\*\*" recurses down          |
|                             |                              |                               | subdirectories; "\*" doesn't  |
+-----------------------------+------------------------------+-------------------------------+-------------------------------+

Rsync
-----

``rsync`` is the swiss army knife of file copying and
synchronization. It has many advantages over ``cp`` and ``cp -r``:

- It only copies the files that have actually changed.
- It can work remotely over ssh.
- When working remotely, it achieves bandwidth efficiency by only
  copying the parts of files that have actually changed (link_).
- You can include or exclude particular paths.

.. _link: https://en.wikipedia.org/wiki/Rsync#Determining_which_parts_of_a_file_have_changed

Note that ``rsync`` has different semantics from ``cp``, so be
careful. In particular, note that when it is called with a single
source argument, whether that source has a trailing "/" matters! See::

  # creates "dir/" if it doesn't exist, then copies `file1` and `file2` into it
  rsync -a file1 file2 dir/

  # copies dir1/X to dir2/X
  rsync -a dir1/ dir2/

  # copies dir1/X to dir2/dir1/X
  rsync -a dir1 dir2/

  # copies everything under dir1/ to dir2/, and removes everything in dir2/ that wasn't under dir1/
  # under this usage, it's basically a directory sync tool
  rsync -a --delete dir1/ dir2/

  # copy to remote ssh host "workstation"
  rsync -a dir1/ workstation:dir2/

I personally like to edit my code in a local fossil checkout, then
rsync over the code to the group server, and run it there. The command
I use is something like::

  rsync /path/to/devel/project/ --exclude '/out/**' --exclude '*.pyc' --exclude '__pycache__' --include project/'**' --include doc/'**' --include pint/'**' --include 'data/**' --include '*/' --exclude '*' -av server:j/

.. _shell-rsync-remote-path:

Rsync remote paths
^^^^^^^^^^^^^^^^^^

Remote directory paths are relative to the remote home directory
(typically ``/home/<username/``). For example, if you'd like to copy
local "dir1" to remote directory "$HOME/dir2" (where "$HOME" is the
remote home directory)::

  rsync -a dir1/ foo:dir2/

You can also specify an absolute path (one that starts with "/"). For
example, if you want to copy local "dir1" to remote "/tmp/dir2" (which
is not inside the home directory), you can do::

  rsync -a dir1/ foo:/tmp/dir2/

.. warning::
   If you have spaces or special characters inside the remote path,
   you may need to add a lot of backslashes. ZSH autocompletion works
   for rsync remote paths, which avoids you having to type something
   like::

     rsync -a foo:hello\\\ world.txt "hw.txt"

Secure Shell (SSH)
==================

SSH is a secure way to interact with other UNIX/Linux computers (such
as the Compute Canada clusters, or your own group's server). Here's
how to make it super comfortable and secure using public-key
authentication.

Generating a keypair
--------------------

You only need to do this once. After you've generated a keypair you
can use it on every server.

To generate a keypair (made up of a private and public part), run the
command::

  ssh-keygen -t rsa -b 4096 -N ''

By default, this will save your private key in ``~/.ssh/id_rsa``, and
your public key in ``~/.ssh/id_rsa.pub``.

Now **back up these two files** somewhere safe where other people
can't get to them. Don't just drop them as-is into Dropbox, at the
very least zip them up and set a secure password on the zip file (and
write that down somewhere). If an attacker gets their hands on your
``id_rsa`` file, they can login as you to any remote machine!

Setting up a new host
---------------------

Basic config
^^^^^^^^^^^^

Let's say you were given a Compute Canada login on a server
"foo5.compute.ca" with username "yak" and password "bop". Edit the
file ``~/.ssh/config`` and add the following to it::

  Host foo
    HostName foo5.compute.ca
    HostKeyAlias foo5.compute.ca
    User yak

(If you more info on this config file, use ``man ssh_config``.)

Now you can ``ssh`` into it (annoyingly, it will ask for your password
"bop" every time)::

  ssh foo

Setting up passwordless authentication
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Now that you're on the remote machine, you need to create the ``.ssh``
directory (if it doesn't exist), then add the contents of your public
key file ``~/.ssh/id_rsa.pub`` (on your local machine) to
``~/.ssh/authorized_keys`` (on the remote machine). If you're lazy
like I am, you can actually achieve all of that in a *single command*
on your local machine::

  ssh-copy-id foo

From now on, when you ``ssh foo`` it won't ask for your password
anymore, and you can use ``rsync -a local/dir/ foo:remote/dir/`` to
copy files effortlessly!

SSHFS
^^^^^

If you're on GNU/Linux or OSX, you can access a remote directory as if
it were local using SSH **f**\ ile\ **s**\ ystem (sshfs). First,
create/choose an empty directory ``mountdir`` (you can call it
anything you want of course) where you would like the remote directory
to appear (to be "mounted"). Then simply run::

  sshfs foo:remote/directory/ mountdir

You can now navigate inside ``mountdir`` as if it were a local
directory, using the shell or using a graphical file manager!

.. warning::
   File management commands typically don't care that ``mountdir`` is
   actually a remote directory. In particular, this includes the
   remove/``rm`` command. If you use ``rm -r whatever`` and
   ``mountdir`` is below ``whatever/``, that command *will* delete
   everything on the remote side corresponding to the mounted
   directory.

The remote directory paths are relative to the remote home directory
(typically ``/home/<username/``). If you'd like the whole remote home
directory to appear at ``mountdir``, run::

  sshfs foo: mountdir

And if you'd like to see the whole filesystem on "foo", run::

  sshfs foo:/ mountdir

.. _shell-terminating:

Terminating processes
=====================

- If it's an interactive process, "q" usually **q**\ uits it.
- Hit Ctrl-C. This is the standard way, and gives the process a chance
  to clean things up as it's closing.
- If that doesn't work, hit Ctrl-D (which closes the input channel).
- If that doesn't work, hit Ctrl-Z (which pauses the process),
  followed by ``kill -9 %`` to forcibly kill the process.