{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Linux Basics\n",
    "\n",
    "## Feng Li\n",
    "\n",
    "### Central University of Finance and Economics\n",
    "\n",
    "### [feng.li@cufe.edu.cn](feng.li@cufe.edu.cn)\n",
    "### Course home page: [https://feng.li/distcomp](https://feng.li/distcomp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why Linux?\n",
    "\n",
    "\n",
    "\n",
    "- Linux is a free, open-source operating system. \n",
    "\n",
    "- All most all of distributed software are offered with Linux distributions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## How do I try a Linux computer?\n",
    "\n",
    "\n",
    "- Windows 10+ user: \n",
    "    - Enable the [Windows Subsystem for Linux feature](https://docs.microsoft.com/en-us/windows/wsl/).\n",
    "    - Install a Linux Distribution (Ubuntu or Debian) with [Windows Store](https://www.microsoft.com/en-us/p/ubuntu/9nblggh4msv6).\n",
    "    \n",
    "- Mac user: Terminal commands are very similar to Linux commands.\n",
    "\n",
    "- Install a real Linux system like me."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## How much do we need to know about Linux?\n",
    "\n",
    "- Login to a Linux server \n",
    "\n",
    "- Navigation with basic Linux shell commands \n",
    "\n",
    "- File Manipulation and transformation\n",
    "\n",
    "- Use an editor within a Linux server "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Login to a Linux server \n",
    "\n",
    "- Windows user, you need an SSH client\n",
    "\n",
    "    - Windows 10+: [Windows Terminal](https://www.microsoft.com/en-US/p/windows-terminal/9n0dx20hk701) \n",
    "\n",
    "    - Other Windows systems [PuTTY](https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html) or Xshell \n",
    "    \n",
    "    \n",
    "- Mac or Linux user, use the system Terminal to login to the server\n",
    "\n",
    "      ssh -p 26506 teacher01@hz-t3.matpool.com\n",
    " \n",
    "  where `26506` is your remote server's SSH port (could omitted if `-p 22`),  `teacher01` is your remote server's username, `hz-t3.matpool.com` is your remote server's address.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Run a remote Jupyter Notebook using the SSH tunnel\n",
    "\n",
    "- Login to a remote server\n",
    "    \n",
    "       ssh -p 28596 teacher01@hz-t3.matpool.com\n",
    "    \n",
    "- Start a Jupyter notebook on the server\n",
    "\n",
    "       jupyter notebook --port 8888\n",
    "\n",
    "   Also copy the string `http://localhost:8888/?token=a373718460df0437c443eeadb7250e7793d6524f80f129cc` printed on the terminal.\n",
    "\n",
    "- Start a terminal on your **local machine** and run:\n",
    "        \n",
    "        ssh  -p 28596 -L 8888:127.0.0.1:8888 -N teacher01@hz-t3.matpool.com\n",
    "        \n",
    "    where `8888:127.0.0.1:8888` means forwarding your remote server's `8888` port to your local machine's port `8888`, `-N` to avoid login to a shell on the remote.\n",
    "        \n",
    "- Open your bowser and connect to the remote Jupyter Notebook server locally with the link\n",
    "     \n",
    "     http://localhost:8888/?token=a373718460df0437c443eeadb7250e7793d6524f80f129cc\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Navigation with basic Linux shell commands \n",
    "\n",
    "- Once you have logged into the server, a few things you could do with shell commands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/fli\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "echo $HOME"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fli\n"
     ]
    }
   ],
   "source": [
    "whoami"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/fli/nextcloud/teaching/distributed-computing/distcomp-slides/L01-Introduction-to-Distributed-Computing\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "figuresl\n",
      "google-first-computer.jpeg\n",
      "L01.1-Introduction-to-Distributed-Computing.ipynb\n",
      "L01.1-Introduction-to-Distributed-Computing.slides.html\n",
      "L01.2-Linux-Basics.ipynb\n",
      "L01.2-Linux-Basics.slides.html\n",
      "L01.3-Introduction-to-Hadoop.ipynb\n",
      "L01.3-Introduction-to-Hadoop.slides.html\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 3.6M\n",
      "drwxr-xr-x  4 fli fli 4.0K Feb 20 18:42 .\n",
      "-rw-r--r--  1 fli fli  15K Feb 20 18:42 L01.2-Linux-Basics.ipynb\n",
      "drwxr-xr-x  2 fli fli 4.0K Feb 20 17:36 .ipynb_checkpoints\n",
      "drwxr-xr-x 17 fli fli 4.0K Feb 20 17:32 ..\n",
      "-rw-r--r--  1 fli fli  28K Nov 21 12:46 L01.3-Introduction-to-Hadoop.ipynb\n",
      "-rw-r--r--  1 fli fli 586K Nov 21 11:04 L01.3-Introduction-to-Hadoop.slides.html\n",
      "-rw-r--r--  1 fli fli 815K Nov 21 11:04 L01.2-Linux-Basics.slides.html\n",
      "-rw-r--r--  1 fli fli 560K Nov 21 11:04 L01.1-Introduction-to-Distributed-Computing.slides.html\n",
      "-rw-r--r--  1 fli fli 1.7M Sep 30 22:50 google-first-computer.jpeg\n",
      "-rw-r--r--  1 fli fli 7.3K Mar  3  2020 L01.1-Introduction-to-Distributed-Computing.ipynb\n",
      "drwxr-xr-x  2 fli fli 4.0K Feb 15  2020 figures\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls -htla"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root:x:0:0:root:/root:/bin/zsh\n",
      "daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin\n",
      "bin:x:2:2:bin:/bin:/usr/sbin/nologin\n",
      "sys:x:3:3:sys:/dev:/usr/sbin/nologin\n",
      "sync:x:4:65534:sync:/bin:/bin/sync\n",
      "games:x:5:60:games:/usr/games:/usr/sbin/nologin\n",
      "man:x:6:12:man:/var/cache/man:/usr/sbin/nologin\n",
      "lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin\n",
      "mail:x:8:8:mail:/var/mail:/usr/sbin/nologin\n",
      "news:x:9:9:news:/var/spool/news:/usr/sbin/nologin\n",
      "uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin\n",
      "proxy:x:13:13:proxy:/bin:/usr/sbin/nologin\n",
      "www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin\n",
      "backup:x:34:34:backup:/var/backups:/usr/sbin/nologin\n",
      "list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin\n",
      "irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin\n",
      "gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin\n",
      "nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin\n",
      "_apt:x:100:65534::/nonexistent:/usr/sbin/nologin\n",
      "systemd-timesync:x:101:102:systemd Time Synchronization,,,:/run/systemd:/usr/sbin/nologin\n",
      "systemd-network:x:102:103:systemd Network Management,,,:/run/systemd:/usr/sbin/nologin\n",
      "systemd-resolve:x:103:104:systemd Resolver,,,:/run/systemd:/usr/sbin/nologin\n",
      "messagebus:x:105:110::/nonexistent:/usr/sbin/nologin\n",
      "tss:x:106:111:TPM2 software stack,,,:/var/lib/tpm:/bin/false\n",
      "avahi-autoipd:x:107:114:Avahi autoip daemon,,,:/var/lib/avahi-autoipd:/usr/sbin/nologin\n",
      "usbmux:x:108:46:usbmux daemon,,,:/var/lib/usbmux:/usr/sbin/nologin\n",
      "rtkit:x:109:115:RealtimeKit,,,:/proc:/usr/sbin/nologin\n",
      "sshd:x:110:65534::/run/sshd:/usr/sbin/nologin\n",
      "pulse:x:111:119:PulseAudio daemon,,,:/var/run/pulse:/usr/sbin/nologin\n",
      "avahi:x:113:121:Avahi mDNS daemon,,,:/var/run/avahi-daemon:/usr/sbin/nologin\n",
      "saned:x:114:122::/var/lib/saned:/usr/sbin/nologin\n",
      "colord:x:115:123:colord colour management daemon,,,:/var/lib/colord:/usr/sbin/nologin\n",
      "geoclue:x:116:124::/var/lib/geoclue:/usr/sbin/nologin\n",
      "hplip:x:117:7:HPLIP system user,,,:/var/run/hplip:/bin/false\n",
      "fli:x:1000:1000:Feng Li,,,:/home/fli:/bin/zsh\n",
      "systemd-coredump:x:999:999:systemd Core Dumper:/:/usr/sbin/nologin\n",
      "Debian-exim:x:119:126::/var/spool/exim4:/usr/sbin/nologin\n",
      "Debian-gdm:x:118:125:Gnome Display Manager:/var/lib/gdm3:/bin/false\n",
      "nm-openconnect:x:104:129:NetworkManager OpenConnect plugin,,,:/var/lib/NetworkManager:/usr/sbin/nologin\n",
      "uuidd:x:121:130::/run/uuidd:/usr/sbin/nologin\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cat /etc/passwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## File Manipulation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004l\r",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "touch lifeng.txt # create a file "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 fli fli 0 Feb 20 18:44 lifeng.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls -htla lifeng.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004l\r",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir hello_linux"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 3.6M\n",
      "drwxr-xr-x  5 fli fli 4.0K Feb 20 18:44 .\n",
      "drwxr-xr-x  2 fli fli 4.0K Feb 20 18:44 hello_linux\n",
      "-rw-r--r--  1 fli fli    0 Feb 20 18:44 lifeng.txt\n",
      "-rw-r--r--  1 fli fli  15K Feb 20 18:42 L01.2-Linux-Basics.ipynb\n",
      "drwxr-xr-x  2 fli fli 4.0K Feb 20 17:36 .ipynb_checkpoints\n",
      "drwxr-xr-x 17 fli fli 4.0K Feb 20 17:32 ..\n",
      "-rw-r--r--  1 fli fli  28K Nov 21 12:46 L01.3-Introduction-to-Hadoop.ipynb\n",
      "-rw-r--r--  1 fli fli 586K Nov 21 11:04 L01.3-Introduction-to-Hadoop.slides.html\n",
      "-rw-r--r--  1 fli fli 815K Nov 21 11:04 L01.2-Linux-Basics.slides.html\n",
      "-rw-r--r--  1 fli fli 560K Nov 21 11:04 L01.1-Introduction-to-Distributed-Computing.slides.html\n",
      "-rw-r--r--  1 fli fli 1.7M Sep 30 22:50 google-first-computer.jpeg\n",
      "-rw-r--r--  1 fli fli 7.3K Mar  3  2020 L01.1-Introduction-to-Distributed-Computing.ipynb\n",
      "drwxr-xr-x  2 fli fli 4.0K Feb 15  2020 figures\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls -htla ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004l\r",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cd hello_linux # change directory to hello_linux"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/fli/nextcloud/teaching/distributed-computing/distcomp-slides/L01-Introduction-to-Distributed-Computing/hello_linux\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004l\r",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "touch hello.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello.py\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004l\r",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "rm hello.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004l\r",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cd .. # swtich to parent directory "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/fli/nextcloud/teaching/distributed-computing/distcomp-slides/L01-Introduction-to-Distributed-Computing\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "figuresl\n",
      "google-first-computer.jpeg\n",
      "hello_linux\n",
      "L01.1-Introduction-to-Distributed-Computing.ipynb\n",
      "L01.1-Introduction-to-Distributed-Computing.slides.html\n",
      "L01.2-Linux-Basics.ipynb\n",
      "L01.2-Linux-Basics.slides.html\n",
      "L01.3-Introduction-to-Hadoop.ipynb\n",
      "L01.3-Introduction-to-Hadoop.slides.html\n",
      "lifeng.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bash: cd: L1-Introduction: No such file or directory\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cd L1-Introduction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/fli/nextcloud/teaching/distributed-computing/distcomp-slides/L01-Introduction-to-Distributed-Computing\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004l\r",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mv hello_linux goodbye_linux"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "figuresl\n",
      "goodbye_linux\n",
      "google-first-computer.jpeg\n",
      "L01.1-Introduction-to-Distributed-Computing.ipynb\n",
      "L01.1-Introduction-to-Distributed-Computing.slides.html\n",
      "L01.2-Linux-Basics.ipynb\n",
      "L01.2-Linux-Basics.slides.html\n",
      "L01.3-Introduction-to-Hadoop.ipynb\n",
      "L01.3-Introduction-to-Hadoop.slides.html\n",
      "lifeng.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004l\r",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "rm -rf goodbye_linux"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "figuresl\n",
      "google-first-computer.jpeg\n",
      "L01.1-Introduction-to-Distributed-Computing.ipynb\n",
      "L01.1-Introduction-to-Distributed-Computing.slides.html\n",
      "L01.2-Linux-Basics.ipynb\n",
      "L01.2-Linux-Basics.slides.html\n",
      "L01.3-Introduction-to-Hadoop.ipynb\n",
      "L01.3-Introduction-to-Hadoop.slides.html\n",
      "lifeng.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Use an editor within a Linux server\n",
    "\n",
    "- `nano`  : simple to use\n",
    "- `vim`   : take a little time to learn \n",
    "- `emacs` : steady learning curve "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Need help of a command?\n",
    "\n",
    "\n",
    "- We take `mkdir` as an example \n",
    "\n",
    "    - `man mkdir` shows the standard manual of Linux command\n",
    "\n",
    "    - `mkdir --help` prints short help for the command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Linux Pipelines\n",
    "\n",
    "\n",
    "\n",
    "- `cat` – Concatenate files\n",
    "- `sort` – Sort lines of text\n",
    "- `uniq` – Report or omit repeated lines\n",
    "- `grep` – Print lines matching a pattern\n",
    "- `wc` – Print newline, word, and byte counts for each file\n",
    "- `head` – Output the first part of a file\n",
    "- `tail` – Output the last part of a file\n",
    "- `tee` – Read from standard input and write to standard output and files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Standard Input, Output, and Error\n",
    "\n",
    "- Linux theme: \"everything is a file.\"\n",
    "\n",
    "- The output of a shell program often consists of two types:\n",
    "\n",
    "    - The program's results, that is, the data the program is designed to produce.\n",
    "    - Status and error messages that tell us how the program is getting along."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Programs such as `ls` actually send their results to \n",
    "\n",
    "    - a special file called **standard output** (often expressed as `stdout`) and \n",
    "    - their status messages to another file called **standard error** (`stderr`). \n",
    "    \n",
    "    - By default, both standard output and standard error are linked to the screen and not saved into a disk file.\n",
    "\n",
    "- Many programs take input from a facility called standard input (stdin), which is, by default, attached to the keyboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Redirecting Standard Output and Errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004l\r",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls -l /usr/bin > ls-output.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 fli fli 154955 Feb 20 18:44 ls-output.txt\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls -l ls-output.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 422588\n",
      "-rwxr-xr-x 1 root root       60224 Oct 24 03:29 [\n",
      "-rwxr-xr-x 1 root root          96 Jan 27 10:05 2to3-2.7\n",
      "-rwxr-xr-x 1 root root          39 Aug 15  2020 7z\n",
      "-rwxr-xr-x 1 root root          40 Aug 15  2020 7za\n",
      "-rwxr-xr-x 1 root root          40 Aug 15  2020 7zr\n",
      "lrwxrwxrwx 1 root root          52 Feb  5 06:04 a2ping -> ../share/texlive/texmf-dist/scripts/a2ping/a2ping.pl\n",
      "-rwxr-xr-x 1 root root          39 Dec 21 23:28 a2x\n",
      "lrwxrwxrwx 1 root root          54 Feb  5 06:04 a5toa4 -> ../share/texlive/texmf-dist/scripts/pfarrei/a5toa4.tlu\n",
      "lrwxrwxrwx 1 root root          25 Sep 22  2019 aclocal -> /etc/alternatives/aclocal\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "head ls-output.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Append redirected output to a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "ls -l /usr/bin >> ls-output.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Redirecting Standard Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004l\r",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls -l /bin/usr 2> ls-error.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Redirecting Standard Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 422588\n",
      "-rwxr-xr-x 1 root root       60224 Oct 24 03:29 [\n",
      "-rwxr-xr-x 1 root root          96 Jan 27 10:05 2to3-2.7\n",
      "-rwxr-xr-x 1 root root          39 Aug 15  2020 7z\n",
      "-rwxr-xr-x 1 root root          40 Aug 15  2020 7za\n",
      "-rwxr-xr-x 1 root root          40 Aug 15  2020 7zr\n",
      "lrwxrwxrwx 1 root root          52 Feb  5 06:04 a2ping -> ../share/texlive/texmf-dist/scripts/a2ping/a2ping.pl\n",
      "-rwxr-xr-x 1 root root          39 Dec 21 23:28 a2x\n",
      "lrwxrwxrwx 1 root root          54 Feb  5 06:04 a5toa4 -> ../share/texlive/texmf-dist/scripts/pfarrei/a5toa4.tlu\n",
      "lrwxrwxrwx 1 root root          25 Sep 22  2019 aclocal -> /etc/alternatives/aclocal\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "head ls-output.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "cat ls-output.txt ls-error.txt > ls_all.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "ls -htla ls_all.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Pipelines\n",
    "\n",
    "Read data from standard input and send to standard output is utilized by a shell feature called pipelines with the format \n",
    "\n",
    "\n",
    "    command1 | command2 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   2328   22201  154955\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls -l /usr/bin | wc "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?2004l\n",
      "[\n",
      "[\n",
      "2to3-2.7\n",
      "2to3-2.7\n",
      "7z\n",
      "7z\n",
      "7za\n",
      "7za\n",
      "7zr\n",
      "sort: write failed: 'standard output': Broken pipe\n",
      "sort: write error\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "ls /bin /usr/bin | sort | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Upload and download data \n",
    "\n",
    "- The simplest way is to use a graphical tool such as [FileZilla](https://filezilla-project.org/download.php?type=client).\n",
    "\n",
    "- When you are more comfortable with commandline tools, you could try `scp` or `rsync` "
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}