Unix plumbing

This exercise is about redirecting the standard streams of commands to files, and connecting commands to each other with pipes, to analyze text on your cloud server.

๐Ÿ› ๏ธ

Do this exercise on your cloud server, through SSH: its first steps look at files that only exist on a server.

๐Ÿ“œ Legend

Parts of this exercise are annotated with the following icons:

  • โ— A task you MUST perform to complete the exercise
  • โ“ Optional step that you may perform to make sure that everything is working correctly, or to set up additional tools that are not required but can help you, or optional extra exercises to explore the topic further
  • ๐Ÿ‘พ Advanced tips on how to go further (or challenges!)
  • ๐Ÿ The end of the exercise
  • ๐Ÿ›๏ธ The architecture of the software you ran or deployed during this exercise
  • ๐Ÿ’ฅ Troubleshooting tips: how to fix common problems you might encounter

โ— Where does each line go?

Connect to your cloud server and go to your home directory, where the files of this exercise will be created:

$> cd

Search the SSH configuration of your server for the word PasswordAuthentication:

$> grep -r PasswordAuthentication /etc/ssh

Some of the lines it displays are output data, and some are error messages. Which are which, and why are there errors?

๐Ÿ”‘Solution
$> grep -r PasswordAuthentication /etc/ssh
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication no
grep: /etc/ssh/sshd_config.d/50-cloud-init.conf: Permission denied
grep: /etc/ssh/ssh_host_rsa_key: Permission denied
grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication.  Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
grep: /etc/ssh/ssh_host_ed25519_key: Permission denied
/etc/ssh/ssh_config:#   PasswordAuthentication yes

The lines starting with /etc/ssh/ are the lines grep found, its output data, on its standard output stream. The lines starting with grep: are error messages, on its standard error stream: the serverโ€™s private SSH keys and one of its configuration files can only be read by root, and you are running grep as yourself.

The lines on your server may differ slightly. What matters is which stream each one is on.

๐Ÿ“š

The /etc/ssh directory holds the configuration of SSH on your server:

  • sshd_config configures the SSH server, the program that accepts your connection when you run ssh from your computer.
  • The files in sshd_config.d add to the SSH serverโ€™s configuration. Some of them were written when your server was created.
  • ssh_config configures the SSH client, used when you run ssh from this server to connect to another machine.
  • The ssh_host_..._key files are the serverโ€™s private keys, which prove its identity to the clients that connect to it: the fingerprint you checked the first time you connected identifies one of these keys. These files can only be read by root, and you are not root (unless you use sudo).

PasswordAuthentication is the setting that decides whether the SSH server accepts passwords, or only keys like yours. A line starting with # is a comment, which the server ignores: these lines document the setting and its default value. The line PasswordAuthentication no, in sshd_config.d, is the one that applies: your server refuses passwords.

Now run the same command three more times, redirecting:

  • Its standard output stream to a file named found.txt.
  • Its standard error stream to a file named errors.txt.
  • Both output streams to a single file named all.txt.

Each time, what is still displayed in your terminal, and what ends up in the file? Display the file with cat to see.

๐Ÿ”‘Solution
$> grep -r PasswordAuthentication /etc/ssh > found.txt
grep: /etc/ssh/sshd_config.d/50-cloud-init.conf: Permission denied
grep: /etc/ssh/ssh_host_rsa_key: Permission denied
grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied
grep: /etc/ssh/ssh_host_ed25519_key: Permission denied

$> cat found.txt
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication no
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication.  Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
/etc/ssh/ssh_config:#   PasswordAuthentication yes
$> grep -r PasswordAuthentication /etc/ssh 2> errors.txt
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication no
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication.  Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
/etc/ssh/ssh_config:#   PasswordAuthentication yes

$> cat errors.txt
grep: /etc/ssh/sshd_config.d/50-cloud-init.conf: Permission denied
grep: /etc/ssh/ssh_host_rsa_key: Permission denied
grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied
grep: /etc/ssh/ssh_host_ed25519_key: Permission denied
$> grep -r PasswordAuthentication /etc/ssh &> all.txt

$> cat all.txt
grep: /etc/ssh/sshd_config.d/50-cloud-init.conf: Permission denied
grep: /etc/ssh/ssh_host_rsa_key: Permission denied
grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied
grep: /etc/ssh/ssh_host_ed25519_key: Permission denied
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication no
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication.  Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
/etc/ssh/ssh_config:#   PasswordAuthentication yes

With &>, nothing is displayed, and the lines are not in the same order as in your terminal: when its standard output is a file, grep collects the lines it finds and writes them in one go, while it writes each error as soon as it happens.

โ— Did it work?

Every command ends with an exit status, which the shell keeps in the special variable $?. Display it with echo $? right after each of these commands:

$> grep PasswordAuthentication /etc/ssh/sshd_config
$> echo $?

$> grep unicorn /etc/ssh/sshd_config
$> echo $?

$> grep -r PasswordAuthentication /etc/ssh
$> echo $?

What does each exit status mean? The last command found lines: why is its exit status not 0?

๐Ÿ”‘Solution
$> grep PasswordAuthentication /etc/ssh/sshd_config
...
$> echo $?
0

$> grep unicorn /etc/ssh/sshd_config
$> echo $?
1

$> grep -r PasswordAuthentication /etc/ssh
...
$> echo $?
2

grep exits with 0 when it found lines, with 1 when it found none, and with 2 when something went wrong. The last command found lines, but it could not read some files: the errors decide its exit status, whatever it found.

The exit status is a separate answer from the output. A command can print a lot and still fail, or print nothing and succeed.

โ— Download the song

In your home directory, download the song that the next pipelines work on:

$> curl -L https://tinyurl.com/archidep-otr > rainbow.txt
๐Ÿ’ŽTip

This command is already a redirection: curl prints what it downloads to its standard output stream, and > sends it into the rainbow.txt file instead of your terminal.

Display the file:

$> cat rainbow.txt
Somewhere over the rainbow
...

โ— Pipe the song

Use command pipelines to answer questions about the song. For example, how many lines are there in the text?

$> cat rainbow.txt | wc -l
51

Now itโ€™s your turn:

  • Count the number of words in the text
๐Ÿ”‘Solution
$> cat rainbow.txt | wc -w
255
  • Print the lines of the text containing the word rainbow
๐Ÿ”‘Solution
$> cat rainbow.txt | grep rainbow
Somewhere over the rainbow
Somewhere over the rainbow
Somewhere over the rainbow
The colors of the rainbow so pretty in the sky
Oh, somewhere over the rainbow

Note that this looks for occurrences of the word โ€œrainbowโ€ exactly like this, in lowercase. If you wanted to make a case-insensitive search, you would use the grep commandโ€™s -i or --ignore-case option.

  • Do the same but without any duplicates
๐Ÿ”‘Solution
$> cat rainbow.txt | grep rainbow | sort | uniq
Oh, somewhere over the rainbow
Somewhere over the rainbow
The colors of the rainbow so pretty in the sky

uniq only filters out repeated lines that are next to each other, which is why the lines are sorted first.

  • Print the second word of each line in the text
๐Ÿ”‘Solution
$> cat rainbow.txt | cut -d ' ' -f 2
over
up
the
in

over
fly
...
  • Count the number of times the letter e is used (case-insensitive)
๐Ÿ”‘Solution
$> cat rainbow.txt | fold -w 1 | grep -i e | wc -l
131

Here are a few commands you might find useful for these pipelines. They all read the data from their standard input stream (or from a file you name), and print the result on their standard output stream, so they can be piped into each other:

Command Description
cut -d ' ' -f <n> Select word in column <n> of each line (using one space as the delimiter)
fold -w 1 Print one character by line
grep [-i] <letterOrWord> Select only lines that contain a given letter or word, e.g. grep foo (-i to ignore case)
head -n <n> Keep only the first <n> lines
sort [-nr] Sort lines alphabetically (-n to sort numerically, -r to reverse the order)
tr '[:upper:]' '[:lower:]' Convert all uppercase characters to lowercase
tr -s '[[:punct:][:space:]]' '\n' Split by word
uniq [-c] Filter out repeated lines (-c also counts them)
wc [-l] [-w] Count lines or words
๐Ÿ’ŽTip

If you want to know more about any of these commands or their options, type man <command>, e.g. man cut.

๐Ÿ“šMore information

The command line cheatsheet lists pipelines you will often use on a server, by the question they answer.

๐Ÿ‘พ Challenge

What are the five most used words in the song (case-insensitive), and how many times is each one used?

๐Ÿ”‘Solution
$> cat rainbow.txt | \
     tr '[:upper:]' '[:lower:]' | \
     tr -s '[[:punct:][:space:]]' '\n' | \
     sort | \
     uniq -c | \
     sort -nr | \
     head -n 5
     18 the
     17 i
     13 and
      9 you
      8 of

The second sort uses the -n option to sort the counts as numbers rather than as text, and -r to put the largest first.

โ— Pipe your serverโ€™s users

The file /etc/passwd lists the users of your server, one per line, with fields separated by colons (:). The seventh and last field of each line is the userโ€™s login shell, the program started when that user logs in.

Use a pipeline to count how many users have each login shell.

๐Ÿ’ŽTip

Remember the cut command you used to print the second word of each line of the song: its -d option gives the character that separates the fields (the delimiter), and its -f option the number of the field to keep.

๐Ÿ”‘Solution
$> cat /etc/passwd | cut -d : -f 7 | sort | uniq -c
      3 /bin/bash
      1 /bin/false
      1 /bin/sync
     28 /usr/sbin/nologin

The numbers on your server will differ.

Most users have /usr/sbin/nologin as their login shell. Why?

๐Ÿ”‘Solution

They are the system users you met in Meet your server: users that run services rather than people. /usr/sbin/nologin or /bin/false refuses to log them in, so nobody can open a shell as one of them.

โ“ Who is knocking at your door?

Your server is on the internet, and its SSH port is open to anyone. Programs scan the internet for such servers all day long, and try to log in to them with common usernames and passwords.

The SSH server writes each attempt in its log. The journalctl command displays what a service has logged; you will use it again later in the course. Reading the logs of the system requires administrative privileges, so run it with sudo:

$> sudo journalctl -u ssh

The log opens page by page, like a file in less: type q to quit. Many of its lines look like this one, an attempt to log in with a username that does not exist on your server:

Nov 04 10:12:33 jde sshd-session[48213]: Invalid user admin from 203.0.113.42 port 51234

Use pipelines to answer these questions about your serverโ€™s log:

  • How many attempts to log in with a username that does not exist has your server received?
๐Ÿ”‘Solution
$> sudo journalctl -u ssh | grep "Invalid user" | wc -l
347

The quotes make Invalid user a single argument, the text that grep looks for. Your number will differ.

  • Which five usernames do they try most often?
๐Ÿ’ŽTip

Count the words of the line above to find which field holds the username. They are separated by single spaces.

๐Ÿ”‘Solution
$> sudo journalctl -u ssh | \
     grep "Invalid user" | \
     cut -d ' ' -f 8 | \
     sort | \
     uniq -c | \
     sort -nr | \
     head -n 5
52 admin
31 user
18 test
12 ubuntu
 9 oracle

The username is the eighth field. The end of this pipeline is the same as in the challenge on the song: sort | uniq -c | sort -nr counts each line and puts the most frequent first. Your usernames and numbers will differ.

Attempts with usernames that do exist on your server, such as root, are logged with other messages: they are not counted here.

  • None of these attempts has succeeded. Why?
๐Ÿ”‘Solution

Your server refuses passwords: you saw the PasswordAuthentication no setting in the first step. Only a client holding a private key whose public key is in a userโ€™s authorized_keys file can log in, and these programs are guessing passwords. Most of the usernames they try do not even exist on your server.

โ“ Discard the errors

In the first step, you ran this command, which displays both the lines it finds and error messages:

$> grep -r PasswordAuthentication /etc/ssh

Find out how to run it so that it displays only the lines it found, without creating any file. The optional section of the subject on how to discard an output stream explains how.

๐Ÿ”‘Solution

Redirect its standard error stream to the null device, /dev/null, which discards everything written to it:

$> grep -r PasswordAuthentication /etc/ssh 2> /dev/null
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication no
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication.  Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
/etc/ssh/ssh_config:#   PasswordAuthentication yes

๐Ÿ What have I done?

You took commands that each do one small job, and decided where their input comes from and where their output goes: into a file, or straight into another command. None of these commands was written to work with files or with each other: the shell connected them.

Every process has three standard streams: one input and two outputs. A program writes its data to the standard output stream and its errors to the standard error stream. In your terminal, both end up on the same screen, which is why they look alike. They only show that they are separate once you send one of them somewhere else: the data went into found.txt, and the errors kept reaching you.

Redirection is done by the shell, not by the program. The shell connects the streams before the program starts, and the program does not know: grep wrote to its standard output as usual, and it ended up in a file.

The exit status is a third answer, separate from both outputs. grep found lines and still reported a failure, because it also met errors. Scripts and tools that run commands for you rely on the exit status, not on what is printed, to decide whether a command worked.

A pipe connects the standard output stream of one process to the standard input stream of the next. Each command in a pipeline only knows how to do one job, and the pipeline does what none of them can do alone, for example count the uses of a word, or the users of each shell. The error messages are not part of the pipeline: they still go to your terminal, so that you see them.

This is the Unix philosophy: small programs that do one thing well, work together, and handle text streams. It works because every program speaks the same language, a stream of text in and a stream of text out. The same principle also works with binary data when streaming audio or video.

Later in the course, the applications you deploy will also write to their standard streams, and that is where you will look for their logs.

๐Ÿ’ฅ Troubleshooting

๐Ÿ’ฅ Permission denied when redirecting to a file

You are in a directory where you are not allowed to create files, such as /etc. Go back to your home directory with cd or cd ~ and run the command again.

๐Ÿ’ฅ rainbow.txt: No such file or directory

You are not in the directory where you downloaded the file. Go back to your home directory with cd or cd ~, or download it again as in Download the song.

๐Ÿ’ฅ grep: user: No such file or directory

You wrote grep Invalid user without quotes. The shell then gives grep two arguments: Invalid, the text to look for, and user, a file to search, which does not exist. Put quotes around the text: grep "Invalid user".

๐Ÿ’ฅ The grep lines on my server are not the same as the solutionโ€™s

The SSH configuration of your server may differ slightly from the one used to write the solution. What matters is which lines are output data and which are error messages, not their exact text.