Do this exercise on your cloud server, through SSH: its first steps look at files that only exist on a server.
Unix plumbing
This exercise is about redirecting the standard streams of commands to files, and connecting commands to each other with pipes, to analyze text on your cloud server.
Legend
Parts of this exercise are annotated with the following icons:
-
A task you MUST perform to complete the exercise
-
Optional step that you may perform to make sure that everything is working correctly, or to set up additional tools that are not required but can help you, or optional extra exercises to explore the topic further
-
Advanced tips on how to go further (or challenges!)
The end of the exercise
-
The architecture of the software you ran or deployed during this exercise
-
Troubleshooting tips: how to fix common problems you might encounter
Where does each line go?
Connect to your cloud server and go to your home directory, where the files of this exercise will be created:
$> cd
Search the SSH configuration of your server for the word
PasswordAuthentication:
$> grep -r PasswordAuthentication /etc/ssh
Some of the lines it displays are output data, and some are error messages. Which are which, and why are there errors?
$> grep -r PasswordAuthentication /etc/ssh
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication no
grep: /etc/ssh/sshd_config.d/50-cloud-init.conf: Permission denied
grep: /etc/ssh/ssh_host_rsa_key: Permission denied
grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication. Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
grep: /etc/ssh/ssh_host_ed25519_key: Permission denied
/etc/ssh/ssh_config:# PasswordAuthentication yes
The lines starting with /etc/ssh/ are the lines grep found, its output data,
on its standard output stream. The lines starting with grep: are error
messages, on its standard error stream: the serverโs private SSH keys and one of
its configuration files can only be read by root, and you are running grep
as yourself.
The lines on your server may differ slightly. What matters is which stream each one is on.
The /etc/ssh directory holds the configuration of SSH on your server:
sshd_configconfigures the SSH server, the program that accepts your connection when you runsshfrom your computer.- The files in
sshd_config.dadd to the SSH serverโs configuration. Some of them were written when your server was created. ssh_configconfigures the SSH client, used when you runsshfrom this server to connect to another machine.- The
ssh_host_..._keyfiles are the serverโs private keys, which prove its identity to the clients that connect to it: the fingerprint you checked the first time you connected identifies one of these keys. These files can only be read byroot, and you are notroot(unless you usesudo).
PasswordAuthentication is the setting that decides whether the SSH server
accepts passwords, or only keys like yours. A line starting with # is a
comment, which the server ignores: these lines document the setting and its
default value. The line PasswordAuthentication no, in sshd_config.d, is the
one that applies: your server refuses passwords.
Now run the same command three more times, redirecting:
- Its standard output stream to a file named
found.txt. - Its standard error stream to a file named
errors.txt. - Both output streams to a single file named
all.txt.
Each time, what is still displayed in your terminal, and what ends up in the
file? Display the file with cat to see.
$> grep -r PasswordAuthentication /etc/ssh > found.txt
grep: /etc/ssh/sshd_config.d/50-cloud-init.conf: Permission denied
grep: /etc/ssh/ssh_host_rsa_key: Permission denied
grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied
grep: /etc/ssh/ssh_host_ed25519_key: Permission denied
$> cat found.txt
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication no
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication. Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
/etc/ssh/ssh_config:# PasswordAuthentication yes
$> grep -r PasswordAuthentication /etc/ssh 2> errors.txt
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication no
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication. Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
/etc/ssh/ssh_config:# PasswordAuthentication yes
$> cat errors.txt
grep: /etc/ssh/sshd_config.d/50-cloud-init.conf: Permission denied
grep: /etc/ssh/ssh_host_rsa_key: Permission denied
grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied
grep: /etc/ssh/ssh_host_ed25519_key: Permission denied
$> grep -r PasswordAuthentication /etc/ssh &> all.txt
$> cat all.txt
grep: /etc/ssh/sshd_config.d/50-cloud-init.conf: Permission denied
grep: /etc/ssh/ssh_host_rsa_key: Permission denied
grep: /etc/ssh/ssh_host_ecdsa_key: Permission denied
grep: /etc/ssh/ssh_host_ed25519_key: Permission denied
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication no
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication. Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
/etc/ssh/ssh_config:# PasswordAuthentication yes
With &>, nothing is displayed, and the lines are not in the same order as in
your terminal: when its standard output is a file, grep collects the lines it
finds and writes them in one go, while it writes each error as soon as it
happens.
Did it work?
Every command ends with an exit status, which the shell keeps in the special
variable $?. Display it with echo $? right after each of these commands:
$> grep PasswordAuthentication /etc/ssh/sshd_config
$> echo $?
$> grep unicorn /etc/ssh/sshd_config
$> echo $?
$> grep -r PasswordAuthentication /etc/ssh
$> echo $?
What does each exit status mean? The last command found lines: why is its exit
status not 0?
$> grep PasswordAuthentication /etc/ssh/sshd_config
...
$> echo $?
0
$> grep unicorn /etc/ssh/sshd_config
$> echo $?
1
$> grep -r PasswordAuthentication /etc/ssh
...
$> echo $?
2
grep exits with 0 when it found lines, with 1 when it found none, and with
2 when something went wrong. The last command found lines, but it could not
read some files: the errors decide its exit status, whatever it found.
The exit status is a separate answer from the output. A command can print a lot and still fail, or print nothing and succeed.
Download the song
In your home directory, download the song that the next pipelines work on:
$> curl -L https://tinyurl.com/archidep-otr > rainbow.txt
This command is already a redirection: curl prints what it downloads to its
standard output stream, and > sends it into the rainbow.txt file instead of
your terminal.
Display the file:
$> cat rainbow.txt
Somewhere over the rainbow
...
Pipe the song
Use command pipelines to answer questions about the song. For example, how many lines are there in the text?
$> cat rainbow.txt | wc -l
51
Now itโs your turn:
- Count the number of words in the text
$> cat rainbow.txt | wc -w
255- Print the lines of the text containing the word
rainbow
$> cat rainbow.txt | grep rainbow
Somewhere over the rainbow
Somewhere over the rainbow
Somewhere over the rainbow
The colors of the rainbow so pretty in the sky
Oh, somewhere over the rainbow
Note that this looks for occurrences of the word โrainbowโ exactly like this, in
lowercase. If you wanted to make a case-insensitive search, you would use the
grep commandโs -i or --ignore-case option.
- Do the same but without any duplicates
$> cat rainbow.txt | grep rainbow | sort | uniq
Oh, somewhere over the rainbow
Somewhere over the rainbow
The colors of the rainbow so pretty in the sky
uniq only filters out repeated lines that are next to each other, which is why
the lines are sorted first.
- Print the second word of each line in the text
$> cat rainbow.txt | cut -d ' ' -f 2
over
up
the
in
over
fly
...- Count the number of times the letter
eis used (case-insensitive)
$> cat rainbow.txt | fold -w 1 | grep -i e | wc -l
131Here are a few commands you might find useful for these pipelines. They all read the data from their standard input stream (or from a file you name), and print the result on their standard output stream, so they can be piped into each other:
| Command | Description |
|---|---|
cut -d ' ' -f <n> |
Select word in column <n> of each line (using one space as the delimiter) |
fold -w 1 |
Print one character by line |
grep [-i] <letterOrWord> |
Select only lines that contain a given letter or word, e.g. grep foo (-i to ignore case) |
head -n <n> |
Keep only the first <n> lines |
sort [-nr] |
Sort lines alphabetically (-n to sort numerically, -r to reverse the order) |
tr '[:upper:]' '[:lower:]' |
Convert all uppercase characters to lowercase |
tr -s '[[:punct:][:space:]]' '\n' |
Split by word |
uniq [-c] |
Filter out repeated lines (-c also counts them) |
wc [-l] [-w] |
Count lines or words |
If you want to know more about any of these commands or their options, type
man <command>, e.g. man cut.
The command line cheatsheet lists pipelines you will often use on a server, by the question they answer.
Challenge
What are the five most used words in the song (case-insensitive), and how many times is each one used?
$> cat rainbow.txt | \
tr '[:upper:]' '[:lower:]' | \
tr -s '[[:punct:][:space:]]' '\n' | \
sort | \
uniq -c | \
sort -nr | \
head -n 5
18 the
17 i
13 and
9 you
8 of
The second sort uses the -n option to sort the counts as numbers rather than
as text, and -r to put the largest first.
Pipe your serverโs users
The file /etc/passwd lists the users of your server, one per line, with fields
separated by colons (:). The seventh and last field of each line is the userโs
login shell, the program started when that user logs in.
Use a pipeline to count how many users have each login shell.
Remember the cut command you used to print the second word of each line of the
song: its -d option gives the character that separates the fields (the
delimiter), and its -f option the number of the field to keep.
$> cat /etc/passwd | cut -d : -f 7 | sort | uniq -c
3 /bin/bash
1 /bin/false
1 /bin/sync
28 /usr/sbin/nologin
The numbers on your server will differ.
Most users have /usr/sbin/nologin as their login shell. Why?
They are the system users you met in Meet your server: users that run services rather
than people. /usr/sbin/nologin or /bin/false refuses to log them in, so
nobody can open a shell as one of them.
Who is knocking at your door?
Your server is on the internet, and its SSH port is open to anyone. Programs scan the internet for such servers all day long, and try to log in to them with common usernames and passwords.
The SSH server writes each attempt in its log. The journalctl command displays
what a service has logged; you will use it again later in the course. Reading
the logs of the system requires administrative privileges, so run it with
sudo:
$> sudo journalctl -u ssh
The log opens page by page, like a file in less: type q to quit. Many of its
lines look like this one, an attempt to log in with a username that does not
exist on your server:
Nov 04 10:12:33 jde sshd-session[48213]: Invalid user admin from 203.0.113.42 port 51234
Use pipelines to answer these questions about your serverโs log:
- How many attempts to log in with a username that does not exist has your server received?
$> sudo journalctl -u ssh | grep "Invalid user" | wc -l
347
The quotes make Invalid user a single argument, the text that grep looks
for. Your number will differ.
- Which five usernames do they try most often?
Count the words of the line above to find which field holds the username. They are separated by single spaces.
$> sudo journalctl -u ssh | \
grep "Invalid user" | \
cut -d ' ' -f 8 | \
sort | \
uniq -c | \
sort -nr | \
head -n 5
52 admin
31 user
18 test
12 ubuntu
9 oracle
The username is the eighth field. The end of this pipeline is the same as in the
challenge on the song: sort | uniq -c | sort -nr counts each line and puts the
most frequent first. Your usernames and numbers will differ.
Attempts with usernames that do exist on your server, such as root, are
logged with other messages: they are not counted here.
- None of these attempts has succeeded. Why?
Your server refuses passwords: you saw the PasswordAuthentication no setting
in the first step. Only a client holding a private key whose public key is in a
userโs authorized_keys file can log in, and these programs are guessing
passwords. Most of the usernames they try do not even exist on your server.
Discard the errors
In the first step, you ran this command, which displays both the lines it finds and error messages:
$> grep -r PasswordAuthentication /etc/ssh
Find out how to run it so that it displays only the lines it found, without creating any file. The optional section of the subject on how to discard an output stream explains how.
Redirect its standard error stream to the null device, /dev/null, which
discards everything written to it:
$> grep -r PasswordAuthentication /etc/ssh 2> /dev/null
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication no
/etc/ssh/sshd_config:#PasswordAuthentication yes
/etc/ssh/sshd_config:# PasswordAuthentication. Depending on your PAM configuration,
/etc/ssh/sshd_config:# PAM authentication, then enable this but set PasswordAuthentication
/etc/ssh/ssh_config:# PasswordAuthentication yes
What have I done?
You took commands that each do one small job, and decided where their input comes from and where their output goes: into a file, or straight into another command. None of these commands was written to work with files or with each other: the shell connected them.
Every process has three standard streams: one input and two outputs. A program
writes its data to the standard output stream and its errors to the standard
error stream. In your terminal, both end up on the same screen, which is why
they look alike. They only show that they are separate once you send one of
them somewhere else: the data went into found.txt, and the errors kept
reaching you.
Redirection is done by the shell, not by the program. The shell connects the
streams before the program starts, and the program does not know: grep wrote
to its standard output as usual, and it ended up in a file.
The exit status is a third answer, separate from both outputs. grep found
lines and still reported a failure, because it also met errors. Scripts and
tools that run commands for you rely on the exit status, not on what is
printed, to decide whether a command worked.
A pipe connects the standard output stream of one process to the standard input stream of the next. Each command in a pipeline only knows how to do one job, and the pipeline does what none of them can do alone, for example count the uses of a word, or the users of each shell. The error messages are not part of the pipeline: they still go to your terminal, so that you see them.
This is the Unix philosophy: small programs that do one thing well, work together, and handle text streams. It works because every program speaks the same language, a stream of text in and a stream of text out. The same principle also works with binary data when streaming audio or video.
Later in the course, the applications you deploy will also write to their standard streams, and that is where you will look for their logs.
Troubleshooting
Permission denied when redirecting to a file
You are in a directory where you are not allowed to create files, such as /etc.
Go back to your home directory with cd or cd ~ and run the command again.
rainbow.txt: No such file or directory
You are not in the directory where you downloaded the file. Go back to your home
directory with cd or cd ~, or download it again as in Download the song.
grep: user: No such file or directory
You wrote grep Invalid user without quotes. The shell then gives grep two
arguments: Invalid, the text to look for, and user, a file to search, which
does not exist. Put quotes around the text: grep "Invalid user".
The grep lines on my server are not the same as the solutionโs
The SSH configuration of your server may differ slightly from the one used to write the solution. What matters is which lines are output data and which are error messages, not their exact text.