Do these exercises on your cloud server, through SSH, except where a step says to run a command on your own computer. Some of them need two terminals connected to your server at the same time.
Watch your processes
Small exercises on your cloud server to watch processes start, run and stop, and to see how they can and cannot communicate.
Legend
Parts of this exercise are annotated with the following icons:
-
A task you MUST perform to complete the exercise
-
Optional step that you may perform to make sure that everything is working correctly, or to set up additional tools that are not required but can help you, or optional extra exercises to explore the topic further
-
Advanced tips on how to go further (or challenges!)
The end of the exercise
-
The architecture of the software you ran or deployed during this exercise
-
Troubleshooting tips: how to fix common problems you might encounter
How a computer works explained what a computer physically does when it runs a program, and what the processes of your deployment are. Each part of this page is a separate, short exercise that shows one of those ideas happening on your own server. They do not depend on each other: if one goes wrong, you can still do the next.
Most parts ask you to predict what will happen before you run a command. Make your prediction, then run the command and check your answer. Each part ends with an explanation of what it showed.
In the commands below, replace jde with your username, and W.X.Y.Z with
your server’s public IP address.
Watch a process start and die
Open two terminals, and connect both to your server.
In the first terminal, list your own processes:
$> ps -f -u jde
In the second terminal, run a sleep command, which does nothing for 60
seconds:
$> sleep 60
While it is running, list your processes again in the first terminal. Note the
PID of the sleep process.
Wait for the 60 seconds to pass, or stop sleep with Ctrl+C, then list your
processes once more. Is the sleep process still there?
Now run sleep 60 again in the second terminal, and list your processes in the
first. Does the new sleep process have the same PID as the previous one?
$> ps -f -u jde
UID PID PPID C STIME TTY TIME CMD
...
jde 12034 12033 0 14:02 pts/1 00:00:00 -bash
jde 12102 12034 0 14:05 pts/1 00:00:00 sleep 60
...
Once sleep has stopped, it is no longer in the list. The second sleep gets
a different PID: it is a new process, not the first one coming back.
Finally, look at the file of the sleep program:
$> ls -lL /usr/bin/sleep
/usr/bin/sleep is a symbolic link, a reference to another file. The -L
option makes ls describe the file it leads to, the program itself, instead of
the link.
It was there before you ran sleep, it was there while sleep was running, and
it is still there now.
That file is a program: inert instructions in the storage of your server.
Each time you ran sleep, the operating system copied that program into memory
and started executing it: a process, with its own PID and its own memory.
When the process ended, the kernel freed its memory, and nothing of it was left,
except the program’s file, ready to be run again.
Running the same program twice gives two different processes. This is the difference between the program and the process described in How a computer works.
Climb the process tree
Open two terminals, and connect both to your server.
In the second terminal, display every process as a tree:
$> ps -ef --forest
Find the ps -ef --forest command you just ran. Follow its parent process ID
(PPID) to its parent, then to its parent’s parent, and so on. Where do you end
up?
$> ps -ef --forest
UID PID PPID C STIME TTY TIME CMD
root 2 0 0 09:12 ? 00:00:00 [kthreadd]
root 3 2 0 09:12 ? 00:00:00 \_ [pool_workqueue_release]
...
root 1 0 0 09:12 ? 00:00:03 /usr/lib/systemd/systemd --system --deserialize=99
...
root 888 1 0 09:12 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 12020 888 0 14:01 ? 00:00:00 \_ sshd-session: jde [priv]
jde 12033 12020 0 14:01 ? 00:00:00 | \_ sshd-session: jde@pts/1
jde 12034 12033 0 14:01 pts/1 00:00:00 | \_ -bash
root 12390 888 0 14:08 ? 00:00:00 \_ sshd-session: jde [priv]
jde 12401 12390 0 14:08 ? 00:00:00 \_ sshd-session: jde@pts/2
jde 12402 12401 0 14:08 pts/2 00:00:00 \_ -bash
jde 12411 12402 0 14:09 pts/2 00:00:00 \_ ps -ef --forest
...
The PIDs on your server are different, but the chain is the same. In this
example, your ps, PID 12411, has the PPID 12402: it was started by your shell
(-bash). The shell’s PPID, 12401, leads to your SSH session, which is two
sshd-session processes, 12401 and its parent 12390. The PPID of 12390 is 888,
the SSH server (sshd), whose own PPID is 1: the init process, PID 1. On
Ubuntu, the init process is systemd: its line may show /sbin/init instead, the
traditional name of the init program.
The names in square brackets at the top of the tree, below PID 2 (kthreadd),
are not programs: they are parts of the kernel itself, which it runs as
processes of its own.
Now, in the first terminal, start Guess It, as you did in the SFTP deployment:
$> cd ~/guessit-ex
$> npm run dev
In the second terminal, display the tree again. Which processes did running
Guess It create? Then find PostgreSQL in the tree (look for postgres). Who
started it?
$> ps -ef --forest
...
jde 12034 12033 0 14:01 pts/1 00:00:00 | \_ -bash
jde 12305 12034 0 14:12 pts/1 00:00:00 | \_ npm run dev
jde 12316 12305 0 14:12 pts/1 00:00:00 | \_ sh -c node --watch server.js
jde 12317 12316 0 14:12 pts/1 00:00:01 | \_ node --watch server.js
jde 12328 12317 0 14:12 pts/1 00:00:00 | \_ /usr/bin/node server.js
...
postgres 951 1 0 09:12 ? 00:00:00 /usr/lib/postgresql/.../bin/postgres -D /var/lib/postgresql/...
postgres 960 951 0 09:12 ? 00:00:00 \_ postgres: ...: checkpointer
postgres 961 951 0 09:12 ? 00:00:00 \_ postgres: ...: background writer
...
npm run dev is not one process but several: npm starts a shell (sh), which
starts Node.js, which starts another Node.js process to run server.js and
start it again whenever the file changes. None of them is called “Guess It”: the
program that runs is Node.js, and Guess It’s JavaScript is data that Node.js
reads.
PostgreSQL was not started by you, but by the init process (PID 1): when you installed it, and again each time your server boots. It is several processes too.
Stop Guess It with Ctrl+C once you are done.
Every program running on your server was started by another process, which is its parent. Climbing the tree always ends at the init process, PID 1: the first process the kernel starts once it has booted. It starts the services of the server, such as the SSH server and PostgreSQL, which start others in turn.
What you start yourself hangs below your shell, which hangs below your SSH
connection. That is why Guess It stops when you press Ctrl+C or disconnect,
while PostgreSQL keeps running.
Two shells, two memories
Open two terminals, and connect both to your server.
In the first terminal, create a shell variable named answer, and display it:
$> answer=42
$> echo $answer
42
What will the same echo command display in the second terminal?
$> echo $answer
Nothing: an empty line. The second shell has no variable named answer.
List your processes with ps -f -u jde. How many bash processes do you see?
Which one holds the variable answer?
$> ps -f -u jde
UID PID PPID C STIME TTY TIME CMD
jde 12010 1 0 14:01 ? 00:00:00 /usr/lib/systemd/systemd --user
jde 12012 12010 0 14:01 ? 00:00:00 (sd-pam)
jde 12033 12020 0 14:01 ? 00:00:00 sshd-session: jde@pts/1
jde 12034 12033 0 14:01 pts/1 00:00:00 -bash
jde 12401 12390 0 14:20 ? 00:00:00 sshd-session: jde@pts/2
jde 12402 12401 0 14:20 pts/2 00:00:00 -bash
jde 12450 12402 0 14:21 pts/2 00:00:00 ps -f -u jde
Two bash processes, one per terminal, with different PIDs. Only the shell of
the first terminal holds answer, in its own memory.
To get the value from the first shell to the second, write it to a file in the first terminal:
$> echo $answer > answer.txt
Then read the file in the second terminal:
$> cat answer.txt
42
Both terminals run the same program, Bash, as two separate processes. Each one has its own memory, which the operating system gives it, and no process can read or write the memory of another. A shell variable lives in the memory of one shell, so the other shell cannot see it.
To share anything, isolated processes need a channel that the operating system
provides. A file is one: one process writes it, and another reads it. How a
computer works lists the others. You have already used two of
them: a pipe, in a shell pipeline, and a socket, when you connected two nc
processes over the network in Make TCP connections. Guess It and
PostgreSQL use a socket too.
Which computer runs this?
This part starts on your own computer, not on your server. Open a terminal without connecting to your server, and display the name of the computer:
$> hostname
Then ask your server for its name, through SSH:
$> ssh jde@W.X.Y.Z hostname
Every user has a home directory, whose path is in the variable HOME. Display
it through SSH with these two commands, which look almost the same, one with
double quotes and one with single quotes:
$> ssh jde@W.X.Y.Z "echo $HOME"
/Users/jdoe
$> ssh jde@W.X.Y.Z 'echo $HOME'
/home/jde
The first command displays the home directory on your own computer (here, on
a Mac), and the second the one on your server. Yet echo ran on your server
both times. Can you guess why? Think of the processes involved in each command,
and of the computer each one runs on.
Two shells took part in each command, on two different computers. Your shell, on
your computer, read the command first and started ssh. The ssh process then
had your server start a shell of its own, which ran echo.
Inside double quotes, your shell replaces $HOME with its own value before
it even starts ssh. The server received echo /Users/jdoe, and displayed it
faithfully. Inside single quotes, your shell leaves $HOME as it is, so the
server’s shell received echo $HOME, and replaced it with its own value.
Both shells are processes with their own memory, on their own computer, with
their own value for HOME. Whenever you run a command, ask which computer it
runs on, and when.
Describe your server
Back on your server, ask for a description of its processor with lscpu
(list CPU), which only displays information:
$> lscpu
Then display its memory with free -h and its disk space with df -h /, as you
did in Unix Basics and when you ran your virtual
server:
$> free -h
$> df -h /
These commands display a lot of information. Find the following figures in their output, and say which part of the machine of How a computer works each one is about (the processor, its cores, its caches, the memory, the storage or the firmware):
ArchitectureandCPU(s), in the output oflscpu.- The lines starting with
L1d,L2andL3, in the output oflscpu. Hypervisor vendor, in the output oflscpu.- The
MemandSwaplines offree -h. - The
Sizeofdf -h /.
$> lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
...
CPU(s): 1
...
Model name: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
...
Virtualization features:
Hypervisor vendor: Microsoft
Virtualization type: full
Caches (sum of all):
L1d: 48 KiB (1 instance)
L1i: 32 KiB (1 instance)
L2: 1.3 MiB (1 instance)
L3: 48 MiB (1 instance)
...
$> free -h
total used free shared buff/cache available
Mem: 835Mi 271Mi 192Mi 18Mi 409Mi 563Mi
Swap: 2.0Gi 24Mi 2.0Gi
$> df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/root 28G 5.1G 23G 18% /
Your figures may differ, depending on the size of the server you chose and on the machine Azure runs it on.
x86_64is the processor’s instruction set, the one of most cloud servers. It is a 64-bit processor: its registers, and its addresses, are 64 bits wide, where the machine of the presentation had 8-bit values and 4-bit addresses.CPU(s)is the number of cores the kernel can give work to: 1 or 2 on the servers of this course. On a server with 2,Thread(s) per core: 2means that they are two halves of one physical core, which executes the instructions of two processes at once by interleaving them. The kernel uses them as two cores, and so does this exercise.- If the model name says
64-Core, that is the processor of the machine Azure runs your server on, shared with many other virtual machines. - If the model name ends with a speed in GHz, it is the speed of the processor’s clock: 2.8 billion ticks per second here.
L1d,L2andL3are the processor’s caches, small and fast memories inside the processor’s chip that keep copies of recently used memory.Hypervisor vendor: Microsoftsays that your server is a virtual machine: Azure’s software, not a chip on a motherboard, provides its firmware and its hardware.Memis the memory (RAM): less than 1 GiB in total.Swapis the swap space you created on the disk, where the kernel moves parts of the memory when it runs out.Sizeis the storage, where the programs and files stay when the server is turned off: about 30 GB, more than thirty times the memory.
Every figure of a computer’s description belongs to one part of the machine: the processor, with its instruction set, word size, clock, cores and caches; the memory; the storage. Your server has all of these parts, like your laptop, only smaller.
Compare the size of the memory with the size of the disk. Every program running on your server, Guess It and PostgreSQL included, has to fit in less than 1 GiB of memory, while its files sit on the much larger disk. That is the difference between the program in the storage and the process in memory, once again.
Where does the time go?
Put time in front of a command, and the shell reports how long it took when it
is done. Try it on sleep:
$> time sleep 2
real 0m2.003s
user 0m0.001s
sys 0m0.002s
time reports three durations:
realis the time that passed on a clock, from the start of the command to its end.userandsysare the time the processor spent running the process. What separates the two comes later in this part.
sleep ran for 2 seconds, yet the processor worked for it for only a few
thousandths of a second. Why do you think user and sys are so small?
sleep waited for 2 seconds, and a process that waits does not use the
processor at all.
Now give the processor some work. The following command basically performs computations during 10 seconds, and then stops. Time it:
$> time timeout 10 sha256sum /dev/zero
The sha256sum command computes a SHA-256 fingerprint of the data it reads, the
same kind of fingerprint SSH showed you for your server’s key. /dev/zero is a
device that gives an endless supply of zero bytes, so sha256sum would compute
forever: timeout 10 stops it after 10 seconds.
How does user compare with real this time?
real 0m10.006s
user 0m9.678s
sys 0m0.304s
user is almost as long as real: the processor was busy executing this
process for the whole time.
Finally, time a command that lists every file under /usr, and throws the list
away into the null device. It may take several seconds:
$> time ls -R /usr > /dev/null
How does sys compare with user this time? And real with both of them?
real 0m14.900s
user 0m0.241s
sys 0m0.518s
This time, sys is as large as user, or larger, where it was a small fraction
of user for sha256sum. And real is much longer than both put together, at
least the first time you run the command.
real is how long you waited. user and sys are how long the processor
worked for the process, and they only count while it does:
useris the time the processor spent executing the program’s own instructions, such as the calculations ofsha256sum. Computing a fingerprint is arithmetic on the bytes it reads, repeated in a loop: like the multiplication of the presentation, it comes down to instructions that compute, and jumps.sysis the time the processor spent executing the kernel’s instructions, on behalf of the process.lscannot read a disk by itself: it asks the operating system, with system calls, to read every directory under/usr, and the kernel does the work.- The rest of
realis time when the process waited, and the processor did something else, or nothing.sleeponly waited, andlsspent most of its time waiting for the disk to answer.
This is what the process boxes of an architecture diagram hide. A process that is “running”, such as Guess It waiting for a request, mostly waits, and uses no processor until something happens.
Take turns on the processor
This part reuses the sha256sum command of Where does the time
go?. Check how many cores your server has: it is the
CPU(s) line of lscpu, as in Describe your server.
Open one more terminal than your server has cores: two terminals if it has one core, three if it has two. Connect all of them to your server.
Before you run anything, predict: if you start the command in all of these
terminals at the same time, how will their real and user durations compare
with the ones you measured for a single command?
Then run the command in every terminal, starting them as close together as you can:
$> time timeout 10 sha256sum /dev/zero
On a server with one core, running two commands:
real 0m10.005s
user 0m5.007s
sys 0m0.159s
On a server with two cores, running three commands:
real 0m10.004s
user 0m6.552s
sys 0m0.213s
real is still 10 seconds, since timeout stops each command after 10 seconds.
But user is less than it was for a single command: about half of it with
two commands on one core, and about two thirds with three commands on two
cores.
A core executes the instructions of one process at a time. With more busy processes than cores, the kernel makes them take turns: it switches the core from one process to another many times per second, so that all of them seem to run at once. This is concurrency, like two queues at a single coffee machine in the presentation. Each command ran for 10 seconds, but only got the processor for its share of them, and spent the rest waiting for its turn: it computed less in the same time.
With as many cores as busy processes, they would run in parallel, one per core, and each would get the processor for the whole 10 seconds. More cores only help when there is work for each of them.
What have I done?
You watched the ideas of How a computer works happen on your own server, with commands you already knew.
None of these ideas is specific to Linux. They are how computers work,
whatever their operating system: your own computer runs programs as isolated
processes in its memory, started one from another since it booted, in the same
way. macOS and Windows do the same things, sometimes under other names and with
other tools. Both list their processes with their PIDs, in the Activity Monitor
on macOS and the Task Manager on Windows. On macOS, the init process is called
launchd; on Windows, several programs share its job.
A program is a file in the storage. A process is that program copied into memory and being executed, with its own PID. When the process ends, the kernel frees its memory, and only the file is left. Running the same program again gives a new process, with a new PID.
Every process was started by another one. Climbing the process tree always ends at the init process, PID 1, which the kernel starts at the end of the boot. Services such as PostgreSQL hang from it. What you run yourself, such as Guess It, hangs from your shell, and stops with it.
Each process has its own memory, which no other process can read. Two shells running the same program do not share their variables. Isolated processes can only communicate through channels the operating system provides: files, pipes, sockets, or shared memory. Guess It and PostgreSQL talk through a socket.
A command does not run “somewhere”: it runs on one computer, in the memory of a
process, at one moment. With SSH, two shells on two computers were involved,
each with its own HOME, and the quotes decided which one read $HOME.
Your server is a computer like the one in the presentation, only much faster: a 64-bit processor with its cores, clock and caches, a memory of less than 1 GiB, and a disk far larger than the memory. It is a virtual machine, whose hardware Azure’s software provides.
A process only uses the processor while it has instructions to execute: its own
(user), or the kernel’s working for it through system calls (sys). A process
that waits uses no processor at all.
Troubleshooting
answer: command not found
You put spaces around the = sign. A shell variable is created with no spaces:
answer=42, not answer = 42.
My sleep process is not in the list
Check that both terminals are connected to the same server, with hostname, and
that you replaced jde with your own username in the ps command.
Error: listen EADDRINUSE: address already in use :::3000
Guess It is already running, in another terminal. Use that one, or stop it with
Ctrl+C in the terminal where it is running before you start it again.