You can list the signals of your system by running kill -l. Here is a more
complete list: POSIX signals.
Process Management on Linux
Learn how processes are told to stop or reload with signals, and how to manage them on Linux with systemd.
You will need
- A Unix CLI
- An Ubuntu server to connect to
Recommended reading
What is process management?
Many processes must run for a computer to do its job.
Which processes to run and when to run them depends on the use case:
- On a desktop computer, you need processes to be launched to display the graphical user interface (GUI), for example.
- On a web-facing server, you probably need to run a database and a web server at least.
Some processes may have dependencies: i.e. they might need other processes to be already running in order to run properly themselves. For example, Guess It needs its PostgreSQL database to load and save the leaderboard.
Processes also need to be managed, i.e. it should be easy to start them, stop them, restart them, or to configure them to automatically restart if they crash.
Process managers
Process managers are programs that fulfill this role: they make sure that the correct processes are run in the correct order, and keep managing them afterwards. A process manager is a process like any other, whose job is to start other processes, watch them, and restart them when they stop.
The most important one is the first process of the machine. As you saw when the machine boots, the kernel starts a single program, the init process, with process ID 1. Everything else the machine runs is started by the init process, or by a process that it started. This makes it the natural process manager of the whole system.
Most operating systems have a process manager built in:
| Process manager | Operating system |
|---|---|
| launchd | macOS |
| Service Control Manager (SCM) | Windows |
| systemd | Many Linux distributions |
| Unix System V init system | Unix |
Lightweight process managers
Simpler process managers also exist, meant to be used to run and manage one application. Some of them are generic while others are specific to a programming language:
| Process manager | Written in | Can run |
|---|---|---|
| PHP FPM | PHP | PHP programs |
| PM2 | Node.js | Node.js programs |
| Supervisor | Python | Any executable |
They are easier to use but have fewer features. For example, none of those can handle dependencies between processes.
Signals
To stop a process, restart it or tell it to reload its configuration, a process manager has to be able to tell the process so. Processes are isolated from each other, so it cannot simply reach into the other process’s memory. It uses the operating system to send it a signal instead.
What is a signal?
A signal is an asynchronous notification sent to a process to notify it that an event has occurred. The process may be doing anything at the time: it does not wait for the signal, or ask for it.
Signals are always delivered by the kernel. A process sends one by asking the kernel to, with a system call, and the kernel also sends signals of its own, for example when the terminal a process is running in is closed.
A signal does to a process what an interrupt does to a processor. The process stops whatever it was doing, and runs the code that handles that signal:
- If the program has registered a signal handler for that specific signal, that handler runs. When it is done, the process carries on where it stopped.
- Otherwise the default handler runs. For most signals, the default handler terminates the process.
Common Unix signals
A signal is named with the SIG prefix followed by a short name. Most signals
also have a number. Here are the ones you are most likely to meet:
| Signal | Number | Default handler | Description |
|---|---|---|---|
SIGHUP |
1 | Terminate | Hangup (the terminal was closed) |
SIGINT |
2 | Terminate | Interrupt (sent when you press Ctrl-C) |
SIGQUIT |
3 | Terminate (with core dump) | Quit |
SIGKILL |
9 | Terminate | Kill (cannot be caught or ignored) |
SIGTERM |
15 | Terminate | Termination signal (default signal sent by the kill command) |
SIGUSR1 |
- | Terminate | User-defined signal 1 |
SIGUSR2 |
- | Terminate | User-defined signal 2 |
SIGWINCH |
- | Ignore | Terminal window size changed |
SIGUSR1, SIGUSR2 and SIGWINCH have a number too, but it is not the same on
every system, so programs refer to them by name.
The terminal interrupt signal (SIGINT)
When you press Ctrl-C to stop a command, you are using signals. You have done
it since the start of the course, for example to stop Guess It when it was
running in your terminal.
The program does not read Ctrl-C as a key. The terminal is managed by the
kernel, which recognizes the key combination and sends a terminal interrupt
signal, or SIGINT, to the processes running in the foreground of that
terminal. When you stopped npm run dev, both npm and the node processes it
had started received it.
Most programs handle that signal by terminating. Some interactive programs use
it for something else instead. For example, Ctrl-C does not quit the pager of
man ls: you have to press q.
The kill command
The kill command sends a signal to a process, identified by its PID.
Since the default handler for most signals is to terminate the process, it often has that effect, hence the name “kill”.
Its syntax is:
kill [-<SIGNAL>] <PID>
kill [-s <SIGNAL>] <PID>
| Command | Effect |
|---|---|
kill 10000 |
Send the default SIGTERM signal to the process with PID 10000 |
kill -s HUP 10000 |
Send the SIGHUP signal to that same process |
kill -HUP 10000 |
Equivalent to the previous command |
kill -1 10000 |
Equivalent to the previous command (1 is the POSIX number of SIGHUP) |
You can only send signals to your own processes. To send a signal to a process
of another user, such as a service run by systemd, you need sudo.
Trapping signals
This script is badly behaved on purpose: it registers a handler for every signal it can, which prints a message and does nothing else. You can run it like this:
$> curl -fsSL https://tinyurl.com/archidep-trap | sh
Hi, I'm running with PID 10000
Try and kill me!
Since it handles the SIGINT signal itself, among others, Ctrl-C does not
stop it. The script prints a message, and carries on.
From another terminal, for example a second SSH connection to the same server,
you can send it other signals with kill, using the PID it printed (10000 in
this example, which will be different on your machine):
$> kill 10000
$> kill -s HUP 10000
$> kill -s QUIT 10000
$> kill -s USR1 10000
The script handles each of them the same way, and stays alive:
Nice try, I am ignoring your SIGTERM signal
Nice try, I am ignoring your SIGHUP signal
Nice try, I am ignoring your SIGQUIT signal
Nice try, I am ignoring your SIGUSR1 signal
The KILL signal
There is one signal that cannot be handled or ignored: SIGKILL.
The kernel does not deliver it to the process. It ends the process itself, without running any more of its code, so the program cannot prevent it:
$> kill -s KILL 10000
The script finally dies, without printing anything.
Because the program gets no chance to run any code, it cannot clean up before
it ends: finish writing a file, answer the requests it is handling, or close its
connection to a database. This is why you ask a process to stop with SIGTERM
first, and only use SIGKILL on a process that does not stop.
Unix signals in the real world
Many widely-used programs react to Unix signals, for example:
A common example is the SIGHUP signal. Originally, it meant that the modem
connecting a terminal to the computer had hung up.
Terminals are no longer connected to computers by modem, but the kernel still
sends SIGHUP to the processes of a terminal that is closed. When your SSH
connection closes, the shell on the server and the programs it was running in
the foreground receive it, and terminate. This is why an application you started
in an SSH session stops when you disconnect.
Programs that do not run in a terminal never receive it that way, so some of
them repurposed it. Many background services like Nginx and PostgreSQL reload
their configuration, without shutting down, when they receive a SIGHUP.

Systemd

What is systemd?
On most Linux distributions, including Ubuntu, systemd is the init process: the first process the kernel starts, with PID 1. It is the process manager of the whole system.
It also provides many of the other fundamental building blocks of a Linux operating system, such as the logging of all the processes it runs:

Systemd is a replacement for the Unix System V init system and has been the de facto standard for most Linux distributions since 2015.
Unit files
Systemd records the instructions on how to launch and manage a process in a
configuration file referred to as a unit file. It supports
many unit file types such as service, socket, timer, etc.
A unit file usually looks like this:
[Unit]
Description=Something to run
After=something-else.service
[Install]
WantedBy=multi-user.target
...
[Unit] is one section in the configuration file. There will be other sections
depending on the type of unit file. It holds the unit’s description and its
dependencies. Here, the After option specifies that this unit must be started
after the something-else unit, if both are to be started.
[Install] is an optional section that says when the unit should be started
automatically, once it has been enabled. Here, the WantedBy option specifies
that the unit is started when the system reaches the multi-user.target, which
it does every time it boots.
Service unit file
The unit file type that interests us is service, which will
configure, launch and manage a long-running process such as a database or
application.
A service unit file will have a [Service] section. Common options are:
| Option | Description |
|---|---|
ExecStart |
Command to run to start the service |
WorkingDirectory |
Directory in which to run the command |
User |
User to run the process as (can be used to limit access) |
EnvironmentFile |
File of environment variables to set for the process |
Restart |
When to restart the process automatically after it has ended |
And many more.
With Restart=on-failure, systemd restarts the process when it fails: when it
exits with an exit status other than 0, or when it is terminated
by a signal such as SIGKILL. A process that exits with a status of 0, or that
is terminated by SIGTERM, SIGINT or SIGHUP, is considered to have stopped
as it was asked to, and is not restarted.
Location of unit files
Unit files are named after their type. For example, a service unit file will
be named something.service.
Systemd will look in several directories for unit files:
| Location | Description |
|---|---|
/usr/lib/systemd/system |
Unit files installed by the operating system or by packages (e.g. a database) |
/etc/systemd/system |
Unit files installed by the system administrator |
On Ubuntu, /lib is a link to /usr/lib, so you will also see the first
directory written as /lib/systemd/system.
The systemctl command
The systemctl or system control command can be used to
enable/disable and start/stop units once their configuration file is in place:
| Command | Description |
|---|---|
systemctl status <unit> |
Display the current status of a unit, including the PID of its main process. |
sudo systemctl enable <unit> |
Enable a unit, so that it starts automatically as its [Install] section says, e.g. on boot. |
sudo systemctl disable <unit> |
Disable a unit, so that it no longer starts automatically. |
sudo systemctl start <unit> |
Start a unit. |
sudo systemctl stop <unit> |
Stop a unit. |
sudo systemctl restart <unit> |
Restart (stop and start) a unit. |
sudo systemctl reload <unit> |
Ask a unit to reload its configuration, without stopping it, if it supports it. |
sudo systemctl reenable <unit> |
Enable a unit again, after its [Install] section has been modified. |
sudo systemctl daemon-reload |
Reload unit files after they have been modified. |
systemctl does not stop a service itself: it asks systemd, which sends the
service’s process a SIGTERM signal. If the process is still running 90
seconds later, systemd sends it a SIGKILL. Many services implement reload
by having systemd send them a SIGHUP.
The journalctl command
Services started with systemd have their standard output and standard
error streams collected and logged by journald,
one of the default services provided by systemd itself. The systemctl status
command sometimes shows you an excerpt of these logs.
The journalctl command allows you to read the full logs:
| Command | Description |
|---|---|
sudo journalctl -u <unit> |
Display a unit’s logs. |
sudo journalctl -f -u <unit> |
Display and follow a unit’s logs in real time. |
The logs are often the most useful thing to read when a service does not behave as expected. A service has no terminal to show you its errors in: if it fails to start, keeps restarting or does not answer, the error messages it printed are in its logs, along with what systemd did to it and when.
References
- Signals - Unix fundamentals 201 - Ops School Curriculum
- SIGINT And Other Termination Signals in Linux
- Systemd Unit Configuration
- Systemd Service Configuration
- Systemd Execution Environment Configuration
- Systemd Process Killing Procedure Configuration
- Red Hat Enterprise Linux - System Administrator’s Guide - Managing Services With Systemd
- Systemd journald Service