2024-01-07 How to stop Linux threads cleanly

Let’s say you’re writing a long running multi-threaded application, on Linux. Maybe it’s a database or a server of some sort. Let’s also imagine that you’re not running on some managed runtime (maybe the JVM, Go, or BEAM), but rather managing threads spawned using the clone syscall. Think of threads created in C with pthread_create, or using C++’s std::thread.¹

Once you get into the business of starting threads, you’re probably also in the business of stopping them. However the former is much easier than the latter. With “stopping” I mean stopping the thread while giving it a chance to run some cleanup operations before fully terminating. Or in other words, we want to terminate a thread while ensuring that memory is freed, locks are released, logs are flushed, and so on.²

This task is sadly not as straightforward as it should be, and there definitely isn’t a one-size-fits-all solution. This blog post aims to give an overview of the problem space and to highlight some pitfalls in an area with no shortage, and present a little magic trick at the end.

Which prints:

iterated 0 times
iterated 1 times
iterated 2 times
iterated 3 times
iterated 4 times
thread terminating
thread terminated

If you can write or refactor your code to work in such time slices, then terminating threads is very easy.

Note that the loop body does not need to be fully non blocking – it just needs to be terminated as quickly as we want our termination to be quick. For instance, if our thread is reading from a socket, we could set SO_TIMEOUT to be 100 milliseconds so that we know that every iteration of the loop will terminate quickly.³

Quasi-busy loops are all well and good, but they’re sometimes not desirable. The most common roadblock is foreign code that we don’t control which does not fit this pattern – think of a third-party library doing some blocking network call.

As we’ll see later, there’s essentially no clean way to stop a thread running code we don’t control, but there are other reasons to not want to write all our code with the quasi-busy loop pattern.

If we have many threads even relatively slow timeouts might cause significant scheduling overhead due to spurious wakeups, especially on an already busy system. The timeouts will also make debugging and inspecting the system considerably more annoying (e.g. imagine what the output of strace would look like).

So it is worth thinking about how to stop a thread while it is blocked on a syscall. The most straightforward way to do that is through signals.⁴

Signals are the main way to interrupt execution of a thread without explicit coordination of the interrupted thread, and are therefore very relevant to the topic of this blog post. They’re also a bit of a mess. These two facts generate unhappiness.

For a good overview on signals I recommend the surprisingly informative man page, but I’ll give a sufficient overview here. If you already know how signals work,⁵ you can skip to the next section.

Signals can arise because of some hardware exception⁶ or be initiated by software. The most familiar instance of a software-initiated signal is your shell sending SIGINT to the foreground process when you press ctrl-c. All signals initiated by software originate from a handful of syscalls – for instance pthread_kill will send a signal to a thread.⁷

Hardware initiated signals are generally handled immediately, while software initiated signals are handled when a CPU is about to re-enter user mode after the kernel has done some work.⁸ In any event, when a signal needs to handled in a given thread:

1. If the signal has been blocked by the receiving thread, it’ll wait to be handled until it is unblocked;

2. If the signal is not blocked, it might be:

   1. ignored;
   2. handled in the “default” manner;
   3. handled using some custom signal handler.

If you’re a C++/Rust programmer, you might have sneered at the explicit locking above – you’ve got RAII to handle such cases:

{
  const std::lock_guard<std::mutex> lock(mutex);
  // critical work here
  // The destructor for `lock` will do the unlocking
}

You might have also been wondering what happens if a thread cancellation arrives in the RAII-managed critical section here.

The answer is that thread cancellation will trigger a stack unwinding very much like throwing an exception would (in fact it’s implemented with a special exception), which means that destructors will be run on cancellation. This mechanism is known as forced unwinding. Great, right?⁹

Well, since thread cancellation is implemented using exceptions, and thread cancellation can happen in arbitrary places, we’re always liable to a cancellation happening in a noexcept block, which will cause your program to crash via std::terminate.

So since C++11, and especially since C++14 where destructors are marked as noexcept by default, thread cancellation is essentially useless in C++.¹⁰

However note that even if this mechanism worked in C++, it’d still not be safe in many situations. Consider situations like:

{
  const std::lock_guard<std::mutex> lock(mutex);
  balance_1 += x;
  balance_2 -= x;
}

If we get forcefully unwound after balance_1 += x, our invariants go out of the window. This is why Java’s form of forced unwinding, Thread.stop, was deprecated.¹¹

Hopefully you’re now convinced that unrestricted thread cancellation is not a great idea in most circumstances. However we can pick the circumstances explicitly by enabling thread cancellation only at specific times. So our event loop becomes:

pthread_setcancelstate(PTHREAD_CANCEL_DISABLE);
while (true) {
  pthread_setcancelstate(PTHREAD_CANCEL_ENABLE);
  // syscall that might block indefinitely, e.g. reading
  // from a socket
  pthread_setcancelstate(PTHREAD_CANCEL_DISABLE);
  // Perform some work completing in a reasonable time
}

We turn off thread cancellation by default, but turn it back on as we do our blocking syscall.¹²

Refactoring our code to fit this pattern might seem onerous. However many applications with long lived threads already contain loops with a blocking syscall at the beginning (reading from a socket, sleeping on a timer, and so on), followed by some work that will not block indefinitely.

However once we’ve done this, it might be worth getting rid of thread cancellation entirely. Relying on the stack unwinding to free resources would not be portable to alternative libcs, and we’d need to be fairly careful if we wanted to perform some explicit cleanup actions outside destructors.

So instead we can work with signals directly. We can pick SIGUSR1 as our “stopping” signal, install a handler which sets our stopping variable, and check the variable before doing blocking syscalls.¹³

Here’s a worked out example in C++. The interesting parts of the code are setting up the signal handler:

There’s no easy way to check the flag and run the syscall atomically.¹⁴

Another approach to this problem would be to have USR1 blocked normally, and unblock it only when the syscall runs, similarly to what we did with the temporary thread cancellation. If the syscall terminates with EINTR, we know that we should quit.¹⁵

Sadly the race is still there, just between the unblocking and running the syscall:

However, there often is an easy to atomically change the sigmask and run a syscall:

• select/poll/epoll_wait have pselect/ppoll/epoll_pwait variants which take a sigmask argument;
• read/write and similar syscalls can be replaced by their non-blocking versions and a blocking ppoll;
• To sleep one can use timerfd or just ppoll with no file descriptors but with a timeout;
• The newly added io_uring_enter supports this use case out of the box.

The syscalls above already cover a very large footprint.¹⁶

Sadly, not all syscalls have variants which let us atomically change the sigmask as they execute. futex, the main syscall used to implement userspace concurrency primitives, is a notable example of a syscall which does not include such a facility.¹⁷

In the case of futex one can interrupt threads through FUTEX_WAKE, but it turns out we can setup a mechanism to safely check the boolean stop flag atomically with starting any syscall.¹⁸

To recap, the problematic code looks like this:

If we could know that no signal handler is ran between the flag check and the syscall, then we’d be safe.

Linux 4.18 introduced a syscall, rseq (“restartable sequences”),¹⁹ which lets us achieve this, although with some effort.²⁰ The rseq machinery works as follows:

• You write some code which you want to run atomically with regards to preemption or signals – the critical section.

• Before the critical section is entered, we inform the kernel that the critical section is about to run by writing to a bit of memory shared between the kernel and userspace.

• This bit of memory contains:

  1. start_ip, the instruction pointer which marks the begin of the critical section;
  2. post_commit_offset, the length of the critical section;
  3. abort_ip, the instruction pointer to jump to if the kernel needs to preempt the critical section.

• If the kernel has preempted a thread, or if a signal needs to be delivered to the thread, it checks if the thread is in a rseq critical section, and if it does sets the program counter for the thread to abort_ip.

Note that rather than disabling preemption entirely, rseq lets us specify some code (the code starting at abort_ip) to perform some cleanup if the critical section is interrupted. The correct functioning of the critical section therefore often depends on a “commit instruction” at the very end of the critical section which makes the changes in the critical section visible.

In our case the “commit instruction” is syscall – the instruction which will invoke the syscall that we’re interested in.²¹

Which leads us to the following x86-64 widget for a 6-argument syscall stub which atomically checks a stop flag and executes a syscall: