LinuxThreads Frequently Asked Questions
(with answers)


A. The big picture
B. Getting more information
C. Issues related to the C library
D. Problems, weird behaviors, potential bugs
E. Missing functions, wrong types, etc
F. C++ issues
G. Debugging LinuxThreads programs
H. Compiling multithreaded code; errno madness
I. X-Windows and other libraries
J. Signals and threads
K. Internals of LinuxThreads


A. The big picture

A.1: What is LinuxThreads?

LinuxThreads is a Linux library for multi-threaded programming. It implements the Posix 1003.1c API (Application Programming Interface) for threads. It runs on any Linux system with kernel 2.0.0 or more recent, and a suitable C library (see section B).

A.2: What are threads?

A thread is a sequential flow of control through a program. Multi-threaded programming is, thus, a form of parallel programming where several threads of control are executing concurrently in the program. All threads execute in the same memory space, and can therefore work concurrently on shared data.

Multi-threaded programming differs from Unix-style multi-processing in that all threads share the same memory space (and a few other system resources, such as file descriptors), instead of running in their own memory space as is the case with Unix processes.

Threads are useful for two reasons. First, they allow a program to exploit multi-processor machines: the threads can run in parallel on several processors, allowing a single program to divide its work between several processors, thus running faster than a single-threaded program, which runs on only one processor at a time. Second, some programs are best expressed as several threads of control that communicate together, rather than as one big monolithic sequential program. Examples include server programs, overlapping asynchronous I/O, and graphical user interfaces.

A.3: What is POSIX 1003.1c?

It's an API for multi-threaded programming standardized by IEEE as part of the POSIX standards. Most Unix vendors have endorsed the POSIX 1003.1c standard. Implementations of the 1003.1c API are already available under Sun Solaris 2.5, Digital Unix 4.0, Silicon Graphics IRIX 6, and should soon be available from other vendors such as IBM and HP. More generally, the 1003.1c API is replacing relatively quickly the proprietary threads library that were developed previously under Unix, such as Mach cthreads, Solaris threads, and IRIX sprocs. Thus, multithreaded programs using the 1003.1c API are likely to run unchanged on a wide variety of Unix platforms.

A.4: What is the status of LinuxThreads?

In short, it's not completely finished (hence the version numbers in 0.x), but what is done is pretty mature. LinuxThreads implements almost all of Posix 1003.1c, as well as a few extensions. The only part of LinuxThreads that does not conform yet to Posix is signal handling (see section J). Apart from the signal stuff, all the Posix 1003.1c base functionality is provided and conforms to the standard (to the best of my knowledge). The signal stuff is hard to get right, at least without special kernel support, and while I'm definitely looking at ways to implement the Posix behavior for signals, this might take a long time before it's completed.

A.5: How stable is LinuxThreads?

The basic functionality (thread creation and termination, mutexes, conditions, semaphores) is very stable. Several industrial-strength programs, such as the AOL multithreaded Web server, use LinuxThreads and seem quite happy about it. There are some rough edges in the LinuxThreads / C library interface, at least with libc 5, but most of these rough edges are fixed in glibc 2, which should soon become the standard C library for Linux distributions (see section C).


B. Getting more information

B.1: What are good books and other sources of information on POSIX threads?

The FAQ for comp.programming.threads lists several books: http://www.serpentine.com/~bos/threads-faq/.

There are also some online tutorials. Follow the links from the LinuxThreads web page: http://pauillac.inria.fr/~xleroy/linuxthreads.

B.2: I'd like to be informed of future developments on LinuxThreads. Is there a mailing list for this purpose?

I post LinuxThreads-related announcements on the newsgroup comp.os.linux.announce, and also on the mailing list linux-threads@magenet.com. You can subscribe to the latter by writing majordomo@magenet.com.

B.3: What are good places for discussing LinuxThreads?

For questions about programming with POSIX threads in general, use the newsgroup comp.programming.threads. Be sure you read the FAQ for this group before you post.

For Linux-specific questions, use comp.os.linux.development.apps and comp.os.linux.development.kernel. The latter is especially appropriate for questions relative to the interface between the kernel and LinuxThreads.

Very specific LinuxThreads questions, and in particular everything that looks like a potential bug in LinuxThreads, should be mailed directly to me (Xavier.Leroy@inria.fr). Before mailing me, make sure that your question is not answered in this FAQ.

B.4: I'd like to read the POSIX 1003.1c standard. Is it available online?

Unfortunately, no. POSIX standards are copyrighted by IEEE, and IEEE does not distribute them freely. You can buy paper copies from IEEE, but the price is fairly high ($120 or so). If you disagree with this policy and you're an IEEE member, be sure to let them know.

On the other hand, you probably don't want to read the standard. It's very hard to read, written in standard-ese, and targeted to implementors who already know threads inside-out. A good book on POSIX threads provides the same information in a much more readable form. I can personally recommend Dave Butenhof's book, Programming with POSIX threads (Addison-Wesley). Butenhof was part of the POSIX committee and also designed the Digital Unix implementations of POSIX threads, and it shows.

Another good source of information is the Open Group Single Unix specification (Unix 98) which is available both on-line and as a book and CD/ROM. That specification includes pretty much all the POSIX standards, with some extensions and clarifications.


C. Issues related to the C library

C.1: Which version of the C library should I use with LinuxThreads?

Most current Linux distributions come with libc version 5, maintained by H.J.Lu. For LinuxThreads to work properly, you must use either libc 5.2.18 or libc 5.4.12 or later. Avoid 5.3.12 and 5.4.7: these have problems with the per-thread errno variable.

Unfortunately, many popular Linux distributions (e.g. RedHat 4.2) come with libc 5.3.12 preinstalled -- the one that does not work with LinuxThreads. Fortunately, you can often find pre-packaged binaries of more recent versions of libc for these distributions. In the case of RedHat 4, there is a RPM package for libc-5.4 in the "contrib" area of RedHat FTP sites.

C.2: What about glibc 2, a.k.a. libc 6?

It's the next generation libc for Linux, developed by Ulrich Drepper and other FSF collaborators. glibc 2 offers much better support for threads than libc 5. Indeed, thread support was planned from the very early stages of glibc 2, while it's a last-minute addition to libc 5. glibc 2 actually comes with a specially adapted version of LinuxThreads, which you can drop in the glibc 2 sources as an add-on package.

C.3: So, should I switch to glibc 2, or stay with a recent libc 5?

Depends how you plan to do it. Switching an already installed system from libc 5 to glibc 2 is not completely straightforward. See the Glibc2 HOWTO for more information. But (re-)installing a Linux distribution based on glibc 2 is easy. One such distribution available now is RedHat 5.0. Debian and other Linux distributors will also provide glibc 2-based distributions in the near future.

C.4: Where can I find glibc 2 and the version of LinuxThreads that goes with it?

On prep.ai.mit.edu and its many, many mirrors around the world. See http://www.gnu.org/order/ftp.html for a list of mirrors.


D. Problems, weird behaviors, potential bugs

D.1: When I compile LinuxThreads, I run into problems in file libc_r/dirent.c

You probably mean:
 
        libc_r/dirent.c:94: structure has no member named `dd_lock'
I haven't actually seen this problem, but several users reported it. My understanding is that something is wrong in the include files of your Linux installation (/usr/include/*). Make sure you're using a supported version of the C library. (See section B).

D.2: When I compile LinuxThreads, I run into problems with /usr/include/sched.h: there are several occurrences of _p that the C compiler does not understand

Yes, /usr/include/sched.h that comes with libc 5.3.12 is broken. Replace it with the sched.h file contained in the LinuxThreads distribution. But really you should not be using libc 5.3.12 with LinuxThreads! (See question C.1.)

D.3: My program does fdopen() on a file descriptor opened on a pipe. When I link it with LinuxThreads, fdopen() always returns NULL!

You're using one of the buggy versions of libc (5.3.12, 5.4.7., etc). See question C.1 above.

D.4: My program crashes the first time it calls pthread_create() !

You wouldn't be using glibc 2.0, by any chance? That's a known bug with glibc 2.0. Please upgrade to 2.0.1 or later.

D.5: When I'm running a program that creates N threads, top or ps display N+2 processes that are running my program. What do all these processes correspond to?

Due to the general "one process per thread" model, there's one process for the initial thread and N processes for the threads it created using pthread_create. That leaves one process unaccounted for. That extra process corresponds to the "thread manager" thread, a thread created internally by LinuxThreads to handle thread creation and thread termination. This extra thread is asleep most of the time.

D.6: Scheduling seems to be very unfair when there is strong contention on a mutex: instead of giving the mutex to each thread in turn, it seems that it's almost always the same thread that gets the mutex. Isn't this completely broken behavior?

What happens is the following: when a thread unlocks a mutex, all other threads that were waiting on the mutex are sent a signal which makes them runnable. However, the kernel scheduler may or may not restart them immediately. If the thread that unlocked the mutex tries to lock it again immediately afterwards, it is likely that it will succeed, because the threads haven't yet restarted. This results in an apparently very unfair behavior, when the same thread repeatedly locks and unlocks the mutex, while other threads can't lock the mutex.

This is perfectly acceptable behavior with respect to the POSIX standard: for the default scheduling policy, POSIX makes no guarantees of fairness, such as "the thread waiting for the mutex for the longest time always acquires it first". This allows implementations of mutexes to remain simple and efficient. Properly written multithreaded code avoids that kind of heavy contention on mutexes, and does not run into fairness problems. If you need scheduling guarantees, you should consider using the real-time scheduling policies SCHED_RR and SCHED_FIFO, which have precisely defined scheduling behaviors.

D.7: I have a simple test program with two threads that do nothing but printf() in tight loops, and from the printout it seems that only one thread is running, the other doesn't print anything!

If you wait long enough, you should see the second thread kick in. But still, you're right, one thread prevents the other one from running for long periods of time. The reason is explained in question D.6 above: printf() performs locking on stdout, and thus your two threads contend very heavily for the mutex associated with stdout. But if you do some real work between two calls to printf(), you'll see that scheduling becomes much smoother.

D.8: I've looked at <pthread.h> and there seems to be a gross error in the pthread_cleanup_push macro: it opens a block with { but does not close it! Surely you forgot a } at the end of the macro, right?

Nope. That's the way it should be. The closing brace is provided by the pthread_cleanup_pop macro. The POSIX standard requires pthread_cleanup_push and pthread_cleanup_pop to be used in matching pairs, at the same level of brace nesting. This allows pthread_cleanup_push to open a block in order to stack-allocate some data structure, and pthread_cleanup_pop to close that block. It's ugly, but it's the standard way of implementing cleanup handlers.


E. Missing functions, wrong types, etc

E.1: Where is pthread_yield() ? How comes LinuxThreads does not implement it?

Because it's not part of the (final) POSIX 1003.1c standard. Several drafts of the standard contained pthread_yield(), but then the POSIX guys discovered it was redundant with sched_yield() and dropped it. So, just use sched_yield() instead.

E.2: I've found some type errors in <pthread.h>. For instance, the second argument to pthread_create() should be a pthread_attr_t, not a pthread_attr_t *. Also, didn't you forget to declare pthread_attr_default?

No, I didn't. What you're describing is draft 4 of the POSIX standard, which is used in OSF DCE threads. LinuxThreads conforms to the final standard. Even though the functions have the same names as in draft 4 and DCE, their calling conventions are slightly different. In particular, attributes are passed by reference, not by value, and default attributes are denoted by the NULL pointer. Since draft 4/DCE will eventually disappear, you'd better port your program to use the standard interface.

E.3: I'm porting an application from Solaris and I have to rename all thread functions from thr_blah to pthread_blah. This is very annoying. Why did you change all the function names?

POSIX did it. The thr_* functions correspond to Solaris threads, an older thread interface that you'll find only under Solaris. The pthread_* functions correspond to POSIX threads, an international standard available for many, many platforms. Even Solaris 2.5 and later support the POSIX threads interface. So, do yourself a favor and rewrite your code to use POSIX threads: this way, it will run unchanged under Linux, Solaris, and quite a lot of other platforms.

E.4: How can I suspend and resume a thread from another thread? Solaris has the thr_suspend() and thr_resume() functions to do that; why don't you?

The POSIX standard provides no mechanism by which a thread A can suspend the execution of another thread B, without cooperation from B. The only way to implement a suspend/restart mechanism is to have B check periodically some global variable for a suspend request and then suspend itself on a condition variable, which another thread can signal later to restart B.

Notice that thr_suspend() is inherently dangerous and prone to race conditions. For one thing, there is no control on where the target thread stops: it can very well be stopped in the middle of a critical section, while holding mutexes. Also, there is no guarantee on when the target thread will actually stop. For these reasons, you'd be much better off using mutexes and conditions instead. The only situations that really require the ability to suspend a thread are debuggers and some kind of garbage collectors.

If you really must suspend a thread in LinuxThreads, you can send it a SIGSTOP signal with pthread_kill. Send SIGCONT for restarting it. Beware, this is specific to LinuxThreads and entirely non-portable. Indeed, a truly conforming POSIX threads implementation will stop all threads when one thread receives the SIGSTOP signal! One day, LinuxThreads will implement that behavior, and the non-portable hack with SIGSTOP won't work anymore.

E.5: LinuxThreads does not implement pthread_attr_setstacksize() nor pthread_attr_setstackaddr(). Why?

These two functions are part of optional components of the POSIX standard, meaning that portable applications should test for the "feature test" macros _POSIX_THREAD_ATTR_STACKSIZE and _POSIX_THREAD_ATTR_STACKADDR (respectively) before using these functions.

pthread_attr_setstacksize() lets the programmer specify the maximum stack size for a thread. In LinuxThreads, stacks start small (4k) and grow on demand to a fairly large limit (2M), which cannot be modified on a per-thread basis for architectural reasons. Hence there is really no need to specify any stack size yourself: the system does the right thing all by itself. Besides, there is no portable way to estimate the stack requirements of a thread, so setting the stack size is pretty useless anyway.

pthread_attr_setstackaddr() is even more questionable: it lets users specify the stack location for a thread. Again, LinuxThreads takes care of that for you. Why you would ever need to set the stack address escapes me.

E.6: LinuxThreads does not support the PTHREAD_SCOPE_PROCESS value of the "contentionscope" attribute. Why?

With a "one-to-one" model, as in LinuxThreads (one kernel execution context per thread), there is only one scheduler for all processes and all threads on the system. So, there is no way to obtain the behavior of PTHREAD_SCOPE_PROCESS.

E.7: LinuxThreads does not implement process-shared mutexes, conditions, and semaphores. Why?

This is another optional component of the POSIX standard. Portable applications should test _POSIX_THREAD_PROCESS_SHARED before using this facility.

The goal of this extension is to allow different processes (with different address spaces) to synchronize through mutexes, conditions or semaphores allocated in shared memory (either SVR4 shared memory segments or mmap()ed files).

The reason why this does not work in LinuxThreads is that mutexes, conditions, and semaphores are not self-contained: their waiting queues contain pointers to linked lists of thread descriptors, and these pointers are meaningful only in one address space.

Matt Messier and I spent a significant amount of time trying to design a suitable mechanism for sharing waiting queues between processes. We came up with several solutions that combined two of the following three desirable features, but none that combines all three:

We concluded that kernel support is required to share mutexes, conditions and semaphores between processes. That's one place where Linus Torvalds's intuition that "all we need in the kernel is clone()" fails.

Until suitable kernel support is available, you'd better use traditional interprocess communications to synchronize different processes: System V semaphores and message queues, or pipes, or sockets.


F. C++ issues

F.1: Are there C++ wrappers for LinuxThreads?

Douglas Schmidt's ACE library contains, among a lot of other things, C++ wrappers for LinuxThreads and quite a number of other thread libraries. Check out http://www.cs.wustl.edu/~schmidt/ACE.html

F.2: I'm trying to use LinuxThreads from a C++ program, and the compiler complains about the third argument to pthread_create() !

You're probably trying to pass a class member function or some other C++ thing as third argument to pthread_create(). Recall that pthread_create() is a C function, and it must be passed a C function as third argument.

F.3: I'm trying to use LinuxThreads in conjunction with libg++, and I'm having all sorts of trouble.

From what I understand, thread support in libg++ is completely broken, especially with respect to locking of iostreams. H.J.Lu wrote:
If you want to use thread, I can only suggest egcs and glibc. You can find egcs at http://www.cygnus.com/egcs. egcs has libsdtc++, which is MT safe under glibc 2. If you really want to use the libg++, I have a libg++ add-on for egcs.

G. Debugging LinuxThreads programs

G.1: Can I debug LinuxThreads program using gdb?

Essentially, no. gdb is basically not aware of the threads. It will let you debug the main thread, and also inspect the global state, but you won't have any control over the other threads. Worse, you can't put any breakpoint anywhere in the code: if a thread other than the main thread hits the breakpoint, it will just crash!

For running gdb on the main thread, you need to instruct gdb to ignore the signals used by LinuxThreads. Just do:

        handle SIGUSR1 nostop pass noprint
        handle SIGUSR2 nostop pass noprint

G.2: What about attaching to a running thread using the attach command of gdb?

For reasons I don't fully understand, this does not work.

G.3: But I know gdb supports threads on some platforms! Why not on Linux?

You're correct that gdb has some built-in support for threads, in particular the IRIX "sprocs" model, which is a "one thread = one process" model fairly close to LinuxThreads. But gdb under IRIX uses ioctls on /proc to control debugged processes, while under Linux it uses the traditional ptrace(). The support for threads is built in the /proc interface, but some work remains to be done to have it in the ptrace() interface. In summary, it should not be impossible to get gdb to work with LinuxThreads, but it's definitely not trivial.

G.4: OK, I'll do post-mortem debugging, then. But gdb cannot read core files generated by a multithreaded program! Or, the core file is readable from gcc, but does not correspond to the thread that crashed! What happens?

Some versions of gdb do indeed have problems with post-mortem debugging in general, but this is not specific to LinuxThreads. Recent Linux distributions seem to have corrected this problem, though.

Regarding the fact that the core file does not correspond to the thread that crashed, the reason is that the kernel will not dump core for a process that shares its memory with other processes, such as the other threads of your program. So, the thread that crashes silently disappears without generating a core file. Then, all other threads of your program die on the same signal that killed the crashing thread. (This is required behavior according to the POSIX standard.) The last one that dies is no longer sharing its memory with anyone else, so the kernel generates a core file for that thread. Unfortunately, that's not the thread you are interested in.

G.5: How can I debug multithreaded programs, then?

Assertions and printf() are your best friends. Try to debug sequential parts in a single-threaded program first. Then, put printf() statements all over the place to get execution traces. Also, check invariants often with the assert() macro. In truth, there is no other effective way (save for a full formal proof of your program) to track down concurrency bugs. Debuggers are not really effective for concurrency problems, because they disrupt program execution too much.


H. Compiling multithreaded code; errno madness

H.1: You say all multithreaded code must be compiled with _REENTRANT defined. What difference does it make?

It affects include files in three ways:

H.2: Why is it so important that each thread has its own errno variable?

If all threads were to store error codes in the same, global errno variable, then the value of errno after a system call or library function returns would be unpredictable: between the time a system call stores its error code in the global errno and your code inspects errno to see which error occurred, another thread might have stored another error code in the same errno location.

H.3: What happens if I link LinuxThreads with code not compiled with -D_REENTRANT?

Lots of trouble. If the code uses getc() or putc(), it will perform I/O without proper interlocking of the stdio buffers; this can cause lost output, duplicate output, or just crash other stdio functions. If the code consults errno, it will get back the wrong error code. The following code fragment is a typical example:
        do {
          r = read(fd, buf, n);
          if (r == -1) {
            if (errno == EINTR)   /* an error we can handle */
              continue;
            else {                /* other errors are fatal */
              perror("read failed");
              exit(100);
            }
          }
        } while (...);
Assume this code is not compiled with -D_REENTRANT, and linked with LinuxThreads. At run-time, read() is interrupted. Since the C library was compiled with -D_REENTRANT, read() stores its error code in the location pointed to by __errno_location(), which is the thread-local errno variable. Then, the code above sees that read() returns -1 and looks up errno. Since _REENTRANT is not defined, the reference to errno accesses the global errno variable, which is most likely 0. Hence the code concludes that it cannot handle the error and stops.

H.4: With LinuxThreads, I can no longer use the signals SIGUSR1 and SIGUSR2 in my programs! Why?

LinuxThreads needs two signals for its internal operation. One is used to suspend and restart threads blocked on mutex, condition or semaphore operations. The other is used for thread cancellation. Since the only two signals not reserved for the Linux kernel are SIGUSR1 and SIGUSR2, LinuxThreads has no other choice than using them. I know this is unfortunate, and hope this problem will be addressed in future Linux kernels, either by freeing some of the regular signals (unlikely), or by providing more than 32 signals (as per the POSIX 1003.1b realtime extensions).

In the meantime, you can try to use kernel-reserved signals either in your program or in LinuxThreads. For instance, SIGSTKFLT and SIGUNUSED appear to be unused in the current Linux kernels for the Intel x86 architecture. To use these in LinuxThreads, the only file you need to change is internals.h, more specifically the two lines:

        #define PTHREAD_SIG_RESTART SIGUSR1
        #define PTHREAD_SIG_CANCEL SIGUSR2
Replace them by e.g.
        #define PTHREAD_SIG_RESTART SIGSTKFLT
        #define PTHREAD_SIG_CANCEL SIGUNUSED
Warning: you're doing this at your own risks.

H.5: Is the stack of one thread visible from the other threads? Can I pass a pointer into my stack to other threads?

Yes, you can -- if you're very careful. The stacks are indeed visible from all threads in the system. Some non-POSIX thread libraries seem to map the stacks for all threads at the same virtual addresses and change the memory mapping when they switch from one thread to another. But this is not the case for LinuxThreads, as it would make context switching between threads more expensive, and at any rate might not conform to the POSIX standard.

So, you can take the address of an "auto" variable and pass it to other threads via shared data structures. However, you need to make absolutely sure that the function doing this will not return as long as other threads need to access this address. It's the usual mistake of returning the address of an "auto" variable, only made much worse because of concurrency. It's much, much safer to systematically heap-allocate all shared data structures.


I. X-Windows and other libraries

I.1: My program uses both Xlib and LinuxThreads. It stops very early with an "Xlib: unknown 0 error" message. What does this mean?

That's a prime example of the errno problem described in question H.2. The binaries for Xlib you're using have not been compiled with -D_REENTRANT. It happens Xlib contains a piece of code very much like the one in question H.2. So, your Xlib fetches the error code from the wrong errno location and concludes that an error it cannot handle occurred.

I.2: So, what can I do to build a multithreaded X Windows client?

The best solution is to recompile the X libraries with multithreading options set. They contain optional support for multithreading; it's just that all binary distributions for Linux were built without this support. See the file README.Xfree3.3 in the LinuxThreads distribution for patches and info on how to compile thread-safe X libraries from the Xfree3.3 distribution. The Xfree3.3 sources are readily available in most Linux distributions, e.g. as a source RPM for RedHat. Be warned, however, that X Windows is a huge system, and recompiling even just the libraries takes a lot of time and disk space.

Another, less involving solution is to call X functions only from the main thread of your program. Even if all threads have their own errno location, the main thread uses the global errno variable for its errno location. Thus, code not compiled with -D_REENTRANT still "sees" the right error values if it executes in the main thread only.

This is a lot of work. Don't you have precompiled thread-safe X libraries that you could distribute?

No, I don't. Sorry. But you could approach the maintainers of your Linux distribution to see if they would be willing to provide thread-safe X libraries.

I.3: Can I use library FOO in a multithreaded program?

Most libraries cannot be used "as is" in a multithreaded program. For one thing, they are not necessarily thread-safe: calling simultaneously two functions of the library from two threads might not work, due to internal use of global variables and the like. Second, the libraries must have been compiled with -D_REENTRANT to avoid the errno problems explained in question H.2.

I.4: What if I make sure that only one thread calls functions in these libraries?

This avoids problems with the library not being thread-safe. But you're still vulnerable to errno problems. At the very least, a recompile of the library with -D_REENTRANT is needed.

I.5: What if I make sure that only the main thread calls functions in these libraries?

That might actually work. As explained in question I.1, the main thread uses the global errno variable, and can therefore execute code not compiled with -D_REENTRANT.

I.6: SVGAlib doesn't work with LinuxThreads. Why?

Because both LinuxThreads and SVGAlib use the signals SIGUSR1 and SIGUSR2. One of the two should be recompiled to use different signals. See question H.4.


J. Signals and threads

J.1: When it comes to signals, what is shared between threads and what isn't?

Signal handlers are shared between all threads: when a thread calls sigaction(), it sets how the signal is handled not only for itself, but for all other threads in the program as well.

On the other hand, signal masks are per-thread: each thread chooses which signals it blocks independently of others. At thread creation time, the newly created thread inherits the signal mask of the thread calling pthread_create(). But afterwards, the new thread can modify its signal mask independently of its creator thread.

J.2: When I send a SIGKILL to a particular thread using pthread_kill, all my threads are killed!

That's how it should be. The POSIX standard mandates that all threads should terminate when the process (i.e. the collection of all threads running the program) receives a signal whose effect is to terminate the process (such as SIGKILL or SIGINT when no handler is installed on that signal). This behavior makes a lot of sense: when you type "ctrl-C" at the keyboard, or when a thread crashes on a division by zero or a segmentation fault, you really want all threads to stop immediately, not just the one that caused the segmentation violation or that got the SIGINT signal. (This assumes default behavior for those signals; see question J.3 if you install handlers for those signals.)

If you're trying to terminate a thread without bringing the whole process down, use pthread_cancel().

J.3: I've installed a handler on a signal. Which thread executes the handler when the signal is received?

If the signal is generated by a thread during its execution (e.g. a thread executes a division by zero and thus generates a SIGFPE signal), then the handler is executed by that thread. This also applies to signals generated by raise().

If the signal is sent to a particular thread using pthread_kill(), then that thread executes the handler.

If the signal is sent via kill() or the tty interface (e.g. by pressing ctrl-C), then the POSIX specs say that the handler is executed by any thread in the process that does not currently block the signal. In other terms, POSIX considers that the signal is sent to the process (the collection of all threads) as a whole, and any thread that is not blocking this signal can then handle it.

The latter case is where LinuxThreads departs from the POSIX specs. In LinuxThreads, there is no real notion of ``the process as a whole'': in the kernel, each thread is really a distinct process with a distinct PID, and signals sent to the PID of a thread can only be handled by that thread. As long as no thread is blocking the signal, the behavior conforms to the standard: one (unspecified) thread of the program handles the signal. But if the thread to which PID the signal is sent blocks the signal, and some other thread does not block the signal, then LinuxThreads will simply queue in that thread and execute the handler only when that thread unblocks the signal, instead of executing the handler immediately in the other thread that does not block the signal.

This is to be viewed as a LinuxThreads bug, but I currently don't see any way to implement the POSIX behavior without kernel support.

J.3: How shall I go about mixing signals and threads in my program?

The less you mix them, the better. Notice that all pthread_* functions are not async-signal safe, meaning that you should not call them from signal handlers. This recommendation is not to be taken lightly: your program can deadlock if you call a pthread_* function from a signal handler!

The only sensible things you can do from a signal handler is set a global flag, or call sem_post on a semaphore, to record the delivery of the signal. The remainder of the program can then either poll the global flag, or use sem_wait() and sem_trywait() on the semaphore.

Another option is to do nothing in the signal handler, and dedicate one thread (preferably the initial thread) to wait synchronously for signals, using sigwait(), and send messages to the other threads accordingly.

J.4: When one thread is blocked in sigwait(), other threads no longer receive the signals sigwait() is waiting for! What happens?

It's an unfortunate consequence of how LinuxThreads implements sigwait(). Basically, it installs signal handlers on all signals waited for, in order to record which signal was received. Since signal handlers are shared with the other threads, this temporarily deactivates any signal handlers you might have previously installed on these signals.

Though surprising, this behavior actually seems to conform to the POSIX standard. According to POSIX, sigwait() is guaranteed to work as expected only if all other threads in the program block the signals waited for (otherwise, the signals could be delivered to other threads than the one doing sigwait(), which would make sigwait() useless). In this particular case, the problem described in this question does not appear.

One day, sigwait() will be implemented in the kernel, along with others POSIX 1003.1b extensions, and sigwait() will have a more natural behavior (as well as better performances).


K. Internals of LinuxThreads

K.1: What is the implementation model for LinuxThreads?

LinuxThreads follows the so-called "one-to-one" model: each thread is actually a separate process in the kernel. The kernel scheduler takes care of scheduling the threads, just like it schedules regular processes. The threads are created with the Linux clone() system call, which is a generalization of fork() allowing the new process to share the memory space, file descriptors, and signal handlers of the parent.

Advantages of the "one-to-one" model include:

The main disadvantage is more expensive context switches on mutex and condition operations, which must go through the kernel. This is mitigated by the fact that context switches in the Linux kernel are pretty efficient.

K.2: Have you considered other implementation models?

There are basically two other models. The "many-to-one" model relies on a user-level scheduler that context-switches between the threads entirely in user code; viewed from the kernel, there is only one process running. This model is completely out of the question for me, since it does not take advantage of multiprocessors, and require unholy magic to handle blocking I/O operations properly. There are several user-level thread libraries available for Linux, but I found all of them deficient in functionality, performance, and/or robustness.

The "many-to-many" model combines both kernel-level and user-level scheduling: several kernel-level threads run concurrently, each executing a user-level scheduler that selects between user threads. Most commercial Unix systems (Solaris, Digital Unix, IRIX) implement POSIX threads this way. This model combines the advantages of both the "many-to-one" and the "one-to-one" model, and is attractive because it avoids the worst-case behaviors of both models -- especially on kernels where context switches are expensive, such as Digital Unix. Unfortunately, it is pretty complex to implement, and requires kernel support which Linux does not provide. Linus Torvalds and other Linux kernel developers have always been pushing the "one-to-one" model in the name of overall simplicity, and are doing a pretty good job of making kernel-level context switches between threads efficient. LinuxThreads is just following the general direction they set.

K.3: I looked at the LinuxThreads sources, and I saw quite a lot of spinlocks and busy-waiting loops to acquire these spinlocks. Isn't this a big waste of CPU time?

Look more carefully. Spinlocks are used internally to protect LinuxThreads's data structures, but these locks are held for very short periods of time: 10 instructions or so. The probability that a thread has to loop busy-waiting on a taken spinlock for more than, say, 100 cycles is very, very low. When a thread needs to wait on a mutex, condition, or semaphore, it actually puts itself on a waiting queue, then suspends on a signal, consuming no CPU time at all. The thread will later be restarted by sending it a signal when the state of the mutex, condition, or semaphore changes.


Xavier.Leroy@inria.fr