# What and Why

Thread Local Storage (TLS) gives each thread a distinct instance of a variable while preserving global-like access. Accesses to the TLS object itself do not race with another thread's instance, but data reached through a TLS-held pointer can still be shared and require synchronization.

Classic use cases include:

• errno: The quintessential example. System calls set errno, but in a multi-threaded program each thread needs its own errno to avoid races.
• Allocator caches: jemalloc uses thread-specific caches; TCMalloc can run in legacy per-thread mode, while modern TCMalloc also supports a per-logical-CPU front end. These caches reduce contention on central allocator structures.
• Random number generator state: Thread-local RNG state avoids locking on every random number generation.
• Scratch buffers: Thread-local temporary buffers avoid repeated allocation.

# C/C++ Basics

In C and C++, TLS variables are declared with storage class specifiers:

// GCC extension (works in C and C++)
__thread int counter = 0;

// C11 standard
_Thread_local int counter = 0;

// C++11 standard
thread_local int counter = 0;

Initialization constraints differ:

• __thread (C/GCC): Only constant initializers allowed. No constructors, no function calls, no runtime computation.
• thread_local (C++11): Dynamic initialization is permitted. For a non-block variable, initialization may occur before the thread's initial function or be deferred; if deferred, it is sequenced before the relevant first non-initialization odr-use. Constructed objects are destroyed at thread exit.

// C with __thread: only constants allowed
__thread int x = 42;              // OK
__thread int y = some_function(); // ERROR: not a constant
__thread MyClass obj;             // ERROR: has constructor

// C++ with thread_local: dynamic init OK
thread_local int x = compute();   // OK: runs once per thread
thread_local MyClass obj(args);   // OK: constructed per thread

# ELF TLS Models and Linkage

How TLS variables are accessed depends on the TLS model chosen by the compiler and linker. This is where linkage significantly impacts TLS behavior and performance. There are four models, trading off generality against efficiency:

ELF TLS Models: Performance vs Flexibility Tradeoff

* Cached per function — one call gets base, then offsets added.

Local-exec is the fastest model. The compiler knows the TLS variable is in the main executable, so it can compute the offset at link time and emit a single instruction accessing %fs:offset directly.

Initial-exec works for shared libraries loaded at program startup (not via dlopen). The offset isn't known until load time, so it's stored in the GOT and requires a GOT lookup, but still avoids function calls.

General-dynamic is the most flexible but slowest. It works for any TLS variable anywhere, including in dlopen'd libraries. Each access calls __tls_get_addr() to resolve the address at runtime.

Local-dynamic optimizes the case where a function accesses multiple TLS variables from the same module. One __tls_get_addr() call gets the module's TLS base address, then individual variables are accessed via offsets from that base.

Access Code Comparison (x86-64):

Local-exec (fastest):
    mov   %fs:tls_var@TPOFF, %eax    # Single instruction, offset known at link time

Initial-exec:
    mov   tls_var@GOTTPOFF(%rip), %rax   # Load offset from GOT
    mov   %fs:(%rax), %eax               # Access via offset

General-dynamic (slowest):
    lea   tls_var@TLSGD(%rip), %rdi  # Load TLS descriptor address
    call  __tls_get_addr@PLT         # Function call!
    mov   (%rax), %eax               # Dereference result

How the compiler chooses: By default, compilers use general-dynamic for shared libraries (safest) and initial-exec or local-exec for executables. You can override with -ftls-model=:

# Force a specific TLS model
gcc -ftls-model=local-exec  ...   # Only for main executable
gcc -ftls-model=initial-exec ...  # Won't work if dlopen'd
gcc -ftls-model=local-dynamic ... # Optimizes multi-var access
gcc -ftls-model=global-dynamic ...# Default for shared libs

# TLS Access Under the Hood

On x86-64 Linux, TLS works through the %fs segment register. Each thread has its %fs base pointing to its Thread Control Block (TCB), which contains or points to that thread's TLS data.

For local-exec and initial-exec loads on x86-64, access can use a %fs-relative instruction.

The static TLS offset is either:

• Embedded at link time (local-exec)
• Loaded from the GOT at runtime (initial-exec)
• For traditional general/local-dynamic sequences, __tls_get_addr returns the variable or module-base address rather than a %fs-relative offset.

# __tls_get_addr and the Dynamic Thread Vector

For general-dynamic and local-dynamic TLS, the runtime function __tls_get_addr resolves TLS addresses. Understanding how it works explains why it's slower and why it enables dlopen compatibility.

Dynamic Thread Vector (DTV): per-thread module index. Each thread has a DTV — an array of pointers to TLS blocks:

Module IDs are assigned by the dynamic linker:

• Executable is always module 1
• Libraries get IDs as they're loaded

How __tls_get_addr works:

// A traditional GNU dynamic-TLS sequence calls:
void *addr = __tls_get_addr(&tls_index);

// tls_index contains:
struct tls_index {
    unsigned long module_id;  // Which module? (filled by dynamic linker)
    unsigned long offset;     // Offset within module's TLS block
};

// In glibc, __tls_get_addr does:
1. Read this thread's DTV
2. If its generation is stale, update/resize it
3. Read DTV[module_id]
4. If it is TLS_DTV_UNALLOCATED, allocate and initialize the dynamic block
5. Return TLS_block + offset

For example, a conceptual traditional-ABI __tls_get_addr(module=3, offset=16) lookup:

The glibc generation-counter trick: When dlopen loads a new library with TLS, a thread's DTV may need to grow. Instead of updating every thread's DTV immediately (expensive!), the loader increments a global generation counter. When __tls_get_addr is called, it compares the thread's DTV generation to the global one. If stale, it updates the DTV, resizing it only when the existing array is too small. This lazy update avoids touching threads that never access the new library's TLS.

# Static TLS Exhaustion

The infamous error "cannot allocate memory in static TLS block" occurs when dlopen'ing a library that uses initial-exec TLS model, but there's insufficient space reserved in the static TLS block.

Typical surplus sizes:

• glibc: With default tunables, current glibc preserves a historic total static-TLS surplus of 1664 bytes. The glibc.rtld.optional_static_tls component defaults to 512 bytes and can be set through GLIBC_TUNABLES; additional internal/alignment space can apply.
• musl: musl does not reserve post-startup static TLS for new definitions. It rejects an initial-exec relocation when that relocation resolves to a TLS definition in a module loaded after startup.

Mitigation strategies:

// BAD: 4KB in static TLS per thread
__thread char buffer[4096];

// TRADE-OFF: one pointer-sized TLS slot (8 bytes on LP64); allocate data on demand
__thread char *buffer;

void ensure_buffer(void) {
    if (!buffer) {
        buffer = malloc(4096);  // Heap allocation, not TLS
    }
}

// Free each thread's allocation at thread exit or transfer it to a deliberate owner;
// this minimal example otherwise leaks it.

Diagnosing TLS issues:

# Inspect the PT_TLS template size and alignment
readelf -Wl libfoo.so | grep TLS

# Check whether the DSO carries the static-TLS requirement
readelf -Wd libfoo.so | grep STATIC_TLS

# Inspect x86-64 TLS relocations
readelf -Wr libfoo.so | grep -E 'DTPMOD|DTPOFF|TPOFF|TLSDESC'

# glibc vs musl: Critical Differences

Code that works on glibc may fail on musl (used by Alpine Linux and many embedded systems). The differences stem from fundamentally different design philosophies:

musl's philosophy: Pre-allocate everything so failures happen early (at dlopen) rather than late (random crash when TLS is first accessed). The tradeoff is stricter constraints on what can be dlopen'd, and libraries are permanent once loaded.

# Gotchas and Pitfalls

• Memory overhead: N threads × M bytes per TLS variable. A 1KB TLS buffer with 100 threads = 100KB. Use pointers and lazy allocation for large data.
• Destruction order (C++): thread_local objects whose construction is sequenced are destroyed in reverse order. Where their initialization is not sequenced, a dependable single order is not guaranteed. A destructor that touches another TLS object may therefore encounter it already destroyed; avoid fragile cross-TLS dependencies.
• fork() and TLS: After fork() in a multithreaded process, the child contains only the calling thread, and that thread's TLS state is copied with the address space. Cached thread IDs or synchronization/coordination state may need reinitialization. Open file descriptors themselves remain inherited and refer to the same open file descriptions.
• Signal handlers: For portable, conservative code, do not access ordinary application TLS-backed state or perform ordinary application work in an asynchronous signal handler. Limit the handler to a documented async-signal-safe notification—for example, setting a volatile sig_atomic_t flag or making a small, best-effort write() to a pre-opened nonblocking self-pipe. A handler that may call write() must save errno on entry and restore it before returning, including when a full pipe makes the nonblocking write return EAGAIN. POSIX permits fetching and setting errno under that save-and-restore rule; let ordinary execution access application TLS and do the real work.
• dlopen with RTLD_LOCAL vs RTLD_GLOBAL: Affects symbol visibility, which can impact TLS resolution if libraries have interdependencies.
• Cross-compilation: TLS model availability varies by platform. Code assuming initial-exec may fail on platforms with different TLS implementations.

# Rust Specifics

Rust provides TLS through two mechanisms with different tradeoffs:

thread_local! macro (stable):

use std::cell::Cell;

thread_local! {
    static COUNTER: Cell<u32> = Cell::new(0);
}

fn main() {
    // Access requires .with() closure - cannot get direct reference
    COUNTER.with(|c| {
        c.set(c.get() + 1);
        println!("Counter: {}", c.get());
    });
}

The .with() pattern exists because thread_local! uses lazy initialization. The closure ensures the TLS is initialized before access and prevents returning references that could outlive the thread.

thread_local! access via .with():

  COUNTER.with(|c| ...)
        |
        v
  +--> LocalKey::with()
        +--> Is this thread's slot initialized?
              +--> No:  Run initializer (Cell::new(0))
              |        Store in TLS
              |        Then call closure with &Cell
              |
              +--> Yes: Call closure with &Cell directly

#[thread_local] attribute (unstable/nightly):

#![feature(thread_local)]

use std::cell::Cell;

#[thread_local]
static COUNTER: Cell<u32> = Cell::new(0);

fn main() {
    // Direct access - no closure needed
    COUNTER.set(COUNTER.get() + 1);
    println!("Counter: {}", COUNTER.get());
}

#[thread_local] is unstable and places a static in target TLS. Access cost depends on the selected TLS model, target, visibility, and linker relaxation; local-exec can produce a single %fs-relative load on x86-64. As a static item, its initializer is a constant expression.

Rust TLS gotchas:

• Destructors are best effort: Values implementing Drop are destroyed on normal thread exit, including a panic that unwinds to the thread boundary, but process abort/exit and the documented platform-specific cases may skip them. Do not rely on TLS destruction for critical cleanup.
• Performance difference: thread_local! adds function call overhead for the lazy init check. For hot paths, this can matter. #[thread_local] avoids this but requires nightly.
• No direct references: thread_local! intentionally prevents getting &'static T to avoid lifetime issues. Use .with() or consider other patterns if you need persistent references.

# Debugging TLS

When TLS goes wrong, these tools help diagnose issues:

# GDB: select a thread, then evaluate that thread's TLS instance
(gdb) info threads
(gdb) thread 2
(gdb) p my_tls_var
(gdb) p &my_tls_var

# Inspect the PT_TLS template
readelf -Wl ./mybinary | grep TLS

# Inspect x86-64 TLS relocations / static-TLS requirement
readelf -Wr libfoo.so | grep -E 'DTPMOD|DTPOFF|TPOFF|TLSDESC'
readelf -Wd libfoo.so | grep STATIC_TLS

# glibc 2.43+: trace TLS TCB allocation, deallocation, and reuse
# Check installed-loader support with: LD_DEBUG=help ./myprogram
LD_DEBUG=tls ./myprogram

Common symptoms and causes:

• "cannot allocate memory in static TLS block": dlopen'd library uses initial-exec TLS, static block exhausted. Recompile library with -ftls-model=global-dynamic.
• TLS variable has an unexpected copied value after fork: The child retained the calling thread's TLS state; reinitialize process- or thread-specific caches in the child when needed.
• Signal handling needs TLS-backed work: Do not access ordinary application TLS-backed state in the handler. Have it send only a documented async-signal-safe notification, such as setting a volatile sig_atomic_t flag or making a small, best-effort write() to a pre-opened nonblocking self-pipe. A handler that may call write() must save errno on entry and restore it before returning, including when a full pipe makes the nonblocking write return EAGAIN. POSIX permits fetching and setting errno under that rule. Access application TLS and perform the work during ordinary execution.
• TLS works in tests, fails in production: One possible cause is different linkage or TLS-model constraints—for example, startup-loaded test code versus a plugin whose TLS definition is introduced after startup. Inspect the ELF TLS relocations and loader behavior rather than assuming the model from “static” or “dlopen” alone.