Using GCC's Nested Functions with Wide Pointers and No Trampolines II

This explains recent changes in how GCC implements nested functions and eliminates the need for executable stack trampolines. Historically, when code took the address of a nested function GCC created a trampoline on the stack, forcing an executable stack and undermining a key security mitigation. GCC offered heap-allocated trampolines as an alternative, but heap allocation is slower and trampolines can leak across longjmp. The practical solution described replaces runtime trampolines with a wide-pointer/closure mechanism that pairs a code pointer with a static-chain pointer so calls into a nested function carry its parent context without writable executable memory.

This matters because nested functions that do not capture their parent frame can now be used safely and portably without trampoline-induced security or performance problems. Starting with GCC 16 the compiler guarantees that a nested function which does not access parent-local variables requires no trampoline, even in unoptimized builds, and emits warnings when returning a nested function that clearly does capture locals. Such non-capturing nested functions retain access to static variables, constants, and non‑variably modified types, enabling common callback patterns without extra runtime machinery. Capturing nested functions are still valuable for direct lexical access, so a complete solution must handle the captured‑context case as well.

A development‑branch patch merged for GCC 17 implements two new builtins that, together with the existing builtin_call_with_static_chain, let the compiler materialize closures without trampolines: builtin_call_code_address and builtin_call_static_chain (used alongside builtin_call_with_static_chain). The compiler transforms a capturing nested function into a pair of values - a code address and a static‑chain pointer passed via a special ABI register reserved for static chains - and calls are routed through these builtins. A practical C pattern uses a generic wide(T) struct (e.g. struct wide_cb_f { typeof(cb_f) code; void *chain; }) and macros like CLOSURE/CALL to present a closure type; GCC can optimize such examples down to a single instruction (the sample foo compiled to lea eax, rdi+rdi*2; ret). Limitations: the builtins live in the GCC development branch (GCC 17) and aren’t language‑level features, portability remains an issue with other compilers (clang) and older GCC versions, and the compiler only warns about obvious simple captures.