this is the first part of a series of patches intended to make
__syscall fully self-contained in the object file produced using
syscall.h, which will make it possible for crt1 code to perform
syscalls.
the (confusingly named) i386 __vsyscall mechanism, which this commit
removes, was introduced before the presence of a valid thread pointer
was mandatory; back then the thread pointer was setup lazily only if
threads were used. the intent was to be able to perform syscalls using
the kernel's fast entry point in the VDSO, which can use the sysenter
(Intel) or syscall (AMD) instruction instead of int $128, but without
inlining an access to the __syscall global at the point of each
syscall, which would incur a significant size cost from PIC setup
everywhere. the mechanism also shuffled registers/calling convention
around to avoid spills of call-saved registers, and to avoid
allocating ebx or ebp via asm constraints, since there are plenty of
broken-but-supported compiler versions which are incapable of
allocating ebx with -fPIC or ebp with -fno-omit-frame-pointer.
the new mechanism preserves the properties of avoiding spills and
avoiding allocation of ebx/ebp in constraints, but does it inline,
using some fairly simple register shuffling, and uses a field of the
thread structure rather than global data for the vdso-provided syscall
code address.
for now, the external __syscall function is refactored not to use the
old __vsyscall so it can be kept, but the intent is to remove it too.
this is a workaround to avoid a crashing regression on qemu-user when
dynamic TLS is installed at dlopen time. the sigaction syscall should
not be able to fail, but it does fail for implementation-internal
signals under qemu user-level emulation if the host libc qemu is
running under reserves the same signals for implementation-internal
use, since qemu makes no provision to redirect/emulate them. after
sigaction fails, the subsequent tkill would terminate the process
abnormally as the default action.
no provision to account for membarrier failing is made in the dynamic
linker code that installs new TLS. at the formal level, the missing
barrier in this case is incorrect, and perhaps we should fail the
dlopen operation, but in practice all the archs we support (and
probably all real-world archs except alpha, which isn't yet supported)
should give the right behavior with no barrier at all as a consequence
of consume-order properties.
in the long term, this workaround should be supplemented or replaced
by something better -- a different fallback approach to ensuring
memory consistency, or dynamic allocation of implementation-internal
signals. the latter is appealing in that it would allow cancellation
to work under qemu-user too, and would even allow many levels of
nested emulation.
This parameter was incorrectly declared to be a pointer to a function
accepting zero parameters. The intent of makecontext is that it is
possible to pass integer parameters to the function, so this should
have been a pointer to a function accepting an unspecified set of
parameters.
Mark atanhi, atanlo, and aT in atanl.c as static, as they're not
intended to be part of the public API.
These are already static in the LDBL_MANT_DIG == 64 code, so this
patch is just making the LDBL_MANT_DIG == 113 code do the same thing.
The result is the same but takes less code.
Note that __execvpe calls getenv which calls __strchrnul so even
using static output the size of the executable won't grow.
commit 54ca677983 inadvertently
introduced bitwise and where logical and was intended. since the
right-hand operand is always 0 or -1 whenever the left-hand operand is
nonzero, the behavior happened to be equivalent.
priority inheritance is a feature to mitigate priority inversion
situations, where a execution of a medium-priority thread can
unboundedly block forward progress of a high-priority thread when a
lock it needs is held by a low-priority thread.
the natural way to do priority inheritance would be with a simple
futex flag to donate the calling thread's priority to a target thread
while it waits on the futex. unfortunately, linux does not offer such
an interface, but instead insists on implementing the whole locking
protocol in kernelspace with special futex commands that exist solely
for the purpose of doing PI mutexes. this would require the entire
"trylock" logic to be duplicated in the timedlock code path for PI
mutexes, since, once the previous lock holder releases the lock and
the futex call returns, the lock is already held by the caller.
obviously such code duplication is undesirable.
instead, I've made the PI timedlock success path set the mutex lock
count to -1, which can be thought of as "not yet complete", since a
lock count of 0 is "locked, with no recursive references". a simple
branch in a non-hot path of pthread_mutex_trylock can then see and act
on this state, skipping past the code that would check and take the
lock to the same code path that runs after the lock is obtained for a
non-PI mutex.
because we're forced to let the kernel perform the actual lock and
unlock operations whenever the mutex is contended, we have to patch
things up when it does the wrong thing:
1. the lock operation is not aware of whether the mutex is
error-checking, so it will always fail with EDEADLK rather than
deadlocking.
2. the lock operation is not aware of whether the mutex is robust, so
it will successfully obtain mutexes in the owner-died state even if
they're non-robust, whereas this operation should deadlock.
3. the unlock operation always sets the lock value to zero, whereas
for robust mutexes, we want to set it to a special value indicating
that the mutex obtained after its owner died was unlocked without
marking it consistent, so that future operations all fail with
ENOTRECOVERABLE.
the first of these is easy to solve, just by performing a futex wait
on a dummy futex address to simulate deadlock or ETIMEDOUT as
appropriate. but problems 2 and 3 interact in a nasty way. to solve
problem 2, we need to back out the spurious success. but if waiters
are present -- which we can't just ignore, because even if we don't
want to wake them, the calling thread is incorrectly inheriting their
priorities -- this requires using the kernel's unlock operation, which
will zero the lock value, thereby losing the "owner died with lock
held" state.
to solve these problems, we overload the mutex's waiters field, which
is unused for PI mutexes since they don't call the normal futex wait
functions, as an indicator that the PI mutex is permanently
non-lockable. originally I wanted to use the count field, but there is
one code path that needs to access this flag without synchronization:
trylock's CAS failure path needs to be able to decide whether to fail
with EBUSY or ENOTRECOVERABLE, the waiters field is already treated as
a relaxed-order atomic in our memory model, so this works out nicely.
there was no point in masking off the pshared bit when first loading
the type, since every subsequent access involves a mask anyway. not
masking it may avoid a subsequent load to check the pshared flag, and
it's just simpler.
commit 84d061d5a3 wrongly moved the
access to the global next_key outside of the scope of the lock. the
error manifested as spurious failure to find an available key slot
under concurrent calls to pthread_key_create, since the stopping
condition could be met after only a small number of slots were
examined.
commit d6c855caa8 caused this
"regression", though the behavior was undefined before, overlooking
that f->shend=0 was being used as a sentinel for "EOF" status (actual
EOF or hitting the scanf field width) of the stream helper (shgetc)
functions.
obviously the shgetc macro could be adjusted to check for a null
pointer in addition to the != comparison, but it's the hot path, and
adding extra code/branches to it begins to defeat the purpose.
so instead of setting shend to a null pointer to block further reads,
which no longer works, set it to the current position (rpos). this
makes the shgetc macro work with no change, but it breaks shunget,
which can no longer look at the value of shend to determine whether to
back up. Szabolcs Nagy suggested a solution which I'm using here:
setting shlim to a negative value is inexpensive to test at shunget
time, and automatically re-trips the cnt>=shlim stop condition in
__shgetc no matter what the original limit was.
commit 2de29bc994 left behind one
reference to pthread_mutex_trylock. fixing this also improves code
generation due to the namespace-safe version being hidde.
HWCAP_SB - speculation barrier instruction available added in linux
commit bd4fb6d270bc423a9a4098108784f7f9254c4e6d
HWCAP_PACA, HWCAP_PACG - pointer authentication instructions available
(address and generic) added in linux commit
7503197562567b57ec14feb3a9d5400ebc56812f
aarch64 pointer authentication code related prctl that allows
reinitializing the key for the thread, added in linux commit
ba830885656414101b2f8ca88786524d4bb5e8c1
NT_MIPS_MSA for ptrace access to mips simd arch reg set, added in linux
commit 3cd640832894b85b5929d5bda74505452c800421
NT_ARM_PAC_MASK for ptrace access to pointer auth code mask, added in
commit ec6e822d1a22d0eef1d1fa260dff751dba9a4258
C-SKY support was added to binutils 2.32 in commit
b8891f8d622a31306062065813fc278d8a94fe21
the elf.h change was added to glibc 2.29 in commit
4975f0c3d0131fdf697be0b1631c265e5fd39088
NT_MIPS_FP_MODE is new in linux commit
1ae22a0e35636efceab83728ba30b013df761592
NT_MIPS_DSP is new in linux commit
44109c60176ae73924a42a6bef64ef151aba9095
new fields for RFC 4898 tcp stats in linux
tcpi_bytes_sent added in commit ba113c3aa79a7f941ac162d05a3620bdc985c58d
tcpi_bytes_retrans added in commit fb31c9b9f6c85b1bad569ecedbde78d9e37cd87b
tcpi_dsack_dups added in commit 7e10b6554ff2ce7f86d5d3eec3af5db8db482caa
tcpi_reord_seen added in commit 7ec65372ca534217b53fd208500cf7aac223a383
The new fields change the size of a public struct and thus an ABI break,
but this is how the getsockopt TCP_INFO api is designed: the tcp_info
type must only be used with a length parameter in extern interfaces.
inotify_add_watch flag to prevent modifying existing watch descriptors,
when used on an already watched inode it fails with EEXIST.
added in linux commit 4d97f7d53da7dc830dbf416a3d2a6778d267ae68
The original logic considered each byte until it either found a 0
value or a value >= 192. This means if a string segment contained any
byte >= 192 it was interepretted as a compressed segment marker even
if it wasn't in a position where it should be interpretted as such.
The fix is to adjust dn_skipname to increment by each segments size
rather than look at each character. This avoids misinterpretting
string segment characters by not considering those bytes.
On s390x, POSIX_FADV_DONTNEED and POSIX_FADV_NOREUSE have different
values than on all other architectures that Linux supports.
Handle this difference by wrapping their definitions in
include/fcntl.h in #ifdef, so that arch/s390x/bits/fcntl.h can
override them.
as noted in Austin Group issue #1236, the XSI shading for TSVTX is
misplaced in the html version of the standard; it was only supposed to
be on the description text. the intent was that the definition always
be visible, which is reflected in the pdf version of the standard.
this reverts commits d93c0740d8 and
729fef0a93.
C11 removed the requirement that FILE be a complete type, which was
deemed erroneous, as part of the changes introduced by N1439 regarding
completeness of types (see footnote 6 for specific mention of FILE).
however the current version of POSIX is still based on C99 and
incorporates the old requirement that FILE be a complete type.
expose an arbitrary, useless complete type definition because the
actual object used to represent FILE streams cannot be public/ABI.
thanks to commit 13d1afa46f, we now have
a framework for suppressing the public complete-type definition of FILE
when stdio.h is included internally, so that a different internal
definition can be provided. this is perfectly well-defined, since the
same struct tag can refer to different types in different translation
units. it would be a problem if the implementation were accessing the
application's FILE objects or vice versa, but either would be
undefined behavior.
this affected the error path where dlopen successfully found and
loaded the requested dso and all its dependencies, but failed to
resolve one or more relocations, causing the operation to fail after
storage for the ctor queue was allocated.
commit 188759bbee wrongly put the free
for the ctor_queue array in the error path inside a loop over each
loaded dso that needed to be backed-out, rather than just doing it
once. in addition, the exit path also observed the ctor_queue pointer
still being nonzero, and would attempt to call ctors on the backed-out
dsos unless the double-free crashed the process first.
historically, and likely accidentally, sigaltstack was specified to
fail with EINVAL if any flag bit other than SS_DISABLE was set. the
resolution of Austin Group issue 1187 fixes this so that the
requirement is only to fail for SS_ONSTACK (which cannot be set) or
"invalid" flags.
Linux fails on the kernel side for invalid flags, but historically
accepts SS_ONSTACK as a no-op, so it needs to be rejected in userspace
still.
with this change, the Linux-specific SS_AUTODISARM, provided since
commit 9680e1d03a but unusable due to
rejection at runtime, is now usable.
together with the previous two commits, this completes restoration of
the property that dynamic-linked apps with no external deps and no tls
have no failure paths before entry.
neither has or can have any dependencies, but since commit
4035556907, gratuitous zero-length deps
arrays were being allocated for them. use a dummy array instead.
traditionally, we've provided a guarantee that dynamic-linked
applications with no external dependencies (nothing but libc) and no
thread-local storage have no failure paths before the entry point.
normally, thanks to reclaim_gaps, such a malloc will not require a
syscall anyway, but if segment alignment is unlucky, it might. use a
builtin array for this common special case.
in the case where malloc is being replaced, it's not valid to call
malloc between final relocations and main app's crt1 entry point; on
fdpic archs the main app's entry point will not yet have performed the
self-fixups necessary to call its code.
to fix, reorder queue_ctors before final relocations. an alternative
solution would be doing the allocation from __libc_start_init, after
the entry point but before any ctors run. this is less desirable,
since it would leave a call to malloc that might be provided by the
application happening at startup when doing so can be easily avoided.
previously, going way back, there was simply no synchronization here.
a call to exit concurrent with ctor execution from dlopen could cause
a dtor to execute concurrently with its corresponding ctor, or could
cause dtors for newly-constructed libraries to be skipped.
introduce a shutting_down state that blocks further ctor execution,
producing the quiescence the dtor execution loop needs to ensure any
kind of consistency, and that blocks further calls to dlopen so that a
call into dlopen from a dtor cannot deadlock.
better approaches to some of this may be possible, but the changes
here at least make things safe.
previously, shared library constructors at program start and dlopen
time were executed in reverse load order. some libraries, however,
rely on a depth-first dependency order, which most other dynamic
linker implementations provide. this is a much more reasonable, less
arbitrary order, and it turns out to have much better properties with
regard to how slow-running ctors affect multi-threaded programs, and
how recursive dlopen behaves.
this commit builds on previous work tracking direct dependencies of
each dso (commit 4035556907), and
performs a topological sort on the dependency graph at load time while
the main ldso lock is held and before success is committed, producing
a queue of constructors needed by the newly-loaded dso (or main
application). in the case of circular dependencies, the dependency
chain is simply broken at points where it becomes circular.
when the ctor queue is run, the init_fini_lock is held only for
iteration purposes; it's released during execution of each ctor, so
that arbitrarily-long-running application code no longer runs with a
lock held in the caller. this prevents a dlopen with slow ctors in one
thread from arbitrarily delaying other threads that call dlopen.
fully-independent ctors can run concurrently; when multiple threads
call dlopen with a shared dependency, one will end up executing the
ctor while the other waits on a condvar for it to finish.
another corner case improved by these changes is recursive dlopen
(call from a ctor). previously, recursive calls to dlopen could cause
a ctor for a library to be executed before the ctor for its
dependency, even when there was no relation between the calling
library and the library it was loading, simply due to the naive
reverse-load-order traversal. now, we can guarantee that recursive
dlopen in non-circular-dependency usage preserves the desired ctor
execution order properties, and that even in circular usage, at worst
the libraries whose ctors call dlopen will fail to have completed
construction when ctors that depend on them run.
init_fini_lock is changed to a normal, non-recursive mutex, since it
is no longer held while calling back into application code.
Rich Felker
Ilya Matveychikov
Bobby Bingham
Dan Gohman
Frediano Ziglio
Ray
Drew DeVault
Szabolcs Nagy
Ryan Fairfax
Jonathan Neuschäfer
A. Wilcox