In the public header, __errno_location is declared with the "const"
attribute, conditional on __GNUC__. Ensure that its internal alias has
the same attributes.
Maintainer's note: This change also fixes a regression in quality of
code generation -- multiple references to errno in a single function
started generating multiple calls again -- introduced by commit
e13063aad7.
The old/new parameters to pthread_sigmask, sigprocmask, and setitimer
are marked restrict, so passing the same address to both is
prohibited. Modify callers of these functions to use a separate object
for each argument.
as reported by Tavian Barnes, a dup2 file action for the internal pipe
fd used by posix_spawn could cause it to remain open after execve and
allow the child to write an artificial error into it, confusing the
parent. POSIX allows internal use of file descriptors by the
implementation, with undefined behavior for poking at them, so this is
not a conformance problem, but it seems preferable to diagnose and
prevent the error when we can do so easily.
catch attempts to apply a dup2 action to the internal pipe fd and
emulate EBADF for it instead.
Linux v5.1 introduced ipc syscalls on targets where previously only
SYS_ipc was available, change the logic such that the ipc code keeps
using SYS_ipc which works backward compatibly on older kernels.
This changes behaviour on microblaze which had both mechanisms, now
SYS_ipc will be used instead of separate syscalls.
synccall may be called by AS-safe functions such as setuid/setgid after
fork. although fork() resets libc.threads_minus_one, causing synccall to
take the single-threaded path, synccall still takes the thread list
lock. This lock may be held by another thread if for example fork()
races with pthread_create(). After fork(), the value of the lock is
meaningless, so clear it.
maintainer's note: commit 8f11e6127f and
e4235d7067 introduced this regression.
the state protected by this lock is the linked list, which is entirely
replaced in the child path of fork (next=prev=self), so resetting it
is semantically sound.
the linux syscall treats this argument as having type int, so passing
extremely long buffer sizes would be misinterpreted by the kernel.
since "short reads" are always acceptable, just cap it down.
patch based on report and suggested change by Florian Weimer.
previously, POSIX erroneously required this to fail with EINVAL
despite the traditional glibc implementation, on which the POSIX
interface was based, allowing it. the resolution of Austin Group issue
818 removes the requirement to fail.
Author: Alex Suykov <alex.suykov@gmail.com>
Author: Aric Belsito <lluixhi@gmail.com>
Author: Drew DeVault <sir@cmpwn.com>
Author: Michael Clark <mjc@sifive.com>
Author: Michael Forney <mforney@mforney.org>
Author: Stefan O'Rear <sorear2@gmail.com>
This port has involved the work of many people over several years. I
have tried to ensure that everyone with substantial contributions has
been credited above; if any omissions are found they will be noted
later in an update to the authors/contributors list in the COPYRIGHT
file.
The version committed here comes from the riscv/riscv-musl repo's
commit 3fe7e2c75df78eef42dcdc352a55757729f451e2, with minor changes by
me for issues found during final review:
- a_ll/a_sc atomics are removed (according to the ISA spec, lr/sc
are not safe to use in separate inline asm fragments)
- a_cas[_p] is fixed to be a memory barrier
- the call from the _start assembly into the C part of crt1/ldso is
changed to allow for the possibility that the linker does not place
them nearby each other.
- DTP_OFFSET is defined correctly so that local-dynamic TLS works
- reloc.h LDSO_ARCH logic is simplified and made explicit.
- unused, non-functional crti/n asm files are removed.
- an empty .sdata section is added to crt1 so that the
__global_pointer reference is resolvable.
- indentation style errors in some asm files are fixed.
with the glibc generation counter model for reusing dynamic tls slots
after dlclose, it's really not possible to get away with fewer than 4
working registers. for us however it's always been possible, but
tricky, and only became apparent after the switch to installing new
dynamic tls at dlopen time. by merging the negated thread pointer into
the addend early, the register holding the thread pointer can
immediately be reused, bringing the working register count down to
three. this allows saving/restoring via a single stp/ldp pair, since
the return register x0 does not need to be saved.
net reduction of 3 instructions, 2 of which were push/pop.
currently the bfd linker does not seem to create tls segments where
p_vaddr%p_align != 0, but this is valid in ELF and then the runtime
computed tls offset must satisfy
offset%p_align == (base+p_vaddr)%p_align
and in case of local exec tls (main executable) the smallest such
offset must be used (otherwise it is incompatible with the offset
computed by the static linker). the !TLS_ABOVE_TP case is handled
correctly (the offset is negative then in the formula).
the ldso code for TLS_ABOVE_TP is changed so the static tls offset
of each module satisfies the formula.
commit 648c3b4e18 omitted this change,
which is needed to be able to use uid/gid values greater than INT_MAX
with these interfaces. it fixes alpine linux bug #10460.
this is a requirement in POSIX that's omitted, and seemed potentially
non-conforming, in the C standard. as such it was omitted here.
however, as part of Austin Group issue #1170, the discrepancy was
raised with WG14 and determined to be unintended; future versions of
the C standard will require the error indicator to be set, as POSIX
does.
commit 788d5e24ca noted that we could
add this if needed, and in fact it is needed, but not for one of the
archs documented as having a 7th syscall arg register. rather, it's
needed for mips (o32), where all but the first 4 arguments are passed
on the stack, and the stack can accommodate a 7th.
commit b50d315fd2 introduced
fp_force_eval implemented by default with a dead store to a volatile
variable. unfortunately introduces warnings with -Wunused-variable and
breaks the ability to use -Werror with the default warning options set
by configure when warnings are enabled.
we could just call fp_barrier instead, but that results in a spurious
load after the store due to volatile semantics.
the fix committed here avoids the load. it will still produce warnings
without -Wno-unused-but-set-variable, but that's part of our default
warning profile, and there are already other locations in the source
where an unused variable warning will occur without it.
from https://github.com/ARM-software/optimized-routines,
commit 04884bd04eac4b251da4026900010ea7d8850edc
The underflow exception is signaled if the result is in the subnormal
range even if the result is exact.
code size change: +3421 bytes.
benchmark on x86_64 before, after, speedup:
-Os:
pow rthruput: 102.96 ns/call 33.38 ns/call 3.08x
pow latency: 144.37 ns/call 54.75 ns/call 2.64x
-O3:
pow rthruput: 98.91 ns/call 32.79 ns/call 3.02x
pow latency: 138.74 ns/call 53.78 ns/call 2.58x
from https://github.com/ARM-software/optimized-routines,
commit 04884bd04eac4b251da4026900010ea7d8850edc
POWF_SCALE != 1.0 case only matters if TOINT_INTRINSICS is set, which
is currently not supported for any target.
SNaN is not supported, it would require an issignalingf
implementation.
code size change: -816 bytes.
benchmark on x86_64 before, after, speedup:
-Os:
powf rthruput: 95.14 ns/call 20.04 ns/call 4.75x
powf latency: 137.00 ns/call 34.98 ns/call 3.92x
-O3:
powf rthruput: 92.48 ns/call 13.67 ns/call 6.77x
powf latency: 131.11 ns/call 35.15 ns/call 3.73x
Musl currently aims to support non-nearest rounding mode and does not
support SNaNs. These macros allow marking relevant code paths in case
these decisions are changed later (they also help documenting the
corner cases involved).
These don't have an effectw with -Os so not useful with default settings
other than documenting the expectation.
With --enable-optimize=internal,malloc,string,math the libc.so code size
increases by 18K on x86_64 and performance varies in -2% .. +10%.
These are supposed to be used in tail call positions when handling
special cases in new code. (fp exceptions may be raised "naturally"
by the common code path if special casing is more effort.)
This implements the error handling apis used in
https://github.com/ARM-software/optimized-routines
without errno setting.
Previously type casts or assignments were used for handling excess
precision, which assumed standard C99 semantics, but since it's a
rarely needed obscure detail, it's better to use explicit helper
functions to document where we rely on this. It also helps if the
code is used outside of the libc in non-C99 compilation mode: with the
default excess precision handling of gcc, explicit inline asm barriers
are needed for narrowing on FLT_EVAL_METHOD!=0 targets.
I plan to use this in new code with the existing style that uses
double_t and float_t as much as possible.
One ugliness is that it is required for almost every return statement
since that does not drop excess precision (the standard changed this
in C11 annex F, but that does not help in non-standard compilation
modes or with old compilers).
C99 has ways to support fenv access, but compilers don't implement it
and assume nearest rounding mode and no fp status flag access. (gcc has
-frounding-math and then it does not assume nearest rounding mode, but
it still assumes the compiled code itself does not change the mode.
Even if the C99 mechanism was implemented it is not ideal: it requires
all code in the library to be compiled with FENV_ACCESS "on" to make it
usable in non-nearest rounding mode, but that limits optimizations more
than necessary.)
The math functions should give reasonable results in all rounding modes
(but the quality may be degraded in non-nearest rounding modes) and the
fp status flag settings should follow the spec, so fenv side-effects are
important and code transformations that break them should be prevented.
Unfortunately compilers don't give any help with this, the best we can
do is to add fp barriers to the code using volatile local variables
(they create a stack frame and undesirable memory accesses to it) or
inline asm (gcc specific, requires target specific fp reg constraints,
often creates unnecessary reg moves and multiple barriers are needed to
express that an operation has side-effects) or extern call (only useful
in tail-call position to avoid stack-frame creation and does not work
with lto).
We assume that in a math function if an operation depends on the input
and the output depends on it, then the operation will be evaluated at
runtime when the function is called, producing all the expected fenv
side-effects (this is not true in case of lto and in case the operation
is evaluated with excess precision that is not rounded away). So fp
barriers are needed (1) to prevent the move of an operation within a
function (in case it may be moved from an unevaluated code path into an
evaluated one or if it may be moved across a fenv access), (2) force the
evaluation of an operation for its side-effect when it has no input
dependency (may be constant folded) or (3) when its output is unused. I
belive that fp_barrier and fp_force_eval can take care of these and they
should not be needed in hot code paths.
Nothing is left from the original fdlibm header nor from the bsd
modifications to it other than some internal api declarations.
Comments are dropped that may be copyrightable content.
This makes it easier to build musl math code with a compiler that
does not support complex types (tcc) and in general more sensible
factorization of the internal headers.
the weak version of __syscall_cp_c was using a tail call to __syscall
to avoid duplicating the 6-argument syscall code inline in small
static-linked programs, but now that __syscall no longer exists, the
inline expansion is no longer duplication.
the syscall.h machinery suppported up to 7 syscall arguments, only via
an external __syscall function, but we presently have no syscall call
points that actually make use of that many, and the kernel only
defines 7-argument calling conventions for arm, powerpc (32-bit), and
sh. if it turns out we need them in the future, they can easily be
added.
this is the first part of a series of patches intended to make
__syscall fully self-contained in the object file produced using
syscall.h, which will make it possible for crt1 code to perform
syscalls.
the (confusingly named) i386 __vsyscall mechanism, which this commit
removes, was introduced before the presence of a valid thread pointer
was mandatory; back then the thread pointer was setup lazily only if
threads were used. the intent was to be able to perform syscalls using
the kernel's fast entry point in the VDSO, which can use the sysenter
(Intel) or syscall (AMD) instruction instead of int $128, but without
inlining an access to the __syscall global at the point of each
syscall, which would incur a significant size cost from PIC setup
everywhere. the mechanism also shuffled registers/calling convention
around to avoid spills of call-saved registers, and to avoid
allocating ebx or ebp via asm constraints, since there are plenty of
broken-but-supported compiler versions which are incapable of
allocating ebx with -fPIC or ebp with -fno-omit-frame-pointer.
the new mechanism preserves the properties of avoiding spills and
avoiding allocation of ebx/ebp in constraints, but does it inline,
using some fairly simple register shuffling, and uses a field of the
thread structure rather than global data for the vdso-provided syscall
code address.
for now, the external __syscall function is refactored not to use the
old __vsyscall so it can be kept, but the intent is to remove it too.
this is a workaround to avoid a crashing regression on qemu-user when
dynamic TLS is installed at dlopen time. the sigaction syscall should
not be able to fail, but it does fail for implementation-internal
signals under qemu user-level emulation if the host libc qemu is
running under reserves the same signals for implementation-internal
use, since qemu makes no provision to redirect/emulate them. after
sigaction fails, the subsequent tkill would terminate the process
abnormally as the default action.
no provision to account for membarrier failing is made in the dynamic
linker code that installs new TLS. at the formal level, the missing
barrier in this case is incorrect, and perhaps we should fail the
dlopen operation, but in practice all the archs we support (and
probably all real-world archs except alpha, which isn't yet supported)
should give the right behavior with no barrier at all as a consequence
of consume-order properties.
in the long term, this workaround should be supplemented or replaced
by something better -- a different fallback approach to ensuring
memory consistency, or dynamic allocation of implementation-internal
signals. the latter is appealing in that it would allow cancellation
to work under qemu-user too, and would even allow many levels of
nested emulation.
Mark atanhi, atanlo, and aT in atanl.c as static, as they're not
intended to be part of the public API.
These are already static in the LDBL_MANT_DIG == 64 code, so this
patch is just making the LDBL_MANT_DIG == 113 code do the same thing.
The result is the same but takes less code.
Note that __execvpe calls getenv which calls __strchrnul so even
using static output the size of the executable won't grow.
commit 54ca677983 inadvertently
introduced bitwise and where logical and was intended. since the
right-hand operand is always 0 or -1 whenever the left-hand operand is
nonzero, the behavior happened to be equivalent.
priority inheritance is a feature to mitigate priority inversion
situations, where a execution of a medium-priority thread can
unboundedly block forward progress of a high-priority thread when a
lock it needs is held by a low-priority thread.
the natural way to do priority inheritance would be with a simple
futex flag to donate the calling thread's priority to a target thread
while it waits on the futex. unfortunately, linux does not offer such
an interface, but instead insists on implementing the whole locking
protocol in kernelspace with special futex commands that exist solely
for the purpose of doing PI mutexes. this would require the entire
"trylock" logic to be duplicated in the timedlock code path for PI
mutexes, since, once the previous lock holder releases the lock and
the futex call returns, the lock is already held by the caller.
obviously such code duplication is undesirable.
instead, I've made the PI timedlock success path set the mutex lock
count to -1, which can be thought of as "not yet complete", since a
lock count of 0 is "locked, with no recursive references". a simple
branch in a non-hot path of pthread_mutex_trylock can then see and act
on this state, skipping past the code that would check and take the
lock to the same code path that runs after the lock is obtained for a
non-PI mutex.
because we're forced to let the kernel perform the actual lock and
unlock operations whenever the mutex is contended, we have to patch
things up when it does the wrong thing:
1. the lock operation is not aware of whether the mutex is
error-checking, so it will always fail with EDEADLK rather than
deadlocking.
2. the lock operation is not aware of whether the mutex is robust, so
it will successfully obtain mutexes in the owner-died state even if
they're non-robust, whereas this operation should deadlock.
3. the unlock operation always sets the lock value to zero, whereas
for robust mutexes, we want to set it to a special value indicating
that the mutex obtained after its owner died was unlocked without
marking it consistent, so that future operations all fail with
ENOTRECOVERABLE.
the first of these is easy to solve, just by performing a futex wait
on a dummy futex address to simulate deadlock or ETIMEDOUT as
appropriate. but problems 2 and 3 interact in a nasty way. to solve
problem 2, we need to back out the spurious success. but if waiters
are present -- which we can't just ignore, because even if we don't
want to wake them, the calling thread is incorrectly inheriting their
priorities -- this requires using the kernel's unlock operation, which
will zero the lock value, thereby losing the "owner died with lock
held" state.
to solve these problems, we overload the mutex's waiters field, which
is unused for PI mutexes since they don't call the normal futex wait
functions, as an indicator that the PI mutex is permanently
non-lockable. originally I wanted to use the count field, but there is
one code path that needs to access this flag without synchronization:
trylock's CAS failure path needs to be able to decide whether to fail
with EBUSY or ENOTRECOVERABLE, the waiters field is already treated as
a relaxed-order atomic in our memory model, so this works out nicely.
there was no point in masking off the pshared bit when first loading
the type, since every subsequent access involves a mask anyway. not
masking it may avoid a subsequent load to check the pshared flag, and
it's just simpler.
commit 84d061d5a3 wrongly moved the
access to the global next_key outside of the scope of the lock. the
error manifested as spurious failure to find an available key slot
under concurrent calls to pthread_key_create, since the stopping
condition could be met after only a small number of slots were
examined.
commit d6c855caa8 caused this
"regression", though the behavior was undefined before, overlooking
that f->shend=0 was being used as a sentinel for "EOF" status (actual
EOF or hitting the scanf field width) of the stream helper (shgetc)
functions.
obviously the shgetc macro could be adjusted to check for a null
pointer in addition to the != comparison, but it's the hot path, and
adding extra code/branches to it begins to defeat the purpose.
so instead of setting shend to a null pointer to block further reads,
which no longer works, set it to the current position (rpos). this
makes the shgetc macro work with no change, but it breaks shunget,
which can no longer look at the value of shend to determine whether to
back up. Szabolcs Nagy suggested a solution which I'm using here:
setting shlim to a negative value is inexpensive to test at shunget
time, and automatically re-trips the cnt>=shlim stop condition in
__shgetc no matter what the original limit was.
commit 2de29bc994 left behind one
reference to pthread_mutex_trylock. fixing this also improves code
generation due to the namespace-safe version being hidde.
The original logic considered each byte until it either found a 0
value or a value >= 192. This means if a string segment contained any
byte >= 192 it was interepretted as a compressed segment marker even
if it wasn't in a position where it should be interpretted as such.
The fix is to adjust dn_skipname to increment by each segments size
rather than look at each character. This avoids misinterpretting
string segment characters by not considering those bytes.
C11 removed the requirement that FILE be a complete type, which was
deemed erroneous, as part of the changes introduced by N1439 regarding
completeness of types (see footnote 6 for specific mention of FILE).
however the current version of POSIX is still based on C99 and
incorporates the old requirement that FILE be a complete type.
expose an arbitrary, useless complete type definition because the
actual object used to represent FILE streams cannot be public/ABI.
thanks to commit 13d1afa46f, we now have
a framework for suppressing the public complete-type definition of FILE
when stdio.h is included internally, so that a different internal
definition can be provided. this is perfectly well-defined, since the
same struct tag can refer to different types in different translation
units. it would be a problem if the implementation were accessing the
application's FILE objects or vice versa, but either would be
undefined behavior.
James Y Knight
Samuel Holland
Rich Felker
Fangrui Song
Szabolcs Nagy
Matthew Maurer
Dan Gohman
Frediano Ziglio
Drew DeVault
Ryan Fairfax
A. Wilcox