forked from luck/tmp_suning_uos_patched
46ad0840b1
As the generic rwsem-xadd code is using the appropriate acquire and release versions of the atomic operations, the arch specific rwsem.h files will not be that much faster than the generic code as long as the atomic functions are properly implemented. So we can remove those arch specific rwsem.h and stop building asm/rwsem.h to reduce maintenance effort. Currently, only x86, alpha and ia64 have implemented architecture specific fast paths. I don't have access to alpha and ia64 systems for testing, but they are legacy systems that are not likely to be updated to the latest kernel anyway. By using a rwsem microbenchmark, the total locking rates on a 4-socket 56-core 112-thread x86-64 system before and after the patch were as follows (mixed means equal # of read and write locks): Before Patch After Patch # of Threads wlock rlock mixed wlock rlock mixed ------------ ----- ----- ----- ----- ----- ----- 1 29,201 30,143 29,458 28,615 30,172 29,201 2 6,807 13,299 1,171 7,725 15,025 1,804 4 6,504 12,755 1,520 7,127 14,286 1,345 8 6,762 13,412 764 6,826 13,652 726 16 6,693 15,408 662 6,599 15,938 626 32 6,145 15,286 496 5,549 15,487 511 64 5,812 15,495 60 5,858 15,572 60 There were some run-to-run variations for the multi-thread tests. For x86-64, using the generic C code fast path seems to be a little bit faster than the assembly version with low lock contention. Looking at the assembly version of the fast paths, there are assembly to/from C code wrappers that save and restore all the callee-clobbered registers (7 registers on x86-64). The assembly generated from the generic C code doesn't need to do that. That may explain the slight performance gain here. The generic asm rwsem.h can also be merged into kernel/locking/rwsem.h with no code change as no other code other than those under kernel/locking needs to access the internal rwsem macros and functions. Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Linus Torvalds <torvalds@linux-foundation.org> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Borislav Petkov <bp@alien8.de> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Tim Chen <tim.c.chen@linux.intel.com> Cc: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: linux-c6x-dev@linux-c6x.org Cc: linux-m68k@lists.linux-m68k.org Cc: linux-riscv@lists.infradead.org Cc: linux-um@lists.infradead.org Cc: linux-xtensa@linux-xtensa.org Cc: linuxppc-dev@lists.ozlabs.org Cc: nios2-dev@lists.rocketboards.org Cc: openrisc@lists.librecores.org Cc: uclinux-h8-devel@lists.sourceforge.jp Link: https://lkml.kernel.org/r/20190322143008.21313-2-longman@redhat.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
265 lines
7.2 KiB
C
265 lines
7.2 KiB
C
/* SPDX-License-Identifier: GPL-2.0 */
|
|
/*
|
|
* The least significant 2 bits of the owner value has the following
|
|
* meanings when set.
|
|
* - RWSEM_READER_OWNED (bit 0): The rwsem is owned by readers
|
|
* - RWSEM_ANONYMOUSLY_OWNED (bit 1): The rwsem is anonymously owned,
|
|
* i.e. the owner(s) cannot be readily determined. It can be reader
|
|
* owned or the owning writer is indeterminate.
|
|
*
|
|
* When a writer acquires a rwsem, it puts its task_struct pointer
|
|
* into the owner field. It is cleared after an unlock.
|
|
*
|
|
* When a reader acquires a rwsem, it will also puts its task_struct
|
|
* pointer into the owner field with both the RWSEM_READER_OWNED and
|
|
* RWSEM_ANONYMOUSLY_OWNED bits set. On unlock, the owner field will
|
|
* largely be left untouched. So for a free or reader-owned rwsem,
|
|
* the owner value may contain information about the last reader that
|
|
* acquires the rwsem. The anonymous bit is set because that particular
|
|
* reader may or may not still own the lock.
|
|
*
|
|
* That information may be helpful in debugging cases where the system
|
|
* seems to hang on a reader owned rwsem especially if only one reader
|
|
* is involved. Ideally we would like to track all the readers that own
|
|
* a rwsem, but the overhead is simply too big.
|
|
*/
|
|
#define RWSEM_READER_OWNED (1UL << 0)
|
|
#define RWSEM_ANONYMOUSLY_OWNED (1UL << 1)
|
|
|
|
#ifdef CONFIG_DEBUG_RWSEMS
|
|
# define DEBUG_RWSEMS_WARN_ON(c) DEBUG_LOCKS_WARN_ON(c)
|
|
#else
|
|
# define DEBUG_RWSEMS_WARN_ON(c)
|
|
#endif
|
|
|
|
/*
|
|
* R/W semaphores originally for PPC using the stuff in lib/rwsem.c.
|
|
* Adapted largely from include/asm-i386/rwsem.h
|
|
* by Paul Mackerras <paulus@samba.org>.
|
|
*/
|
|
|
|
/*
|
|
* the semaphore definition
|
|
*/
|
|
#ifdef CONFIG_64BIT
|
|
# define RWSEM_ACTIVE_MASK 0xffffffffL
|
|
#else
|
|
# define RWSEM_ACTIVE_MASK 0x0000ffffL
|
|
#endif
|
|
|
|
#define RWSEM_ACTIVE_BIAS 0x00000001L
|
|
#define RWSEM_WAITING_BIAS (-RWSEM_ACTIVE_MASK-1)
|
|
#define RWSEM_ACTIVE_READ_BIAS RWSEM_ACTIVE_BIAS
|
|
#define RWSEM_ACTIVE_WRITE_BIAS (RWSEM_WAITING_BIAS + RWSEM_ACTIVE_BIAS)
|
|
|
|
#ifdef CONFIG_RWSEM_SPIN_ON_OWNER
|
|
/*
|
|
* All writes to owner are protected by WRITE_ONCE() to make sure that
|
|
* store tearing can't happen as optimistic spinners may read and use
|
|
* the owner value concurrently without lock. Read from owner, however,
|
|
* may not need READ_ONCE() as long as the pointer value is only used
|
|
* for comparison and isn't being dereferenced.
|
|
*/
|
|
static inline void rwsem_set_owner(struct rw_semaphore *sem)
|
|
{
|
|
WRITE_ONCE(sem->owner, current);
|
|
}
|
|
|
|
static inline void rwsem_clear_owner(struct rw_semaphore *sem)
|
|
{
|
|
WRITE_ONCE(sem->owner, NULL);
|
|
}
|
|
|
|
/*
|
|
* The task_struct pointer of the last owning reader will be left in
|
|
* the owner field.
|
|
*
|
|
* Note that the owner value just indicates the task has owned the rwsem
|
|
* previously, it may not be the real owner or one of the real owners
|
|
* anymore when that field is examined, so take it with a grain of salt.
|
|
*/
|
|
static inline void __rwsem_set_reader_owned(struct rw_semaphore *sem,
|
|
struct task_struct *owner)
|
|
{
|
|
unsigned long val = (unsigned long)owner | RWSEM_READER_OWNED
|
|
| RWSEM_ANONYMOUSLY_OWNED;
|
|
|
|
WRITE_ONCE(sem->owner, (struct task_struct *)val);
|
|
}
|
|
|
|
static inline void rwsem_set_reader_owned(struct rw_semaphore *sem)
|
|
{
|
|
__rwsem_set_reader_owned(sem, current);
|
|
}
|
|
|
|
/*
|
|
* Return true if the a rwsem waiter can spin on the rwsem's owner
|
|
* and steal the lock, i.e. the lock is not anonymously owned.
|
|
* N.B. !owner is considered spinnable.
|
|
*/
|
|
static inline bool is_rwsem_owner_spinnable(struct task_struct *owner)
|
|
{
|
|
return !((unsigned long)owner & RWSEM_ANONYMOUSLY_OWNED);
|
|
}
|
|
|
|
/*
|
|
* Return true if rwsem is owned by an anonymous writer or readers.
|
|
*/
|
|
static inline bool rwsem_has_anonymous_owner(struct task_struct *owner)
|
|
{
|
|
return (unsigned long)owner & RWSEM_ANONYMOUSLY_OWNED;
|
|
}
|
|
|
|
#ifdef CONFIG_DEBUG_RWSEMS
|
|
/*
|
|
* With CONFIG_DEBUG_RWSEMS configured, it will make sure that if there
|
|
* is a task pointer in owner of a reader-owned rwsem, it will be the
|
|
* real owner or one of the real owners. The only exception is when the
|
|
* unlock is done by up_read_non_owner().
|
|
*/
|
|
#define rwsem_clear_reader_owned rwsem_clear_reader_owned
|
|
static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem)
|
|
{
|
|
unsigned long val = (unsigned long)current | RWSEM_READER_OWNED
|
|
| RWSEM_ANONYMOUSLY_OWNED;
|
|
if (READ_ONCE(sem->owner) == (struct task_struct *)val)
|
|
cmpxchg_relaxed((unsigned long *)&sem->owner, val,
|
|
RWSEM_READER_OWNED | RWSEM_ANONYMOUSLY_OWNED);
|
|
}
|
|
#endif
|
|
|
|
#else
|
|
static inline void rwsem_set_owner(struct rw_semaphore *sem)
|
|
{
|
|
}
|
|
|
|
static inline void rwsem_clear_owner(struct rw_semaphore *sem)
|
|
{
|
|
}
|
|
|
|
static inline void __rwsem_set_reader_owned(struct rw_semaphore *sem,
|
|
struct task_struct *owner)
|
|
{
|
|
}
|
|
|
|
static inline void rwsem_set_reader_owned(struct rw_semaphore *sem)
|
|
{
|
|
}
|
|
#endif
|
|
|
|
#ifndef rwsem_clear_reader_owned
|
|
static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem)
|
|
{
|
|
}
|
|
#endif
|
|
|
|
#ifdef CONFIG_RWSEM_XCHGADD_ALGORITHM
|
|
/*
|
|
* lock for reading
|
|
*/
|
|
static inline void __down_read(struct rw_semaphore *sem)
|
|
{
|
|
if (unlikely(atomic_long_inc_return_acquire(&sem->count) <= 0))
|
|
rwsem_down_read_failed(sem);
|
|
}
|
|
|
|
static inline int __down_read_killable(struct rw_semaphore *sem)
|
|
{
|
|
if (unlikely(atomic_long_inc_return_acquire(&sem->count) <= 0)) {
|
|
if (IS_ERR(rwsem_down_read_failed_killable(sem)))
|
|
return -EINTR;
|
|
}
|
|
|
|
return 0;
|
|
}
|
|
|
|
static inline int __down_read_trylock(struct rw_semaphore *sem)
|
|
{
|
|
long tmp;
|
|
|
|
while ((tmp = atomic_long_read(&sem->count)) >= 0) {
|
|
if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp,
|
|
tmp + RWSEM_ACTIVE_READ_BIAS)) {
|
|
return 1;
|
|
}
|
|
}
|
|
return 0;
|
|
}
|
|
|
|
/*
|
|
* lock for writing
|
|
*/
|
|
static inline void __down_write(struct rw_semaphore *sem)
|
|
{
|
|
long tmp;
|
|
|
|
tmp = atomic_long_add_return_acquire(RWSEM_ACTIVE_WRITE_BIAS,
|
|
&sem->count);
|
|
if (unlikely(tmp != RWSEM_ACTIVE_WRITE_BIAS))
|
|
rwsem_down_write_failed(sem);
|
|
}
|
|
|
|
static inline int __down_write_killable(struct rw_semaphore *sem)
|
|
{
|
|
long tmp;
|
|
|
|
tmp = atomic_long_add_return_acquire(RWSEM_ACTIVE_WRITE_BIAS,
|
|
&sem->count);
|
|
if (unlikely(tmp != RWSEM_ACTIVE_WRITE_BIAS))
|
|
if (IS_ERR(rwsem_down_write_failed_killable(sem)))
|
|
return -EINTR;
|
|
return 0;
|
|
}
|
|
|
|
static inline int __down_write_trylock(struct rw_semaphore *sem)
|
|
{
|
|
long tmp;
|
|
|
|
tmp = atomic_long_cmpxchg_acquire(&sem->count, RWSEM_UNLOCKED_VALUE,
|
|
RWSEM_ACTIVE_WRITE_BIAS);
|
|
return tmp == RWSEM_UNLOCKED_VALUE;
|
|
}
|
|
|
|
/*
|
|
* unlock after reading
|
|
*/
|
|
static inline void __up_read(struct rw_semaphore *sem)
|
|
{
|
|
long tmp;
|
|
|
|
tmp = atomic_long_dec_return_release(&sem->count);
|
|
if (unlikely(tmp < -1 && (tmp & RWSEM_ACTIVE_MASK) == 0))
|
|
rwsem_wake(sem);
|
|
}
|
|
|
|
/*
|
|
* unlock after writing
|
|
*/
|
|
static inline void __up_write(struct rw_semaphore *sem)
|
|
{
|
|
if (unlikely(atomic_long_sub_return_release(RWSEM_ACTIVE_WRITE_BIAS,
|
|
&sem->count) < 0))
|
|
rwsem_wake(sem);
|
|
}
|
|
|
|
/*
|
|
* downgrade write lock to read lock
|
|
*/
|
|
static inline void __downgrade_write(struct rw_semaphore *sem)
|
|
{
|
|
long tmp;
|
|
|
|
/*
|
|
* When downgrading from exclusive to shared ownership,
|
|
* anything inside the write-locked region cannot leak
|
|
* into the read side. In contrast, anything in the
|
|
* read-locked region is ok to be re-ordered into the
|
|
* write side. As such, rely on RELEASE semantics.
|
|
*/
|
|
tmp = atomic_long_add_return_release(-RWSEM_WAITING_BIAS, &sem->count);
|
|
if (tmp < 0)
|
|
rwsem_downgrade_wake(sem);
|
|
}
|
|
|
|
#endif /* CONFIG_RWSEM_XCHGADD_ALGORITHM */
|