kernel_optimize_test/kernel/locking
Will Deacon d133166146 locking/qrwlock: Prevent slowpath writers getting held up by fastpath
When a prospective writer takes the qrwlock locking slowpath due to the
lock being held, it attempts to cmpxchg the wmode field from 0 to
_QW_WAITING so that concurrent lockers also take the slowpath and queue
on the spinlock accordingly, allowing the lockers to drain.

Unfortunately, this isn't fair, because a fastpath writer that comes in
after the lock is made available but before the _QW_WAITING flag is set
can effectively jump the queue. If there is a steady stream of prospective
writers, then the waiter will be held off indefinitely.

This patch restores fairness by separating _QW_WAITING and _QW_LOCKED
into two distinct fields: _QW_LOCKED continues to occupy the bottom byte
of the lockword so that it can be cleared unconditionally when unlocking,
but _QW_WAITING now occupies what used to be the bottom bit of the reader
count. This then forces the slow-path for concurrent lockers.

Tested-by: Waiman Long <longman@redhat.com>
Tested-by: Jeremy Linton <jeremy.linton@arm.com>
Tested-by: Adam Wallis <awallis@codeaurora.org>
Tested-by: Jan Glauber <jglauber@cavium.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Acked-by: Peter Zijlstra <peterz@infradead.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Jeremy.Linton@arm.com
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-arm-kernel@lists.infradead.org
Link: http://lkml.kernel.org/r/1507810851-306-6-git-send-email-will.deacon@arm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-10-25 10:57:25 +02:00
..
lockdep_internals.h locking/lockdep: Avoid creating redundant links 2017-08-10 12:29:04 +02:00
lockdep_proc.c locking/lockdep: Avoid creating redundant links 2017-08-10 12:29:04 +02:00
lockdep_states.h locking/lockdep: Rework FS_RECLAIM annotation 2017-08-10 12:29:03 +02:00
lockdep.c locking/lockdep: Fix stacktrace mess 2017-10-10 10:04:28 +02:00
locktorture.c sched/headers: Prepare for the removal of <linux/rtmutex.h> from <linux/sched.h> 2017-03-02 08:42:32 +01:00
Makefile locking/ww_mutex: Begin kselftests for ww_mutex 2017-01-14 11:37:14 +01:00
mcs_spinlock.h locking/core: Remove cpu_relax_lowlatency() users 2016-11-16 10:15:10 +01:00
mutex-debug.c locking/mutex: Rework mutex::owner 2016-10-25 11:31:50 +02:00
mutex-debug.h locking/mutex: Fix lockdep_assert_held() fail 2017-01-30 11:42:59 +01:00
mutex.c mutex, futex: adjust kernel-doc markups to generate ReST 2017-05-16 08:43:25 -03:00
mutex.h locking/mutex: Fix lockdep_assert_held() fail 2017-01-30 11:42:59 +01:00
osq_lock.c locking/osq_lock: Fix osq_lock queue corruption 2017-08-10 12:28:54 +02:00
percpu-rwsem.c locking/percpu-rwsem: Replace waitqueue with rcuwait 2017-01-14 11:14:35 +01:00
qrwlock.c locking/qrwlock: Prevent slowpath writers getting held up by fastpath 2017-10-25 10:57:25 +02:00
qspinlock_paravirt.h locking/pvqspinlock: Relax cmpxchg's to improve performance on some architectures 2017-08-29 15:14:38 +02:00
qspinlock_stat.h sched/headers: Prepare for new header dependencies before moving code to <linux/sched/clock.h> 2017-03-02 08:42:27 +01:00
qspinlock.c locking: Remove spin_unlock_wait() generic definitions 2017-08-17 08:08:58 -07:00
rtmutex_common.h locking/rtmutex: replace top-waiter and pi_waiters leftmost caching 2017-09-08 18:26:49 -07:00
rtmutex-debug.c locking/rtmutex: replace top-waiter and pi_waiters leftmost caching 2017-09-08 18:26:49 -07:00
rtmutex-debug.h rt_mutex: Add lockdep annotations 2017-06-08 10:35:49 +02:00
rtmutex.c locking/rtmutex: replace top-waiter and pi_waiters leftmost caching 2017-09-08 18:26:49 -07:00
rtmutex.h rt_mutex: Add lockdep annotations 2017-06-08 10:35:49 +02:00
rwsem-spinlock.c locking/rwsem-spinlock: Add killable versions of __down_read() 2017-08-10 12:28:55 +02:00
rwsem-xadd.c locking/rwsem-xadd: Fix missed wakeup due to reordering of load 2017-09-29 10:10:20 +02:00
rwsem.c locking/rwsem: Add down_read_killable() 2017-10-10 11:50:16 +02:00
rwsem.h locking/rwsem: Protect all writes to owner by WRITE_ONCE() 2016-06-08 15:16:59 +02:00
semaphore.c sched/headers: Prepare for new header dependencies before moving code to <linux/sched/debug.h> 2017-03-02 08:42:34 +01:00
spinlock_debug.c locking/spinlock/debug: Remove spinlock lockup detection code 2017-02-10 09:09:49 +01:00
spinlock.c locking/core: Remove {read,spin,write}_can_lock() 2017-10-10 11:50:18 +02:00
test-ww_mutex.c mm: treewide: remove GFP_TEMPORARY allocation flag 2017-09-13 18:53:16 -07:00