kernel_optimize_test/include/asm-generic
Mark Rutland e88d62cd4b percpu: make this_cpu_generic_read() atomic w.r.t. interrupts
As raw_cpu_generic_read() is a plain read from a raw_cpu_ptr() address,
it's possible (albeit unlikely) that the compiler will split the access
across multiple instructions.

In this_cpu_generic_read() we disable preemption but not interrupts
before calling raw_cpu_generic_read(). Thus, an interrupt could be taken
in the middle of the split load instructions. If a this_cpu_write() or
RMW this_cpu_*() op is made to the same variable in the interrupt
handling path, this_cpu_read() will return a torn value.

For native word types, we can avoid tearing using READ_ONCE(), but this
won't work in all cases (e.g. 64-bit types on most 32-bit platforms).
This patch reworks this_cpu_generic_read() to use READ_ONCE() where
possible, otherwise falling back to disabling interrupts.

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Christoph Lameter <cl@linux.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Pranith Kumar <bobby.prani@gmail.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-arch@vger.kernel.org
Cc: stable@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2017-09-26 07:37:33 -07:00
..
bitops bitops: Do not default to __clear_bit() for __clear_bit_unlock() 2016-03-21 10:50:48 +01:00
4level-fixup.h asm-generic: introduce 5level-fixup.h 2017-03-09 11:48:47 -08:00
5level-fixup.h asm-generic: introduce 5level-fixup.h 2017-03-09 11:48:47 -08:00
asm-offsets.h
asm-prototypes.h asm-prototypes: Clear any CPP defines before declaring the functions 2017-01-05 12:57:42 +01:00
atomic-long.h locking/atomic: Introduce inc/dec variants for the atomic_fetch_$op() API 2016-07-07 09:16:20 +02:00
atomic.h openrisc: add optimized atomic operations 2017-02-25 04:14:06 +09:00
atomic64.h locking/atomic: Fix atomic_set_release() for 'funny' architectures 2017-08-10 12:28:54 +02:00
audit_change_attr.h
audit_dir_write.h
audit_read.h
audit_signal.h
audit_write.h
barrier.h locking/spinlock, arch: Update and fix spin_unlock_wait() implementations 2016-06-14 11:55:15 +02:00
bitops.h
bitsperlong.h
bug.h asm-generic/bug.h: declare struct pt_regs; before function prototype 2017-07-10 16:32:34 -07:00
bugs.h
cache.h
cacheflush.h
checksum.h
clkdev.h
cmpxchg-local.h
cmpxchg.h
current.h
delay.h
device.h
div64.h
dma-contiguous.h
dma.h
early_ioremap.h x86/mm: Extend early_memremap() support with additional attrs 2017-07-18 11:38:00 +02:00
emergency-restart.h
exec.h
export.h modversions: treat symbol CRCs as 32 bit quantities 2017-02-03 08:28:25 -08:00
extable.h add asm-generic/extable.h 2017-03-27 20:27:28 -04:00
fb.h
fixmap.h
ftrace.h
futex.h futex: Remove duplicated code and fix undefined behaviour 2017-08-25 22:49:59 +02:00
getorder.h
gpio.h
hardirq.h
hugetlb.h mm/hugetlb: allow architectures to override huge_pte_clear() 2017-07-06 16:24:34 -07:00
hw_irq.h
ide_iops.h
int-ll64.h
io.h x86/io: Remove xlate_dev_kmem_ptr() duplication 2017-07-24 11:18:21 +02:00
ioctl.h
iomap.h asm-generic/io.h: add io{read,write}64 accessors 2016-05-31 16:41:50 +08:00
irq_regs.h
irq_work.h
irq.h
irqflags.h
kdebug.h
kmap_types.h
kprobes.h kprobes: move kprobe declarations to asm-generic/kprobes.h 2017-02-27 18:43:45 -08:00
kvm_para.h
linkage.h
local.h
local64.h
mcs_spinlock.h
memory_model.h
mm_hooks.h mm/gup: Drop the arch_pte_access_permitted() MMU callback 2017-03-18 09:48:01 +01:00
mm-arch-hooks.h
mmu_context.h
mmu.h
module.h
msi.h
page.h
param.h
parport.h
pci_iomap.h
pci.h
percpu.h percpu: make this_cpu_generic_read() atomic w.r.t. interrupts 2017-09-26 07:37:33 -07:00
pgalloc.h
pgtable-nop4d-hack.h asm-generic: introduce __ARCH_USE_5LEVEL_HACK 2017-03-09 11:48:47 -08:00
pgtable-nop4d.h asm-generic: introduce <asm-generic/pgtable-nop4d.h> 2017-03-09 11:48:47 -08:00
pgtable-nopmd.h
pgtable-nopud.h asm-generic: introduce <asm-generic/pgtable-nop4d.h> 2017-03-09 11:48:47 -08:00
pgtable.h mm: soft-dirty: keep soft-dirty bits over thp migration 2017-09-08 18:26:45 -07:00
preempt.h sched/preempt: Fix preempt_count manipulations 2016-05-17 12:24:21 +02:00
ptrace.h
qrwlock_types.h
qrwlock.h locking/qrwlock: Fix write unlock bug on big endian systems 2016-08-10 14:13:27 +02:00
qspinlock_types.h
qspinlock.h locking: Remove spin_unlock_wait() generic definitions 2017-08-17 08:08:58 -07:00
resource.h
rwsem.h locking/rwsem: Remove unnecessary atomic_long_t casts 2017-01-20 09:46:44 +01:00
seccomp.h seccomp: Get compat syscalls from asm-generic header 2016-05-13 14:02:00 +02:00
sections.h irq: Make the irqentry text section unconditional 2017-08-10 16:28:53 +02:00
segment.h
serial.h
set_memory.h treewide: move set_memory_* functions away from cacheflush.h 2017-05-08 17:15:13 -07:00
signal.h
simd.h
sizes.h
spinlock.h
statfs.h
string.h
switch_to.h
syscall.h
syscalls.h
termios-base.h Replace <asm/uaccess.h> with <linux/uaccess.h> globally 2016-12-24 11:46:01 -08:00
termios.h Replace <asm/uaccess.h> with <linux/uaccess.h> globally 2016-12-24 11:46:01 -08:00
timex.h
tlb.h mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem 2017-08-10 15:54:07 -07:00
tlbflush.h
topology.h cpumask: fix spurious cpumask_of_node() on non-NUMA multi-node configs 2017-08-28 16:13:16 -07:00
trace_clock.h
uaccess.h fix the __user misannotations in asm-generic get_user/put_user 2017-09-04 12:07:24 -04:00
unaligned.h
unistd.h
user.h
vga.h
vmlinux.lds.h MTD changes for 4.14: 2017-09-09 14:48:21 -07:00
vtime.h
word-at-a-time.h
xor.h