kernel_optimize_test/kernel/sched
Peter Oskolkov 2a36ab717e rseq/membarrier: Add MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ
This patchset is based on Google-internal RSEQ work done by Paul
Turner and Andrew Hunter.

When working with per-CPU RSEQ-based memory allocations, it is
sometimes important to make sure that a global memory location is no
longer accessed from RSEQ critical sections. For example, there can be
two per-CPU lists, one is "active" and accessed per-CPU, while another
one is inactive and worked on asynchronously "off CPU" (e.g.  garbage
collection is performed). Then at some point the two lists are
swapped, and a fast RCU-like mechanism is required to make sure that
the previously active list is no longer accessed.

This patch introduces such a mechanism: in short, membarrier() syscall
issues an IPI to a CPU, restarting a potentially active RSEQ critical
section on the CPU.

Signed-off-by: Peter Oskolkov <posk@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Link: https://lkml.kernel.org/r/20200923233618.2572849-1-posk@google.com
2020-09-25 14:23:27 +02:00
..
autogroup.c
autogroup.h
clock.c
completion.c
core.c sched: Remove unused inline function uclamp_bucket_base_value() 2020-09-25 14:23:25 +02:00
cpuacct.c
cpudeadline.c sched/deadline: Implement fallback mechanism for !fit case 2020-06-15 14:10:05 +02:00
cpudeadline.h
cpufreq_schedutil.c Power management updates for 5.9-rc1 2020-08-03 20:28:08 -07:00
cpufreq.c
cpupri.c
cpupri.h
cputime.c sched/cputime: Improve cputime_adjust() 2020-06-15 14:10:00 +02:00
deadline.c sched/deadline: Fix stale throttling on de-/boosted tasks 2020-09-25 14:23:24 +02:00
debug.c sched/topology: Move sd_flag_debug out of #ifdef CONFIG_SYSCTL 2020-09-09 10:09:03 +02:00
fair.c sched/fair: Use dst group while checking imbalance for NUMA balancer 2020-09-25 14:23:26 +02:00
features.h sched/rt: Disable RT_RUNTIME_SHARE by default 2020-09-25 14:23:24 +02:00
idle.c Merge branch 'sched/urgent' 2020-07-08 11:38:59 +02:00
isolation.c isolcpus: Affine unbound kernel threads to housekeeping cpus 2020-06-15 14:10:03 +02:00
loadavg.c sched: nohz: stop passing around unused "ticks" parameter. 2020-07-22 10:22:04 +02:00
Makefile
membarrier.c rseq/membarrier: Add MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ 2020-09-25 14:23:27 +02:00
pelt.c sched: Add a tracepoint to track rq->nr_running 2020-07-08 11:39:02 +02:00
pelt.h sched/pelt: Cleanup PELT divider 2020-06-15 14:10:06 +02:00
psi.c sched,psi: Convert to sched_set_fifo_low() 2020-06-15 14:10:25 +02:00
rt.c sched: Remove struct sched_class::next field 2020-06-25 13:45:44 +02:00
sched-pelt.h
sched.h sched: Fix use of count for nr_running tracepoint 2020-08-06 09:36:59 +02:00
smp.h
stats.c
stats.h
stop_task.c sched: Remove struct sched_class::next field 2020-06-25 13:45:44 +02:00
swait.c
topology.c sched/fair: Reduce busy load balance interval 2020-09-25 14:23:26 +02:00
wait_bit.c
wait.c list: add "list_del_init_careful()" to go with "list_empty_careful()" 2020-08-02 20:39:44 -07:00