kernel_optimize_test/block
Tejun Heo ea04a3b572 blk-iocost: fix operation ordering in iocg_wake_fn()
commit 5ab189cf3abbc9994bae3be524c5b88589ed56e2 upstream.

iocg_wake_fn() open-codes wait_queue_entry removal and wakeup because it
wants the wq_entry to be always removed whether it ended up waking the
task or not. finish_wait() tests whether wq_entry needs removal without
grabbing the wait_queue lock and expects the waker to use
list_del_init_careful() after all waking operations are complete, which
iocg_wake_fn() didn't do. The operation order was wrong and the regular
list_del_init() was used.

The result is that if a waiter wakes up racing the waker, it can free pop
the wq_entry off stack before the waker is still looking at it, which can
lead to a backtrace like the following.

  [7312084.588951] general protection fault, probably for non-canonical address 0x586bf4005b2b88: 0000 [#1] SMP
  ...
  [7312084.647079] RIP: 0010:queued_spin_lock_slowpath+0x171/0x1b0
  ...
  [7312084.858314] Call Trace:
  [7312084.863548]  _raw_spin_lock_irqsave+0x22/0x30
  [7312084.872605]  try_to_wake_up+0x4c/0x4f0
  [7312084.880444]  iocg_wake_fn+0x71/0x80
  [7312084.887763]  __wake_up_common+0x71/0x140
  [7312084.895951]  iocg_kick_waitq+0xe8/0x2b0
  [7312084.903964]  ioc_rqos_throttle+0x275/0x650
  [7312084.922423]  __rq_qos_throttle+0x20/0x30
  [7312084.930608]  blk_mq_make_request+0x120/0x650
  [7312084.939490]  generic_make_request+0xca/0x310
  [7312084.957600]  submit_bio+0x173/0x200
  [7312084.981806]  swap_readpage+0x15c/0x240
  [7312084.989646]  read_swap_cache_async+0x58/0x60
  [7312084.998527]  swap_cluster_readahead+0x201/0x320
  [7312085.023432]  swapin_readahead+0x2df/0x450
  [7312085.040672]  do_swap_page+0x52f/0x820
  [7312085.058259]  handle_mm_fault+0xa16/0x1420
  [7312085.066620]  do_page_fault+0x2c6/0x5c0
  [7312085.074459]  page_fault+0x2f/0x40

Fix it by switching to list_del_init_careful() and putting it at the end.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Rik van Riel <riel@surriel.com>
Fixes: 7caa47151a ("blkcg: implement blk-iocost")
Cc: stable@vger.kernel.org # v5.4+
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2021-08-04 12:46:41 +02:00
..
partitions partitions: msdos: fix one-byte get_unaligned() 2021-07-20 16:05:39 +02:00
badblocks.c treewide: Use fallthrough pseudo-keyword 2020-08-23 17:36:59 -05:00
bfq-cgroup.c bfq: fix blkio cgroup leakage v4 2020-08-18 07:48:08 -07:00
bfq-iosched.c kyber: fix out of bounds access when preempted 2021-05-19 10:13:13 +02:00
bfq-iosched.h bfq: fix blkio cgroup leakage v4 2020-08-18 07:48:08 -07:00
bfq-wf2q.c bfq: fix blkio cgroup leakage v4 2020-08-18 07:48:08 -07:00
bio-integrity.c block: make function __bio_integrity_free() static 2020-07-02 12:38:18 -06:00
bio.c block: only update parent bi_status when bio fail 2021-04-16 11:43:21 +02:00
blk-cgroup-rwstat.c blk-cgroup: Fix the recursive blkg rwstat 2021-03-30 14:31:48 +02:00
blk-cgroup-rwstat.h blk-cgroup: separate out blkg_rwstat under CONFIG_BLK_CGROUP_RWSTAT 2019-11-07 12:28:13 -07:00
blk-cgroup.c blk-cgroup: Use cond_resched() when destroy blkgs 2021-02-13 13:55:13 +01:00
blk-core.c scsi: block: Do not accept any requests while suspended 2021-01-12 20:18:17 +01:00
blk-crypto-fallback.c block: rename generic_make_request to submit_bio_noacct 2020-07-01 07:27:24 -06:00
blk-crypto-internal.h block: make blk_crypto_rq_bio_prep() able to fail 2020-10-05 10:47:43 -06:00
blk-crypto.c block: warn if !__GFP_DIRECT_RECLAIM in bio_crypt_set_ctx() 2020-10-05 10:47:43 -06:00
blk-exec.c block: add a blk_account_io_merge_bio helper 2020-05-27 05:21:23 -06:00
blk-flush.c block: avoid double io accounting for flush request 2021-07-14 16:56:11 +02:00
blk-integrity.c block: remove the unused blk_integrity_merge_bio export 2020-10-06 07:29:53 -06:00
blk-ioc.c block: remove retry loop in ioc_release_fn() 2020-07-16 10:22:15 -06:00
blk-iocost.c blk-iocost: fix operation ordering in iocg_wake_fn() 2021-08-04 12:46:41 +02:00
blk-iolatency.c block: Remove redundant 'return' statement 2020-10-08 07:59:48 -06:00
blk-lib.c block: add a bdev_is_partition helper 2020-09-25 08:18:57 -06:00
blk-map.c block: fix bmd->is_null_mapped initialization 2020-09-23 09:18:39 -06:00
blk-merge.c block: fix discard request merge 2021-07-14 16:56:02 +02:00
blk-mq-cpumap.c blk-mq: remove the calling of local_memory_node() 2020-10-20 07:08:17 -06:00
blk-mq-debugfs-zoned.c
blk-mq-debugfs.c blk-mq-debugfs: Add decode for BLK_MQ_F_TAG_HCTX_SHARED 2021-01-19 18:27:29 +01:00
blk-mq-debugfs.h
blk-mq-pci.c
blk-mq-rdma.c
blk-mq-sched.c kyber: fix out of bounds access when preempted 2021-05-19 10:13:13 +02:00
blk-mq-sched.h block-5.10-2020-10-12 2020-10-13 12:12:44 -07:00
blk-mq-sysfs.c blk-mq: move cancel of hctx->run_work to the front of blk_exit_queue 2020-10-09 12:46:28 -06:00
blk-mq-tag.c blk-mq: clear stale request in tags->rq[] before freeing one request pool 2021-07-14 16:55:58 +02:00
blk-mq-tag.h blk-mq: clear stale request in tags->rq[] before freeing one request pool 2021-07-14 16:55:58 +02:00
blk-mq-virtio.c blk-mq: Fix typo in comment 2020-03-17 20:55:21 +01:00
blk-mq.c blk-mq: update hctx->dispatch_busy in case of real scheduler 2021-07-14 16:56:13 +02:00
blk-mq.h blk-mq: grab rq->refcount before calling ->fn in blk_mq_tagset_busy_iter 2021-07-14 16:55:58 +02:00
blk-pm.c scsi: block: Fix a race in the runtime power management code 2021-01-06 14:56:50 +01:00
blk-pm.h scsi: block: Do not accept any requests while suspended 2021-01-12 20:18:17 +01:00
blk-rq-qos.c rq-qos: fix missed wake-ups in rq_qos_throttle try two 2021-07-19 09:45:00 +02:00
blk-rq-qos.h block: fix race between adding/removing rq qos and normal IO 2021-07-14 16:56:00 +02:00
blk-settings.c blk-settings: align max_sectors on "logical_block_size" boundary 2021-03-04 11:38:22 +01:00
blk-stat.c blk-stat: make q->stats->lock irqsafe 2020-09-01 16:48:46 -06:00
blk-stat.h
blk-sysfs.c blk-mq: move cancel of hctx->run_work to the front of blk_exit_queue 2020-10-09 12:46:28 -06:00
blk-throttle.c blk-throttle: Re-use the throtl_set_slice_end() 2020-10-08 08:01:38 -06:00
blk-timeout.c block: blk-timeout: delete duplicated word 2020-07-31 16:29:47 -06:00
blk-wbt.c blk-wbt: make sure throttle is enabled properly 2021-07-14 16:56:12 +02:00
blk-wbt.h blk-wbt: introduce a new disable state to prevent false positive by rwb_enabled() 2021-07-14 16:56:12 +02:00
blk-zoned.c block: Fix REQ_OP_ZONE_RESET_ALL handling 2021-03-30 14:31:51 +02:00
blk.h block: move blk_mq_sched_try_merge to blk-merge.c 2020-10-06 07:29:53 -06:00
bounce.c block: make bio_crypt_clone() able to fail 2020-10-05 10:47:43 -06:00
bsg-lib.c block: drop double zeroing 2020-09-23 09:18:13 -06:00
bsg.c bsg: free the request before return error code 2021-03-04 11:37:42 +01:00
cmdline-parser.c
elevator.c block: fix comment and add lockdep assert 2020-10-09 12:34:06 -06:00
genhd.c block: Suppress uevent for hidden device when removed 2021-03-30 14:31:52 +02:00
ioctl.c block: return -EBUSY when there are open partitions in blkdev_reread_part 2021-04-28 13:39:59 +02:00
ioprio.c block: grant IOPRIO_CLASS_RT to CAP_SYS_NICE 2020-09-01 19:38:33 -06:00
Kconfig blk-wbt: Remove obsolete multiqueue I/O scheduling comment 2020-09-01 16:49:26 -06:00
Kconfig.iosched treewide: replace '---help---' in Kconfig files with 'help' 2020-06-14 01:57:21 +09:00
keyslot-manager.c block/keyslot-manager: prevent crash when num_slots=1 2020-11-20 11:52:52 -07:00
kyber-iosched.c kyber: fix out of bounds access when preempted 2021-05-19 10:13:13 +02:00
Makefile blk-mq: merge blk-softirq.c into blk-mq.c 2020-06-24 09:15:56 -06:00
mq-deadline.c kyber: fix out of bounds access when preempted 2021-05-19 10:13:13 +02:00
opal_proto.h block: sed-opal: Change the check condition for regular session validity 2020-03-12 08:00:10 -06:00
scsi_ioctl.c drivers-5.10-2020-10-12 2020-10-13 13:04:41 -07:00
sed-opal.c block: sed-opal: Change the check condition for regular session validity 2020-03-12 08:00:10 -06:00
t10-pi.c block: Allow t10-pi to be modular 2020-01-06 20:59:04 -07:00