Waiman Long c77c0a8ac4 mm/hugetlb: defer freeing of huge pages if in non-task context ... The following lockdep splat was observed when a certain hugetlbfs test was run: ================================ WARNING: inconsistent lock state 4.18.0-159.el8.x86_64+debug #1 Tainted: G W --------- - - -------------------------------- inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-W} usage. swapper/30/0 [HC0[0]:SC1[1]:HE1:SE0] takes: ffffffff9acdc038 (hugetlb_lock){+.?.}, at: free_huge_page+0x36f/0xaa0 {SOFTIRQ-ON-W} state was registered at: lock_acquire+0x14f/0x3b0 _raw_spin_lock+0x30/0x70 __nr_hugepages_store_common+0x11b/0xb30 hugetlb_sysctl_handler_common+0x209/0x2d0 proc_sys_call_handler+0x37f/0x450 vfs_write+0x157/0x460 ksys_write+0xb8/0x170 do_syscall_64+0xa5/0x4d0 entry_SYSCALL_64_after_hwframe+0x6a/0xdf irq event stamp: 691296 hardirqs last enabled at (691296): [<ffffffff99bb034b>] _raw_spin_unlock_irqrestore+0x4b/0x60 hardirqs last disabled at (691295): [<ffffffff99bb0ad2>] _raw_spin_lock_irqsave+0x22/0x81 softirqs last enabled at (691284): [<ffffffff97ff0c63>] irq_enter+0xc3/0xe0 softirqs last disabled at (691285): [<ffffffff97ff0ebe>] irq_exit+0x23e/0x2b0 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(hugetlb_lock); <Interrupt> lock(hugetlb_lock); *** DEADLOCK *** : Call Trace: <IRQ> __lock_acquire+0x146b/0x48c0 lock_acquire+0x14f/0x3b0 _raw_spin_lock+0x30/0x70 free_huge_page+0x36f/0xaa0 bio_check_pages_dirty+0x2fc/0x5c0 clone_endio+0x17f/0x670 [dm_mod] blk_update_request+0x276/0xe50 scsi_end_request+0x7b/0x6a0 scsi_io_completion+0x1c6/0x1570 blk_done_softirq+0x22e/0x350 __do_softirq+0x23d/0xad8 irq_exit+0x23e/0x2b0 do_IRQ+0x11a/0x200 common_interrupt+0xf/0xf </IRQ> Both the hugetbl_lock and the subpool lock can be acquired in free_huge_page(). One way to solve the problem is to make both locks irq-safe. However, Mike Kravetz had learned that the hugetlb_lock is held for a linear scan of ALL hugetlb pages during a cgroup reparentling operation. So it is just too long to have irq disabled unless we can break hugetbl_lock down into finer-grained locks with shorter lock hold times. Another alternative is to defer the freeing to a workqueue job. This patch implements the deferred freeing by adding a free_hpage_workfn() work function to do the actual freeing. The free_huge_page() call in a non-task context saves the page to be freed in the hpage_freelist linked list in a lockless manner using the llist APIs. The generic workqueue is used to process the work, but a dedicated workqueue can be used instead if it is desirable to have the huge page freed ASAP. Thanks to Kirill Tkhai <ktkhai@virtuozzo.com> for suggesting the use of llist APIs which simplfy the code. Link: http://lkml.kernel.org/r/20191217170331.30893-1-longman@redhat.com Signed-off-by: Waiman Long <longman@redhat.com> Reviewed-by: Mike Kravetz <mike.kravetz@oracle.com> Acked-by: Davidlohr Bueso <dbueso@suse.de> Acked-by: Michal Hocko <mhocko@suse.com> Reviewed-by: Kirill Tkhai <ktkhai@virtuozzo.com> Cc: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Andi Kleen <ak@linux.intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>		2020-01-04 13:55:09 -08:00
..
kasan	kasan: use apply_to_existing_page_range() for releasing vmalloc shadow	2019-12-17 20:59:59 -08:00
backing-dev.c
balloon_compaction.c
cleancache.c
cma_debug.c	mm/cma_debug.c: use DEFINE_DEBUGFS_ATTRIBUTE to define debugfs fops	2019-12-01 12:59:09 -08:00
cma.c	mm/cma.c: switch to bitmap_zalloc() for cma bitmap allocation	2019-12-01 12:59:09 -08:00
cma.h
compaction.c
debug_page_ref.c
debug.c
dmapool.c
early_ioremap.c
fadvise.c
failslab.c
filemap.c	mm: drop mmap_sem before calling balance_dirty_pages() in write fault	2019-12-01 06:29:18 -08:00
frame_vector.c
frontswap.c
gup_benchmark.c	mm/gup: fix memory leak in __gup_benchmark_ioctl	2020-01-04 13:55:09 -08:00
gup.c	mm/gup.c: fix comments of __get_user_pages() and get_user_pages_remote()	2019-12-01 06:29:18 -08:00
highmem.c
hmm.c
huge_memory.c	mm/huge_memory.c: split_huge_pages_fops should be defined with DEFINE_DEBUGFS_ATTRIBUTE	2019-12-01 12:59:09 -08:00
hugetlb_cgroup.c
hugetlb.c	mm/hugetlb: defer freeing of huge pages if in non-task context	2020-01-04 13:55:09 -08:00
hwpoison-inject.c	mm/hwpoison-inject: use DEFINE_DEBUGFS_ATTRIBUTE to define debugfs fops	2019-12-01 12:59:09 -08:00
init-mm.c
internal.h	mm, pcpu: make zone pcp updates and reset internal to the mm	2019-12-01 12:59:06 -08:00
interval_tree.c
Kconfig	mm/Kconfig: fix trivial help text punctuation	2019-12-01 12:59:10 -08:00
Kconfig.debug
khugepaged.c	mm/thp: flush file for !is_shmem PageDirty() case in collapse_file()	2019-12-01 12:59:09 -08:00
kmemleak-test.c
kmemleak.c
ksm.c	* PPC secure guest support	2019-12-04 11:08:30 -08:00
list_lru.c
maccess.c
madvise.c	mm/madvise.c: use PAGE_ALIGN[ED] for range checking	2019-12-01 12:59:09 -08:00
Makefile
mapping_dirty_helpers.c
memblock.c	mm: support memblock alloc on the exact node for sparse_buffer_init()	2019-12-01 12:59:08 -08:00
memcontrol.c	mm/memcontrol: use vmstat names for printing statistics	2019-12-04 19:44:11 -08:00
memfd.c
memory_hotplug.c	mm/memory_hotplug: shrink zones when offlining memory	2020-01-04 13:55:08 -08:00
memory-failure.c	mm/memory-failure.c: use page_shift() in add_to_kill()	2019-12-01 12:59:04 -08:00
memory.c	mm/memory.c: add apply_to_existing_page_range() helper	2019-12-17 20:59:59 -08:00
mempolicy.c	mm/mempolicy.c: fix checking unmapped holes for mbind	2019-12-01 12:59:07 -08:00
mempool.c
memremap.c	mm/memory_hotplug: shrink zones when offlining memory	2020-01-04 13:55:08 -08:00
memtest.c
migrate.c	mm: move_pages: return valid node id in status if the page is already on the target node	2020-01-04 13:55:09 -08:00
mincore.c
mlock.c
mm_init.c
mmap.c	mm/mmap.c: make vma_merge() comment more easy to understand	2019-12-01 12:59:09 -08:00
mmu_context.c
mmu_gather.c
mmu_notifier.c	hmm related patches for 5.5	2019-11-30 10:33:14 -08:00
mmzone.c
mprotect.c	autonuma: reduce cache footprint when scanning page tables	2019-12-01 12:59:09 -08:00
mremap.c	mm/mmap.c: use IS_ERR_VALUE to check return value of get_unmapped_area	2019-12-01 06:29:19 -08:00
msync.c
nommu.c	mm/mmap.c: rb_parent is not necessary in __vma_link_list()	2019-12-01 06:29:19 -08:00
oom_kill.c	mm/oom: fix pgtables units mismatch in Killed process message	2020-01-04 13:55:09 -08:00
page_alloc.c	mm: clean up and clarify lruvec lookup procedure	2019-12-01 12:59:06 -08:00
page_counter.c
page_ext.c
page_idle.c
page_io.c	mm/page_io.c: annotate refault stalls from swap_readpage	2019-12-01 12:59:11 -08:00
page_isolation.c	mm/page_isolation.c: convert SKIP_HWPOISON to MEMORY_OFFLINE	2019-12-01 12:59:04 -08:00
page_owner.c
page_poison.c
page_vma_mapped.c
page-writeback.c
pagewalk.c
percpu-internal.h
percpu-km.c
percpu-stats.c
percpu-vm.c
percpu.c
pgtable-generic.c	asm-generic/mm: stub out p{4,u}d_clear_bad() if __PAGETABLE_P{4,U}D_FOLDED	2019-12-01 06:29:19 -08:00
process_vm_access.c
readahead.c
rmap.c	mm, thp: do not queue fully unmapped pages for deferred split	2019-12-01 12:59:09 -08:00
rodata_test.c
shmem.c	mm/shmem.c: cast the type of unmap_start to u64	2019-12-01 12:59:10 -08:00
shuffle.c
shuffle.h
slab_common.c	mm: memcg/slab: wait for !root kmem_cache refcnt killing on root kmem_cache destruction	2019-12-04 19:44:11 -08:00
slab.c	mm, slab: remove unused kmalloc_size()	2019-12-01 06:29:17 -08:00
slab.h	mm: clean up and clarify lruvec lookup procedure	2019-12-01 12:59:06 -08:00
slob.c
slub.c	mm/slub.c: clean up validate_slab()	2019-12-01 06:29:18 -08:00
sparse-vmemmap.c
sparse.c	mm: support memblock alloc on the exact node for sparse_buffer_init()	2019-12-01 12:59:08 -08:00
swap_cgroup.c
swap_slots.c
swap_state.c
swap.c	mm/swap.c: piggyback lru_add_drain_all() calls	2019-12-01 06:29:19 -08:00
swapfile.c	mm, swap: disallow swapon() on zoned block devices	2019-12-01 06:29:18 -08:00
truncate.c
usercopy.c
userfaultfd.c	mm: fix typos in comments when calling __SetPageUptodate()	2019-12-01 12:59:10 -08:00
util.c	mm/mmap.c: rb_parent is not necessary in __vma_link_list()	2019-12-01 06:29:19 -08:00
vmacache.c
vmalloc.c	kasan: don't assume percpu shadow allocations will succeed	2019-12-17 20:59:59 -08:00
vmpressure.c
vmscan.c	mm: vmscan: protect shrinker idr replace with CONFIG_MEMCG	2019-12-17 20:59:59 -08:00
vmstat.c	mm/memcontrol: use vmstat names for printing statistics	2019-12-04 19:44:11 -08:00
workingset.c	mm: vmscan: detect file thrashing at the reclaim root	2019-12-01 12:59:07 -08:00
z3fold.c	mm/z3fold.c: add inter-page compaction	2019-12-01 12:59:07 -08:00
zbud.c
zpool.c
zsmalloc.c	mm/zsmalloc.c: fix the migrated zspage statistics.	2020-01-04 13:55:09 -08:00
zswap.c