Kernel: Deadlock in SysFS when SMP is enabled

[LucasChollet]

We still have many issues when SMP is enabled, but this was the first time I saw this specific deadlock, so it seemed noteworthy.

It is probably quite hard to reproduce, I only hit this bug once even if I used these steps many times (I was trying to repro another bug):

• Boot serenity with smp=on
• Not sure if this is relevant, but I used 8 cores
• I had at least one terminal doing loop { /usr/Tests/Kernel/TestKernelAlarm }
• And at least one instance of SystemMonitor running

After the deadlock, I got the backtraces with GDB:

Stack Traces

Thread 8 (Thread 1.8 (CPU#7 [running])): #0 Kernel::ProcessorBase::wait_check () at ././Kernel/Arch/x86_64/Processor.h:268 #1 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:79 #2 Kernel::SpinlockLocker >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135 #3 0x000000200072cd1f in Kernel::Scheduler::pick_next () at ./Kernel/Tasks/Scheduler.cpp:221 #4 0x000000200072cf15 in Kernel::Scheduler::yield () at ./Kernel/Tasks/Scheduler.cpp:274 #5 0x000000200072d095 in Kernel::Scheduler::idle_loop () at ./Kernel/Tasks/Scheduler.cpp:521 #6 0x0000000000000000 in ?? ()

Thread 7 (Thread 1.7 (CPU#6 [running])):
#0 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:76
#1 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135
#2 0x000000200072cd1f in Kernel::Scheduler::pick_next () at ./Kernel/Tasks/Scheduler.cpp:221
#3 0x000000200072cf15 in Kernel::Scheduler::yield () at ./Kernel/Tasks/Scheduler.cpp:274
#4 0x000000200072d095 in Kernel::Scheduler::idle_loop () at ./Kernel/Tasks/Scheduler.cpp:521
#5 0x0000000000000000 in ?? ()

Thread 6 (Thread 1.6 (CPU#5 [running])):
#0 Kernel::Processor::smp_process_pending_messages () at ./Kernel/Arch/x86_64/Processor.cpp:970
#1 0x00000020002037f1 in Kernel::ProcessorBase::wait_check () at ././Kernel/Arch/x86_64/Processor.h:269
#2 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:79
#3 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135
#4 0x000000200072cd1f in Kernel::Scheduler::pick_next () at ./Kernel/Tasks/Scheduler.cpp:221
#5 0x000000200072cf15 in Kernel::Scheduler::yield () at ./Kernel/Tasks/Scheduler.cpp:274
#6 0x000000200072d095 in Kernel::Scheduler::idle_loop () at ./Kernel/Tasks/Scheduler.cpp:521
#7 0x0000000000000000 in ?? ()

Thread 5 (Thread 1.5 (CPU#4 [running])):
#0 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:76
#1 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135
#2 0x00000020007388ff in Kernel::Thread::block_impl () at ./Kernel/Tasks/Thread.cpp:116
#3 0x0000002000739945 in Kernel::Thread::block<Kernel::Thread::SleepBlocker, Kernel::Thread::BlockTimeout, AK::Duration*&> () at ././Kernel/Tasks/Thread.h:824
#4 Kernel::Thread::sleep () at ./Kernel/Tasks/Thread.cpp:481
#5 0x00000020005dccc8 in Kernel::Thread::sleep () at ././Kernel/Tasks/Thread.h:841
#6 Kernel::MDIO::Clause22::Interface::mdio_handling_thread () at ./Kernel/Net/MDIO.cpp:98
#7 0x00000020005dd1e1 in operator() () at ./Kernel/Net/MDIO.cpp:22
#8 kernel_process_trampoline<Kernel::MDIO::Clause22::Interface::spawn_mdio_handling_task(u8)::<lambda()> >(void) () at ././Kernel/Tasks/Process.h:199
#9 0x0000000000000000 in ?? ()

Thread 4 (Thread 1.4 (CPU#3 [running])):
#0 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:76
#1 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135
#2 0x00000020007404fe in Kernel::Thread::check_dispatch_pending_signal () at ./Kernel/Tasks/Thread.cpp:637
#3 0x00000020006b546f in syscall_handler () at ./Kernel/Syscalls/SyscallHandler.cpp:173
#4 0x0000002000795b14 in syscall_entry () at ./Kernel/Arch/x86_64/SyscallEntry.cpp:25
#5 0x0000000000000000 in ?? ()

Thread 3 (Thread 1.3 (CPU#2 [running])):
#0 Kernel::InterruptDisabler::InterruptDisabler () at ././Kernel/Interrupts/InterruptDisabler.h:17
#1 Kernel::ProcessorBase::leave_critical () at ./Kernel/Arch/Processor.cpp:91
#2 0x00000020007927c5 in Kernel::Processor::smp_process_pending_messages () at ./Kernel/Arch/x86_64/Processor.cpp:1032
#3 0x00000020004cbdc1 in Kernel::ProcessorBase::wait_check () at ././Kernel/Arch/x86_64/Processor.h:269
#4 Kernel::RecursiveSpinlock<(Kernel::LockRank)8>::lock () at ././Kernel/Locking/Spinlock.h:79
#5 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)8> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135
#6 operator() () at ./Kernel/FileSystem/SysFS/Subsystems/Kernel/Processes.cpp:117
#7 0x00000020004d1ee1 in call () at ././AK/Function.h:201
#8 AK::Function<AK::ErrorOr<void, AK::Error>(Kernel::Thread const&)>::operator() () at ././AK/Function.h:135
#9 Kernel::Process::try_for_each_thread(AK::Function<AK::ErrorOr<void, AK::Error> (Kernel::Thread const&)>) const::{lambda(auto:1&)#1}::operator()<AK::Detail::IntrusiveList<Kernel::Thread, Kernel::Thread*, &Kernel::Thread::m_process_thread_list_node> const>(AK::Detail::IntrusiveList<Kernel::Thread, Kernel::Thread*, &Kernel::Thread::m_process_thread_list_node> const&) const () at ././Kernel/Tasks/Process.h:1170
#10 0x00000020004cdf38 in Kernel::SpinlockProtectedBase<AK::Detail::IntrusiveList<Kernel::Thread, Kernel::Thread*, &Kernel::Thread::m_process_thread_list_node>, Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::with<Kernel::Process::try_for_each_thread(AK::Function<AK::ErrorOr<void, AK::Error> (Kernel::Thread const&)>) const::{lambda(auto:1&)#1}>(Kernel::Process::try_for_each_thread(AK::Function<AK::ErrorOr<void, AK::Error> (Kernel::Thread const&)>) const::{lambda(auto:1&)#1}) const () at ././Kernel/Locking/SpinlockProtectedBase.h:62
#11 Kernel::Process::try_for_each_thread () at ././Kernel/Tasks/Process.h:1168
#12 operator() () at ./Kernel/FileSystem/SysFS/Subsystems/Kernel/Processes.cpp:117
#13 0x00000020004d04d8 in operator() () at ./Kernel/FileSystem/SysFS/Subsystems/Kernel/Processes.cpp:154
#14 call () at ././AK/Function.h:201
#15 0x00000020007260b5 in AK::Function<AK::ErrorOr<void, AK::Error>(Kernel::Process&)>::operator() () at ././AK/Function.h:135
#16 0x0000002000710f81 in operator()<AK::Detail::IntrusiveListRelaxedConst<Kernel::Process, Kernel::Process*, &Kernel::Process::m_all_processes_list_node> > () at ./Kernel/Tasks/Process.cpp:82
#17 with<Kernel::Process::for_each_in_same_process_list(AK::Function<AK::ErrorOr(Kernel::Process&)>)::<lambda(const auto:198&)>::<lambda(const auto:200&)> > () at ././Kernel/Locking/SpinlockProtectedBase.h:69
#18 operator()<AK::RefPtrKernel::ScopedProcessList > () at ./Kernel/Tasks/Process.cpp:80
#19 with<Kernel::Process::for_each_in_same_process_list(AK::Function<AK::ErrorOr(Kernel::Process&)>)::<lambda(const auto:198&)> > () at ././Kernel/Locking/SpinlockProtectedBase.h:69
#20 Kernel::Process::for_each_in_same_process_list () at ./Kernel/Tasks/Process.cpp:68
#21 0x00000020004d08b8 in Kernel::SysFSOverallProcesses::try_generate () at ./Kernel/FileSystem/SysFS/Subsystems/Kernel/Processes.cpp:154
#22 0x00000020004e5929 in Kernel::SysFSGlobalInformation::refresh_data () at ./Kernel/FileSystem/SysFS/Subsystems/Kernel/GlobalInformation.cpp:57
#23 0x000000200045ffee in Kernel::OpenFileDescription::attach () at ./Kernel/FileSystem/OpenFileDescription.cpp:69
#24 0x0000002000469010 in Kernel::OpenFileDescription::try_create () at ./Kernel/FileSystem/OpenFileDescription.cpp:34
#25 0x00000020005155c5 in Kernel::VirtualFileSystem::open () at ./Kernel/FileSystem/VirtualFileSystem.cpp:453
#26 0x00000020005162a6 in Kernel::VirtualFileSystem::open () at ./Kernel/FileSystem/VirtualFileSystem.cpp:362
#27 0x0000002000674119 in Kernel::Process::open_impl () at ./Kernel/Syscalls/open.cpp:59
#28 0x00000020006851f9 in Kernel::Process::sys$open () at ./Kernel/Syscalls/profiled_syscalls.cpp:47
#29 0x00000020006b43b8 in Kernel::Syscall::handle () at ./Kernel/Syscalls/SyscallHandler.cpp:95
#30 0x00000020006b50f5 in syscall_handler () at ./Kernel/Syscalls/SyscallHandler.cpp:158
#31 0x0000002000795b14 in syscall_entry () at ./Kernel/Arch/x86_64/SyscallEntry.cpp:25
#32 0x0000000000000000 in ?? ()

Thread 2 (Thread 1.2 (CPU#1 [running])):
#0 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:76
#1 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135
#2 0x0000002000731b98 in Kernel::Thread::reset_signals_for_exec () at ./Kernel/Tasks/Thread.cpp:746
#3 0x000000200063d66a in Kernel::Process::do_exec () at ./Kernel/Syscalls/execve.cpp:571
#4 0x0000002000640052 in Kernel::Process::exec () at ./Kernel/Syscalls/execve.cpp:951
#5 0x0000002000641d7f in Kernel::Process::exec () at ./Kernel/Syscalls/execve.cpp:1015
#6 Kernel::Process::sys$execve () at ./Kernel/Syscalls/execve.cpp:1015
#7 0x00000020006b43b8 in Kernel::Syscall::handle () at ./Kernel/Syscalls/SyscallHandler.cpp:95
#8 0x00000020006b50f5 in syscall_handler () at ./Kernel/Syscalls/SyscallHandler.cpp:158
#9 0x0000002000795b14 in syscall_entry () at ./Kernel/Arch/x86_64/SyscallEntry.cpp:25
#10 0x0000000000000000 in ?? ()

Thread 1 (Thread 1.1 (CPU#0 [running])):
#0 Kernel::cli () at ././Kernel/Arch/x86_64/ASM_wrapper.h:18
#1 Kernel::ProcessorBase::disable_interrupts () at ././Kernel/Arch/x86_64/Processor.h:242
#2 Kernel::InterruptDisabler::InterruptDisabler () at ././Kernel/Interrupts/InterruptDisabler.h:19
#3 Kernel::ProcessorBase::leave_critical () at ./Kernel/Arch/Processor.cpp:91
#4 0x00000020007927c5 in Kernel::Processor::smp_process_pending_messages () at ./Kernel/Arch/x86_64/Processor.cpp:1032
#5 0x00000020002037f1 in Kernel::ProcessorBase::wait_check () at ././Kernel/Arch/x86_64/Processor.h:269
#6 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:79
#7 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135
#8 0x0000002000710737 in Kernel::SpinlockProtectedBase<AK::Detail::IntrusiveListRelaxedConst<Kernel::Process, Kernel::Process*, &Kernel::Process::m_all_processes_list_node>, Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::Locked<AK::Detail::IntrusiveListRelaxedConst<Kernel::Process, Kernel::Process*, &Kernel::Process::m_all_processes_list_node> >::Locked () at ././Kernel/Locking/SpinlockProtectedBase.h:30
#9 Kernel::SpinlockProtectedBase<AK::Detail::IntrusiveListRelaxedConst<Kernel::Process, Kernel::Process*, &Kernel::Process::m_all_processes_list_node>, Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::lock_mutable () at ././Kernel/Locking/SpinlockProtectedBase.h:49
#10 with<Kernel::Process::from_pid_ignoring_process_lists(Kernel::ProcessID)::<lambda(const auto:222&)> > () at ././Kernel/Locking/SpinlockProtectedBase.h:68
#11 Kernel::Process::from_pid_ignoring_process_lists () at ./Kernel/Tasks/Process.cpp:664
#12 0x000000200073dcb6 in Kernel::Thread::dispatch_signal () at ./Kernel/Tasks/Thread.cpp:919
#13 0x000000200072bec3 in Kernel::Scheduler::context_switch () at ./Kernel/Tasks/Scheduler.cpp:327
#14 0x000000200072cd7c in Kernel::Scheduler::pick_next () at ./Kernel/Tasks/Scheduler.cpp:242
#15 0x000000200072d105 in Kernel::Scheduler::invoke_async () at ./Kernel/Tasks/Scheduler.cpp:498
#16 0x0000002000791848 in Kernel::ProcessorBase::clear_critical () at ./Kernel/Arch/x86_64/Processor.cpp:1283
#17 0x000000200072fba5 in Kernel::Thread::yield_without_releasing_big_lock () at ./Kernel/Tasks/Thread.cpp:434
#18 0x0000002000738c4d in Kernel::Thread::block_impl () at ./Kernel/Tasks/Thread.cpp:185
#19 0x0000002000739945 in Kernel::Thread::block<Kernel::Thread::SleepBlocker, Kernel::Thread::BlockTimeout, AK::Duration*&> () at ././Kernel/Tasks/Thread.h:824
#20 Kernel::Thread::sleep () at ./Kernel/Tasks/Thread.cpp:481
#21 0x000000200062ec49 in Kernel::Process::sys$clock_nanosleep () at ./Kernel/Syscalls/clock.cpp:86
#22 0x00000020006b43b8 in Kernel::Syscall::handle () at ./Kernel/Syscalls/SyscallHandler.cpp:95
#23 0x00000020006b50f5 in syscall_handler () at ./Kernel/Syscalls/SyscallHandler.cpp:158
#24 0x0000002000795b14 in syscall_entry () at ./Kernel/Arch/x86_64/SyscallEntry.cpp:25
#25 0x0000000000000000 in ?? ()

The interesting ones are Thread 1 and 3.

Thread 1 blocks while waiting for Process::all_instances() while holding: 1. the scheduler lock, 2. The thread's own lock (Thread::m_lock, aquired in Scheduler::context_switch).

Thread 3 blocks while waiting for Thread::m_lock while holding all_instances's lock (aquired in for_each_in_same_process_list).

This is the usual ABBA locking order issue. I think the ranking system should have been able to catch that but all_instances's lock is not annotated.