[BUG]redis-plus-plus core dump / crash when deleting AsyncRedisCluster

[yangandyyong]

Describe the bug
Deleting AsyncRedisCluster caused core dump if want to use different host/ip configure when redis-server cluster nodes scale in/out.

To Reproduce
[1.] Create async redis cluster with initial ip resolved from fqdn host in following way.
this->client = new AsyncRedisCluster(cops, poolOption);

[2.] Continue the traffic

[3.] scale in or scale out existing redis server master/slave nodes.(kill -9 redis-server-pid and restart new one or execute kubernetes scale in/out for the redis pods) .
kubectl scale sts rtdb --replicas=18
kubectl scale sts rtdb --replicas 0
kubectl scale sts rtdb --replicas=18

[4.] stop traffic when detecting redis server cluster is not health

[5.] keep cluster audit every 2 seconds by sending "cluster info".

[6.] After audit failure up to threshold number(3), delete existing AsyncRedisCluster and create new one with new ip configure during the redis server cluster nodes changed. The same thread is used to send cluster audit and destroy existing AsyncRedisCluster.

[7.] coredump fired when deleting AsyncRedisCluster

coredump example 1:
(gdb) bt
#0 0xf7ee6129 in __kernel_vsyscall ()
#1 0xf56fcad6 in raise () from /lib/libc.so.6
#2 0xf56e2f12 in abort () from /lib/libc.so.6
#3 0xf59d3f81 in __gnu_cxx::__verbose_terminate_handler() [clone .cold.1] () from /lib/libstdc++.so.6
#4 0xf59dbb08 in __cxxabiv1::__terminate(void ()()) () from /lib/libstdc++.so.6
#5 0xf59da85b in __cxa_call_terminate () from /lib/libstdc++.so.6
#6 0xf59db31a in __gxx_personality_v0 () from /lib/libstdc++.so.6
#7 0xf5886e76 in _Unwind_RaiseException_Phase2 () from /lib/libgcc_s.so.1
#8 0xf58872c3 in _Unwind_RaiseException () from /lib/libgcc_s.so.1
#9 0xf59dbe93 in __cxa_throw () from /lib/libstdc++.so.6
#10 0xf59d6947 in std::__throw_system_error(int) [clone .cold.28] () from /lib/libstdc++.so.6
#11 0xf5a0b0ca in std::thread::join() () from /lib/libstdc++.so.6
#12 0xf54474a3 in sw::redis::AsyncShardsPool::~AsyncShardsPool() () from /lib/libredis++.so.1
#13 0xf5446ed0 in void __gnu_cxx::new_allocatorsw::redis::AsyncShardsPool::destroysw::redis::AsyncShardsPool(sw::redis::AsyncShardsPool) () from /lib/libredis++.so.1
#14 0xf5446e5d in void std::allocator_traits<std::allocatorsw::redis::AsyncShardsPool >::destroysw::redis::AsyncShardsPool(std::allocatorsw::redis::AsyncShardsPool&, sw::redis::AsyncShardsPool*) () from /lib/libredis++.so.1

coredump example 2:

(gdb) bt
#0 0xf7f30129 in __kernel_vsyscall ()
#1 0xf57cbad6 in raise () from /lib/libc.so.6
#2 0xf57b1f12 in abort () from /lib/libc.so.6
#3 0xf5aa2f81 in __gnu_cxx::__verbose_terminate_handler() [clone .cold.1] () from /lib/libstdc++.so.6
#4 0xf5aaab08 in __cxxabiv1::__terminate(void ()()) () from /lib/libstdc++.so.6
#5 0xf5aa985b in __cxa_call_terminate () from /lib/libstdc++.so.6
#6 0xf5aaa31a in __gxx_personality_v0 () from /lib/libstdc++.so.6
#7 0xf5955e76 in _Unwind_RaiseException_Phase2 () from /lib/libgcc_s.so.1
#8 0xf59562c3 in _Unwind_RaiseException () from /lib/libgcc_s.so.1
#9 0xf5aaae93 in __cxa_throw () from /lib/libstdc++.so.6
#10 0xf5aa5947 in std::__throw_system_error(int) [clone .cold.28] () from /lib/libstdc++.so.6
#11 0xf5ada0ca in std::thread::join() () from /lib/libstdc++.so.6
#12 0xf5507916 in std::_Sp_ebo_helper<0, std::allocatorsw::redis::EventLoop, true>::_Sp_ebo_helper(std::allocatorsw::redis::EventLoop const&) () from /lib/libredis++.so.1
#13 0xf5507815 in void std::allocator_traits<std::allocatorsw::redis::AsyncConnectionPool >::construct<sw::redis::AsyncConnectionPool, sw::redis::SimpleAsyncSentinel, std::shared_ptrsw::redis::EventLoop&, sw::redis::ConnectionPoolOptions const&, sw::redis::ConnectionOptions const&>(std::allocatorsw::redis::AsyncConnectionPool&, sw::redis::AsyncConnectionPool, sw::redis::SimpleAsyncSentinel&&, std::shared_ptrsw::redis::EventLoop&, sw::redis::ConnectionPoolOptions const&, sw::redis::ConnectionOptions const&) ()
from /lib/libredis++.so.1
#14 0xf5506fae in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<sw::redis::AsyncConnectionPool, std::allocatorsw::redis::AsyncConnectionPool, std::shared_ptrsw::redis::EventLoop&, sw::redis::ConnectionPoolOptions const&, sw::redis::ConnectionOptions const&>(sw::redis::AsyncConnectionPool*&, std::_Sp_alloc_shared_tag<std::allocatorsw::redis::AsyncConnectionPool >, std::shared_ptrsw::redis::EventLoop&, sw::redis::ConnectionPoolOptions const&, sw::redis::ConnectionOptions const&) () from /lib/libredis++.so.1
#15 0xf5506f75 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<sw::redis::AsyncConnectionPool, std::allocatorsw::redis::AsyncConnectionPool, std::shared_ptrsw::redis::EventLoop&, sw::redis::ConnectionPoolOptions const&, sw::redis::ConnectionOptions const&>(sw::redis::AsyncConnectionPool*&, std::_Sp_alloc_shared_tag<std::allocatorsw::redis::AsyncConnectionPool >, std::shared_ptrsw::redis::EventLoop&, sw::redis::ConnectionPoolOptions const&, sw::redis::ConnectionOptions const&) () from /lib/libredis++.so.1
#16 0xf5506e42 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<sw::redis::AsyncConnectionPool, std::allocatorsw::redis::AsyncConnectionPool, std::shared_ptrsw::redis::EventLoop&, sw::redis::ConnectionPoolOptions const&, sw::redis::ConnectionOptions const&>(sw::redis::AsyncConnectionPool*&, std::_Sp_alloc_shared_tag<std::allocatorsw::redis::AsyncConnectionPool >, std::shared_ptrsw::redis::EventLoop&, sw::redis::ConnectionPoolOptions const&, sw::redis::ConnectionOptions const&) () from /lib/libredis++.so.1
--Type for more, q to quit, c to continue without paging--

Expected behavior
No crash when deleting AsyncRedisCluster.

Environment:
OS: Rocky Linux 8.2-20.el8.0.1
Compiler: gcc version 8.5.0
hiredis version: hiredis 1.2.0
redis-plus-plus version: 1.3.14

Additional context
Redis cluster is used with 9 masters and 9 slaves.

[sewenew]

Please update to version 1.3.15, which fixed a bug on uv handle closing. You can try the new version to see if the problem still exists.

Regards

[yangandyyong]

This crash issue was reproduced again.
After reading the changes of version 1.3.15 fix about "uv loop close" is not related to this crash issue.

The "what(): Resource deadlock avoided" is printed in terminal when application exited.
The function back trace from frame #14 shown this crash was reported in destruction of sw::redis::EventLoop::~EventLoop.

The frames from #14 to #10 shown function call ~EventLoop() -> stop() -> _loop_thread.join() in line 53 in src/sw/redis++/event_loop.cpp. std::thread::join() thrown exception when joining self to avoid "Resource deadlock" and then abort the process.

vi src/sw/redis++/event_loop.cpp

39 void EventLoop::stop() {
40 {
41 std::lock_guardstd::mutex lock(_mtx);
42
43 if (_stopped) {
44 return;
45 }
46
47 _stopped = true;
48 }
49
50 _stop();
51
52 if (_loop_thread.joinable()) {
53 _loop_thread.join();
54 }
55 }

241 void EventLoop::_stop() {
242 assert(_stop_async);
243
244 uv_async_send(_stop_async.get());
245 }

(gdb) bt
#0 0xf7f97129 in __kernel_vsyscall ()
#1 0xf7155ad6 in __libc_signal_restore_set (set=0xf5ef87fc) at ../sysdeps/unix/sysv/linux/internal-signals.h:84
#2 __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:48
#3 0xf713be64 in __GI_abort () at abort.c:79
#4 0xf742cf81 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#5 0xf7434b08 in __cxxabiv1::__terminate (handler=0xf7437170 <__gnu_cxx::__verbose_terminate_handler()>)
at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:47
#6 0xf743385b in __cxa_call_terminate (ue_header=0x89b2df0) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54
#7 0xf743431a in __cxxabiv1::__gxx_personality_v0 (version=1, actions=6, exception_class=5138137972254386944, ue_header=0x89b2df0,
context=0xf5ef8d10) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:676
#8 0xf72dfe76 in _Unwind_RaiseException_Phase2 (exc=0x89b2df0, context=0xf5ef8d10, frames_p=0xf5ef8d90) at ../../../libgcc/unwind.inc:64
#9 0xf72e02c3 in _Unwind_RaiseException (exc=0x89b2df0) at ../../../libgcc/unwind.inc:136
#10 0xf7434e93 in __cxxabiv1::__cxa_throw (obj=0x89b2e10, tinfo=0xf7546ed4 ,
dest=0xf7463a40 std::system_error::~system_error()) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90
#11 0xf742f947 in std::__throw_system_error (__i=)
at /usr/src/debug/gcc-8.5.0-28.el8_10.i386/obj-i686-redhat-linux/i686-redhat-linux/libstdc++-v3/include/ext/new_allocator.h:86
#12 0xf74640ca in std::thread::join (this=0xf5f09c44) at ../../../../../libstdc++-v3/src/c++11/thread.cc:113
#13 0xf76ff756 in sw::redis::EventLoop::stop (this=0xf5f09c3c)
at /usr/src/debug/redis-plus-plus-1.3.14-4_hiredis_1_2_0_tls_on.el8.i386/src/sw/redis++/event_loop.cpp:53
#14 0xf76ff655 in sw::redis::EventLoop::~EventLoop (this=0xf5f09c3c, __in_chrg=)
at /usr/src/debug/redis-plus-plus-1.3.14-4_hiredis_1_2_0_tls_on.el8.i386/src/sw/redis++/event_loop.cpp:36
#15 0xf76fedee in __gnu_cxx::new_allocatorsw::redis::EventLoop::destroysw::redis::EventLoop (this=0xf5f09c3c, __p=0xf5f09c3c)
at /usr/include/c++/8/ext/new_allocator.h:140
#16 0xf76fedb5 in std::allocator_traits<std::allocatorsw::redis::EventLoop >::destroysw::redis::EventLoop (__a=..., __p=0xf5f09c3c)
at /usr/include/c++/8/bits/alloc_traits.h:487
#17 0xf76fec82 in std::_Sp_counted_ptr_inplace<sw::redis::EventLoop, std::allocatorsw::redis::EventLoop, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0xf5f09c30) at /usr/include/c++/8/bits/shared_ptr_base.h:554
#18 0xf7e12af5 in ?? ()
#19 0xf7e113e0 in ?? ()
#20 0xf7e0fc82 in ?? ()
#21 0xf7e0fcaf in ?? ()
#22 0xf76f25e2 in sw::redis::AsyncConnection::_connect (this=0xf4d01340)
at /usr/src/debug/redis-plus-plus-1.3.14-4_hiredis_1_2_0_tls_on.el8.i386/src/sw/redis++/async_connection.cpp:484
#23 0xf76f0d29 in sw::redis::AsyncConnection::event_callback (this=0xf4d01340)
at /usr/src/debug/redis-plus-plus-1.3.14-4_hiredis_1_2_0_tls_on.el8.i386/src/sw/redis++/async_connection.cpp:163
#24 0xf76fff74 in sw::redis::EventLoop::_event_callback (handle=0xf4b01680)
at /usr/src/debug/redis-plus-plus-1.3.14-4_hiredis_1_2_0_tls_on.el8.i386/src/sw/redis++/event_loop.cpp:146
#25 0xf70d735d in uv__async_io (loop=0xf5f09cb0, w=0xf5f09da0, events=1) at /usr/src/debug/libuv-1.51.0-101.nokia.el8.i386/src/unix/async.c:208
#26 0xf70f19ed in uv__io_poll (loop=0xf5f09cb0, timeout=-1) at /usr/src/debug/libuv-1.51.0-101.nokia.el8.i386/src/unix/linux.c:1565
#27 0xf70d820a in uv_run (loop=0xf5f09cb0, mode=UV_RUN_DEFAULT) at /usr/src/debug/libuv-1.51.0-101.nokia.el8.i386/src/unix/core.c:460
#28 0xf76ff3f5 in sw::redis::EventLoop::<lambda()>::operator()(void) const (__closure=0xf5f0b9e4)
at /usr/src/debug/redis-plus-plus-1.3.14-4_hiredis_1_2_0_tls_on.el8.i386/src/sw/redis++/event_loop.cpp:32
--Type for more, q to quit, c to continue without paging--
#29 0xf7700d1a in std::__invoke_impl<void, sw::redis::EventLoop::EventLoop()::<lambda()> >(std::__invoke_other, sw::redis::EventLoop::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#30 0xf7700b54 in std::__invoke<sw::redis::EventLoop::EventLoop()::<lambda()> >(sw::redis::EventLoop::<lambda()> &&) (__fn=...)
at /usr/include/c++/8/bits/invoke.h:95
#31 0xf7700f5e in std::thread::_Invoker<std::tuple<sw::redis::EventLoop::EventLoop()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (
this=0xf5f0b9e4) at /usr/include/c++/8/thread:244
#32 0xf7700f25 in std::thread::_Invoker<std::tuple<sw::redis::EventLoop::EventLoop()::<lambda()> > >::operator()(void) (this=0xf5f0b9e4)
at /usr/include/c++/8/thread:253
#33 0xf7700f01 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<sw::redis::EventLoop::EventLoop()::<lambda()> > > >::_M_run(void) (
this=0xf5f0b9e0) at /usr/include/c++/8/thread:196
#34 0xf7463df1 in std::execute_native_thread_routine (__p=0xf5f0b9e0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#35 0xf7af1ec9 in start_thread (arg=) at pthread_create.c:479
#36 0xf714017a in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:108
(gdb)
(gdb)

[yangandyyong]

@sewenew,

Thanks for your suggestions.

I tried the new version(1.3.15), the same crash was reproduced with same function back trace. This crash was caused by _loop_thread.join() because joining self was detected in pthread_join() of glibc and return EDEADLK with description "Resource deadlock avoided" shown in terminal.

How to avoid this pthread join self dead lock in destruction of EventLoop() and also make sure the libuv run loop can be stopped/closed correctly?

The reproduced procedure as below to scale redis cluster server instances.
kubectl scale sts rtdb --replicas=18
kubectl scale sts rtdb --replicas 1
Wait for long time if no crash, repeat this procedure.
kubectl scale sts rtdb --replicas=18

vi src/sw/redis++/event_loop.cpp

37 EventLoop::~EventLoop() {
38 stop();
39 }
40
41 void EventLoop::stop() {
42 {
43 std::lock_guardstd::mutex lock(_mtx);
44
45 if (_stopped) {
46 return;
47 }
48
49 _stopped = true;
50 }
51
52 _stop();
53
54 if (_loop_thread.joinable()) {
55 _loop_thread.join();
56 }
57 }

#rpm -ql glibc-debugsource-2.28-251.el8_10.25.i686

21 int
22 __pthread_join (pthread_t threadid, void **thread_return)
23 {
24 return __pthread_timedjoin_ex (threadid, thread_return, NULL, true);
25 }
26 weak_alias (__pthread_join, pthread_join)

33 int
34 __pthread_timedjoin_ex (pthread_t threadid, void **thread_return,
35 const struct timespec *abstime, bool block)
36 {
37 struct pthread *pd = (struct pthread ) threadid;
38
39 / Make sure the descriptor is valid. /
40 if (INVALID_NOT_TERMINATED_TD_P (pd))
41 / Not a valid thread handle. /
42 return ESRCH;
43
44 / Is the thread joinable?. /
45 if (IS_DETACHED (pd))
46 / We cannot wait for the thread. */
47 return EINVAL;
48
49 struct pthread self = THREAD_SELF;
50 int result = 0;
51
52 LIBC_PROBE (pthread_join, 1, threadid);
53
54 if ((pd == self
55 || (self->joinid == pd
56 && (pd->cancelhandling
57 & (CANCELING_BITMASK | CANCELED_BITMASK | EXITING_BITMASK
58 | TERMINATED_BITMASK)) == 0))
59 && !CANCEL_ENABLED_AND_CANCELED (self->cancelhandling))
60 / This is a deadlock situation. The threads are waiting for each
61 other to finish. Note that this is a "may" error. To be 100%
62 sure we catch this error we would have to lock the data
63 structures but it is not necessary. In the unlikely case that
64 two threads are really caught in this situation they will
65 deadlock. It is the programmer's problem to figure this
66 out. */
67 return EDEADLK;

#ifdef EDEADLK
/*
TRANS Allocating a system resource would have resulted in a
TRANS deadlock situation. The system does not guarantee that it will notice
TRANS all such situations. This error means you got lucky and the system
TRANS noticed; it might just hang. @Xref{File Locks}, for an example. */
[ERR_REMAP (EDEADLK)] = N_("Resource deadlock avoided"),

The terminal output:
terminate called after throwing an instance of 'std::system_error'
what(): Resource deadlock avoided

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./example'.
Program terminated with signal SIGABRT, Aborted.
#0 0xf7ee0129 in __kernel_vsyscall ()
[Current thread is 1 (Thread 0xf5ecbb40 (LWP 3908))]
(gdb) bt
#0 0xf7ee0129 in __kernel_vsyscall ()
#1 0xf709ead6 in __libc_signal_restore_set (set=0xf5ec47fc) at ../sysdeps/unix/sysv/linux/internal-signals.h:84
#2 __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:48
#3 0xf7084e64 in __GI_abort () at abort.c:79
#4 0xf7375f81 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#5 0xf737db08 in __cxxabiv1::__terminate (handler=0xf7380170 <__gnu_cxx::__verbose_terminate_handler()>)
at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:47
#6 0xf737c85b in __cxa_call_terminate (ue_header=0x9b796a0) at ../../../../libstdc++-v3/libsupc++/eh_call.cc:54
#7 0xf737d31a in __cxxabiv1::__gxx_personality_v0 (version=1, actions=6, exception_class=5138137972254386944, ue_header=0x9b796a0,
context=0xf5ec4d10) at ../../../../libstdc++-v3/libsupc++/eh_personality.cc:676
#8 0xf7228e76 in _Unwind_RaiseException_Phase2 (exc=0x9b796a0, context=0xf5ec4d10, frames_p=0xf5ec4d90) at ../../../libgcc/unwind.inc:64
#9 0xf72292c3 in _Unwind_RaiseException (exc=0x9b796a0) at ../../../libgcc/unwind.inc:136
#10 0xf737de93 in __cxxabiv1::__cxa_throw (obj=0x9b796c0, tinfo=0xf748fed4 ,
dest=0xf73aca40 std::system_error::~system_error()) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:90
#11 0xf7378947 in std::__throw_system_error (__i=)
at /usr/src/debug/gcc-8.5.0-28.el8_10.i386/obj-i686-redhat-linux/i686-redhat-linux/libstdc++-v3/include/ext/new_allocator.h:86
#12 0xf73ad0ca in std::thread::join (this=0xf5f09904) at ../../../../../libstdc++-v3/src/c++11/thread.cc:113
#13 0xf7648756 in sw::redis::EventLoop::stop (this=0xf5f098fc)
at /usr/src/debug/redis-plus-plus-1.3.15-6_hiredis_1_2_0_tls_on.el8.i386/src/sw/redis++/event_loop.cpp:55
#14 0xf7648655 in sw::redis::EventLoop::~EventLoop (this=0xf5f098fc, __in_chrg=)
at /usr/src/debug/redis-plus-plus-1.3.15-6_hiredis_1_2_0_tls_on.el8.i386/src/sw/redis++/event_loop.cpp:38
#15 0xf7647dee in __gnu_cxx::new_allocatorsw::redis::EventLoop::destroysw::redis::EventLoop (this=0xf5f098fc, __p=0xf5f098fc)
at /usr/include/c++/8/ext/new_allocator.h:140
#16 0xf7647db5 in std::allocator_traits<std::allocatorsw::redis::EventLoop >::destroysw::redis::EventLoop (__a=..., __p=0xf5f098fc)
at /usr/include/c++/8/bits/alloc_traits.h:487
#17 0xf7647c82 in std::_Sp_counted_ptr_inplace<sw::redis::EventLoop, std::allocatorsw::redis::EventLoop, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0xf5f098f0) at /usr/include/c++/8/bits/shared_ptr_base.h:554
#18 0xf7d5bb1d in ?? ()
#19 0xf7d5a408 in ?? ()
#20 0xf7d58caa in ?? ()
#21 0xf7d58cd7 in ?? ()
#22 0xf763b5e2 in sw::redis::AsyncConnection::_connect (this=0xf4b08130)
at /usr/src/debug/redis-plus-plus-1.3.15-6_hiredis_1_2_0_tls_on.el8.i386/src/sw/redis++/async_connection.cpp:484
#23 0xf7639d29 in sw::redis::AsyncConnection::event_callback (this=0xf4b08130)
at /usr/src/debug/redis-plus-plus-1.3.15-6_hiredis_1_2_0_tls_on.el8.i386/src/sw/redis++/async_connection.cpp:163
#24 0xf7648f74 in sw::redis::EventLoop::_event_callback (handle=0xf4d018b0)
at /usr/src/debug/redis-plus-plus-1.3.15-6_hiredis_1_2_0_tls_on.el8.i386/src/sw/redis++/event_loop.cpp:148
#25 0xf702035d in uv__async_io (loop=0xf5f09970, w=0xf5f09a60, events=1) at /usr/src/debug/libuv-1.51.0-101.nokia.el8.i386/src/unix/async.c:208
#26 0xf703a9ed in uv__io_poll (loop=0xf5f09970, timeout=124) at /usr/src/debug/libuv-1.51.0-101.nokia.el8.i386/src/unix/linux.c:1565
#27 0xf702120a in uv_run (loop=0xf5f09970, mode=UV_RUN_DEFAULT) at /usr/src/debug/libuv-1.51.0-101.nokia.el8.i386/src/unix/core.c:460
#28 0xf76483f5 in sw::redis::EventLoop::<lambda()>::operator()(void) const (__closure=0xf4d08594)
at /usr/src/debug/redis-plus-plus-1.3.15-6_hiredis_1_2_0_tls_on.el8.i386/src/sw/redis++/event_loop.cpp:34
--Type for more, q to quit, c to continue without paging--
#29 0xf7649d88 in std::__invoke_impl<void, sw::redis::EventLoop::EventLoop()::<lambda()> >(std::__invoke_other, sw::redis::EventLoop::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#30 0xf7649bc2 in std::__invoke<sw::redis::EventLoop::EventLoop()::<lambda()> >(sw::redis::EventLoop::<lambda()> &&) (__fn=...)
at /usr/include/c++/8/bits/invoke.h:95
#31 0xf7649fcc in std::thread::_Invoker<std::tuple<sw::redis::EventLoop::EventLoop()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (
this=0xf4d08594) at /usr/include/c++/8/thread:244
#32 0xf7649f93 in std::thread::_Invoker<std::tuple<sw::redis::EventLoop::EventLoop()::<lambda()> > >::operator()(void) (this=0xf4d08594)
at /usr/include/c++/8/thread:253
#33 0xf7649f6f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<sw::redis::EventLoop::EventLoop()::<lambda()> > > >::_M_run(void) (
this=0xf4d08590) at /usr/include/c++/8/thread:196
#34 0xf73acdf1 in std::execute_native_thread_routine (__p=0xf4d08590) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#35 0xf7a3aec9 in start_thread (arg=) at pthread_create.c:479
#36 0xf708917a in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:108
(gdb)
(gdb)

[yangandyyong]

@sewenew,

Any suggestion for this crash issue?

Thanks.

[sewenew]

I tried but cannot reproduce the problem. Could you please give some compilable code that can reproduce the problem? So that I can debug it. Thanks and sorry for the late reply.

Regards

[yangandyyong]

@sewenew,

This issue is not easy to be reproduced. My testing code includes some product codes and it cannot be shared. Thanks for your understanding. It will take more time to remove product codes before sharing it.

Here is some analysis on this issue for your reference and solution for AsyncShardsPool.
The same solution is applied to EventLoop.

+++++++++++++++++++++++++++++++++++
Cause recap
The crash is from std::thread::join() in
sw::redis::AsyncShardsPool::~AsyncShardsPool()
when the destructor runs on the pool?s own worker thread. That calls join() on itself, which throws std::system_error(EDEADLK), leading to terminate/abort.
Worse, if the last reference to
AsyncShardsPool
is released by code running on the worker thread, the destructor can run while
_run()
is still using this ? potential use?after?free unless we keep the object alive until
_run()
exits.
Fix in redis-plus-plus
There are two levels of fixes: a minimal guard and a robust lifetime fix. Do both.

Minimal: guard join() against self-join in
AsyncShardsPool::~AsyncShardsPool()
.
Robust: make the worker thread hold a self shared_ptr to prevent destruction while
_run()
is running; release it right before
_run()
returns. If the final release happens on the worker thread, destructor will run there after
_run()
returns; then we detach instead of join.

1. async_shards_pool.h
   File:
   src/sw/redis++/async_shards_pool.h
   Changes:
   Add a self-hold and an attach API.
   Optionally derive from std::enable_shared_from_this if you prefer self-acquire in _run (see note), but the approach below avoids calling shared_from_this() in constructor by attaching from the creator.
   cpp
   // add to includes if not present
   #include
   #include

namespace sw {
namespace redis {

class AsyncShardsPool /* optionally: : public std::enable_shared_from_this */ {
public:
// ... existing ctors ...

~AsyncShardsPool();

// Attach self-hold to prevent destruction while _run() is active.
void attach_self(const std::shared_ptr<AsyncShardsPool>& self);

private:
void _run();

// ... existing members ...

std::shared_ptr<AsyncShardsPool> _self_guard;   // keeps this alive during _run

};

} // namespace redis
} // namespace sw
2) async_shards_pool.cpp
File:
src/sw/redis++/async_shards_pool.cpp
Changes:
Implement attach_self.
Release _self_guard at the end of
_run()
.
Guard join() in the destructor.
cpp
using namespace sw::redis;

// New: attach_self
void AsyncShardsPool::attach_self(const std::shared_ptr& self) {
_self_guard = self;
}

AsyncShardsPool::~AsyncShardsPool() {
// Signal worker to stop.
update({}, nullptr);

if (_worker.joinable()) {
    const auto self_tid = std::this_thread::get_id();
    const auto worker_tid = _worker.get_id();
    if (self_tid == worker_tid) {
        // Destructor is running on the worker thread. Do NOT join self.
        // Detach so std::thread destructor won't call std::terminate().
        _worker.detach();
    } else {
        _worker.join();
    }
}

}

void AsyncShardsPool::_run() {
// NOTE: _self_guard is set by attach_self() from the creator.
// It keeps this object alive while _run executes.

while (true) {
    auto events = _fetch_events();

    assert(!events.empty());

    try {
        _update_shards();

        if (_redeliver_events(events)) {
            break;
        }
    } catch (...) {
        if (_fail_events(events, std::current_exception())) {
            break;
        }
        std::this_thread::sleep_for(std::chrono::seconds(1));
        update();
    }
}

// Release self-hold right before returning, so destruction can proceed.
_self_guard.reset();

}
3) async_redis_cluster.cpp
File:
src/sw/redis++/async_redis_cluster.cpp
Change: attach the self-hold immediately after constructing the pool.
cpp
_pool = std::make_shared(_loop, pool_opts, opts, role, ClusterOptions{});
_pool->attach_self(_pool);
and

cpp
_pool = std::make_shared(_loop, pool_opts, opts, role, cluster_opts);
_pool->attach_self(_pool);
Why this works
The self-hold _self_guard ensures
AsyncShardsPool
isn?t destroyed while
_run()
is using this.
Destruction only happens after
_run()
returns and _self_guard.reset() is called.
If the last reference is released by the worker thread, the destructor runs on that worker thread. The join guard switches to detach() to avoid join() on self and prevents std::thread destructor from calling std::terminate().
Alternative if you don?t want attach_self
If you prefer no API change, you can use enable_shared_from_this and acquire self at the beginning of
_run()
:

Make
AsyncShardsPool
derive from std::enable_shared_from_this.
In
_run()
, first line: _self_guard = shared_from_this();
Caveat: since the worker thread starts in the constructor, there?s a theoretical race where
_run()
might call shared_from_this() before the first shared_ptr sets up the internal weak pointer. In practice with std::make_shared, it?s very unlikely but not strictly guaranteed. The attach_self approach avoids this race.
Minimal-only patch (not recommended alone)
If you can?t change ownership, at least guard join():

Only in
~AsyncShardsPool()
:
cpp
update({}, nullptr);
if (_worker.joinable()) {
if (std::this_thread::get_id() == _worker.get_id()) {
_worker.detach();
} else {
_worker.join();
}
}
This prevents the EDEADLK abort, but it does not protect against use-after-free if the last reference is released on the worker thread while
_run()
is still executing. Prefer the robust fix.

Test plan
Create a repro that resets/destroys
AsyncRedisCluster
from a callback executed on the redis++ async worker.
Before patch: observe std::system_error ? terminate/abort.
After patch:
No abort, no exception; worker exits cleanly.
Valgrind/ASan shows no UAF on shutdown.
GDB confirms destructor runs on worker thread in that path, and detach() is taken.
Recommended Actions
[apply_robust_fix] Implement attach_self + destructor join guard as above in:
src/sw/redis++/async_shards_pool.h
src/sw/redis++/async_shards_pool.cpp
src/sw/redis++/async_redis_cluster.cpp
[app_wrap_safety] In your smart_state/dbw layer, still add the earlier safeguards:
Avoid destroying the client from its worker thread; defer destruction to a control thread.
Catch and log std::system_error on teardown to confirm no regressions.
[upstream_PR] Consider opening an upstream PR with this change; it?s small, maintains behavior, and fixes a hard-to-debug crash path.
Summary
Enhance redis-plus-plus by (1) holding a self shared_ptr during
AsyncShardsPool::_run()
and (2) guarding
~AsyncShardsPool()
to detach instead of joining if the destructor runs on the worker thread. This eliminates the EDEADLK abort and avoids use-after-free when the last reference is released by code running on the pool?s own worker thread.
+++++++++++++++++++++++++++++++++++

[jzkiss]

@yangandyyong we had the same issue and as a workaroud we set the slot_map_refresh_interval option. In this way redispp interrogates the cluster slots at every slot_map_refresh_interval period and sooner or later it learns the new slot assignments / topology. So we removed the deletion of the AsyncRedisCluster as it is not safe. The another problem is that we observed that if you delete the AsyncRedisCluster object and create a new then - according to our observations - the tcp connections of the deleted AsyncRedisCluster object remain hanging and if you frequently delete / re-create the object then you run out of the socket descriptors, or you reach the number of client connection limit at the redis master server. We use 1.3.14 version but I think this issue is still exists in the new release.

[lisay-yan]

A safe remove AsyncRedisCluster is required anyway, for disable redis access, or redis DB changed etc.