You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by "xiedeacc (via GitHub)" <gi...@apache.org> on 2023/05/22 06:05:55 UTC

[I] brpc死锁 (brpc)

xiedeacc opened a new issue, #2259:
URL: https://github.com/apache/brpc/issues/2259

   **Describe the bug (描述bug)**
   进程运行一段时间后,突然死锁。service的方法中会使用DynamicPartitionChannel访问另外一个服务,开启和不开启usercode_in_pthread都会出现死锁
   
   **Versions (各种版本)**
   OS:18.04
   Compiler: gcc11.3.0
   brpc:1.3.0 compile from source
   protobuf:3.19.2
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1647625173

   > > DynamicPartitionChannel
   > 
   > 请教一下 ,有什么业务形态会用到DynamicPartitionChannel这个功能呢
   
   多库


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1557311485

   FLAGS_usercode_in_pthread为true和不为false我都编译过,运行过,反正都会死锁


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc closed issue #2259: brpc死锁
URL: https://github.com/apache/brpc/issues/2259


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1647624873

   改完后,死锁概率降低了,但是还是会发生,目前只是把brpc_thread_num改成几百来规避这个死锁,但是brpc看起来是无法解决bthread使用pthread锁导致的死锁问题,brpc这个问题真的太无语了


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1559634244

   基本可以确定DynamicPartitionChannel,改完之后,目前没有发生死锁了
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556648914

   ```
   process: /proc/1924/mem
   thread: 0x7fa54ecb8080, lwp: 1924, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000a900148 in _ZN4brpc8Acceptor4JoinEv!()+71 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a8d312b in _ZN4brpc6Server4JoinEv!()+42 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000082e8d3 in _ZN4wifi2ad7ranking7service18RankingServiceImpl3RunEv!()+1666 in /data/app/nads-rankingdump/bin/ranking
   #4  0x00000000005b07a4 in main!()+259 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00007fa54d51ac87 in __libc_start_main()+230 in /lib/x86_64-linux-gnu/libc.so.6 at libc-start.c:310
   #6  0x000000000080713a in _start!()+41 in /data/app/nads-rankingdump/bin/ranking
   
   thread: 0x7fa50d1fa700, lwp: 2008, type: 1
   #0  0x00007fa54e6cf9e2 in __libc_recvfrom()+98 in /lib/x86_64-linux-gnu/libpthread.so.0 at recvfrom.c:27
   #1  0x000000000a83182a in _ZN5nacos24UdpNamingServiceListener18listenerThreadFuncEPv!()+601 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a826bb0 in _ZN5nacos6Thread10threadFuncEPv!()+31 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa50d9fb700, lwp: 2007, type: 1
   #0  0x00007fa54d5dd680 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libc.so.6 at nanosleep.c:28
   #1  0x00007fa54d5dd55a in __sleep()+57 in /lib/x86_64-linux-gnu/libc.so.6 at sleep.c:55
   #2  0x000000000a832cd0 in _ZN5nacos18SubscriptionPoller17pollingThreadFuncEPv!()+799 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000a826bb0 in _ZN5nacos6Thread10threadFuncEPv!()+31 in /data/app/nads-rankingdump/bin/ranking
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa50e1fc700, lwp: 2006, type: 1
   #0  0x00007fa54e6cc065 in __pthread_cond_timedwait()+821 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:205
   #1  0x000000000a826f41 in _ZN5nacos13DelayedWorker3runEv!()+640 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a86eb75 in _ZN5nacos10ThreadPool11runInThreadEPv!()+180 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000a826bb0 in _ZN5nacos6Thread10threadFuncEPv!()+31 in /data/app/nads-rankingdump/bin/ranking
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa50e9fd700, lwp: 2005, type: 1
   #0  0x00007fa54e6cc065 in __pthread_cond_timedwait()+821 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:205
   #1  0x000000000a826f41 in _ZN5nacos13DelayedWorker3runEv!()+640 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a86eb75 in _ZN5nacos10ThreadPool11runInThreadEPv!()+180 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000a826bb0 in _ZN5nacos6Thread10threadFuncEPv!()+31 in /data/app/nads-rankingdump/bin/ranking
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa50f1fe700, lwp: 2004, type: 1
   #0  0x00007fa54e6cc065 in __pthread_cond_timedwait()+821 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:205
   #1  0x000000000a826f41 in _ZN5nacos13DelayedWorker3runEv!()+640 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a86eb75 in _ZN5nacos10ThreadPool11runInThreadEPv!()+180 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000a826bb0 in _ZN5nacos6Thread10threadFuncEPv!()+31 in /data/app/nads-rankingdump/bin/ranking
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa50f9ff700, lwp: 2003, type: 1
   #0  0x00007fa54e6cc065 in __pthread_cond_timedwait()+821 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:205
   #1  0x000000000a826f41 in _ZN5nacos13DelayedWorker3runEv!()+640 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a86eb75 in _ZN5nacos10ThreadPool11runInThreadEPv!()+180 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000a826bb0 in _ZN5nacos6Thread10threadFuncEPv!()+31 in /data/app/nads-rankingdump/bin/ranking
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5155fe700, lwp: 2002, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5163fe700, lwp: 2001, type: 1
   #0  0x00007fa54d61a947 in epoll_wait()+87 in /lib/x86_64-linux-gnu/libc.so.6 at epoll_wait.c:30
   #1  0x000000000a8b7841 in _ZN4brpc15EventDispatcher3RunEv!()+64 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a8b7969 in _ZN4brpc15EventDispatcher7RunThisEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #5  0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #6  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #7  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa516bff700, lwp: 2000, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa27b9a in _ZN7bthread9TaskGroup9wait_taskEPm!()+121 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa2ac4f in _ZN7bthread9TaskGroup13run_main_taskEv!()+126 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5179fe700, lwp: 1999, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5181ff700, lwp: 1998, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa518ffb700, lwp: 1997, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5197fc700, lwp: 1996, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa51a5fc700, lwp: 1995, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa51adfd700, lwp: 1994, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa51b5fe700, lwp: 1993, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa51c3fe700, lwp: 1992, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa51cbff700, lwp: 1991, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa51d9ff700, lwp: 1990, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa51e5fc700, lwp: 1989, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa51effd700, lwp: 1988, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa51f7fe700, lwp: 1987, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa27b9a in _ZN7bthread9TaskGroup9wait_taskEPm!()+121 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa2ac4f in _ZN7bthread9TaskGroup13run_main_taskEv!()+126 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5205fe700, lwp: 1986, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa520dff700, lwp: 1985, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa521bfe700, lwp: 1984, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5223ff700, lwp: 1983, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa522ffd700, lwp: 1982, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5237fe700, lwp: 1981, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa523fff700, lwp: 1980, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa524dfe700, lwp: 1979, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5255ff700, lwp: 1978, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5261fd700, lwp: 1977, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa526bfe700, lwp: 1975, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa27b9a in _ZN7bthread9TaskGroup9wait_taskEPm!()+121 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa2ac4f in _ZN7bthread9TaskGroup13run_main_taskEv!()+126 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5273ff700, lwp: 1974, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa52857f700, lwp: 1973, type: 1
   #0  0x00007fa54e6cfd50 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libpthread.so.0 at nanosleep.c:28
   #1  0x000000000a87ba56 in AbslInternalSleepFor_lts_20220623!()+133 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a877a9d in _ZN4wifi2ad7ranking4util11RankingUtil10SleepUntilERKN4absl12lts_202206234TimeEPVb!()+76 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00000000008662b9 in _ZN4wifi2ad7ranking7feature12RankingCache19BackGroundUpdateRunEv!()+312 in /data/app/nads-rankingdump/bin/ranking
   #4  0x00007fa54dbdcf64 in std::execute_native_thread_routine()+19 in /data/app/nads-rankingdump/lib/libstdc++.so.6 at thread.cc:82
   #5  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #6  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa52d5ec700, lwp: 1972, type: 1
   #0  0x00007fa54e6cfd50 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libpthread.so.0 at nanosleep.c:28
   #1  0x00000000008582a6 in _ZN12redis_client4util7SlotMap6ReloadEv!()+181 in /data/app/nads-rankingdump/bin/ranking
   #2  0x00007fa54dbdcf64 in std::execute_native_thread_routine()+19 in /data/app/nads-rankingdump/lib/libstdc++.so.6 at thread.cc:82
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa52dded700, lwp: 1971, type: 1
   #0  0x00007fa54e6cfd50 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libpthread.so.0 at nanosleep.c:28
   #1  0x000000000a877926 in _ZN4wifi2ad7ranking4util11RankingUtil5SleepEl!()+101 in /data/app/nads-rankingdump/bin/ranking
   #2  0x0000000000834110 in _ZN4wifi2ad7ranking7service18RankingServiceImpl9PrintTaskEv!()+175 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54dbdcf64 in std::execute_native_thread_routine()+19 in /data/app/nads-rankingdump/lib/libstdc++.so.6 at thread.cc:82
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa52e5ee700, lwp: 1970, type: 1
   #0  0x00007fa54e6cfd50 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libpthread.so.0 at nanosleep.c:28
   #1  0x000000000a877926 in _ZN4wifi2ad7ranking4util11RankingUtil5SleepEl!()+101 in /data/app/nads-rankingdump/bin/ranking
   #2  0x0000000000839b6c in _ZN4wifi2ad7ranking7service18RankingServiceImpl12ShutdownTaskEv!()+587 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54dbdcf64 in std::execute_native_thread_routine()+19 in /data/app/nads-rankingdump/lib/libstdc++.so.6 at thread.cc:82
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa52edef700, lwp: 1969, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa52f5f0700, lwp: 1968, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa52fdf1700, lwp: 1967, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  inlined            in __gthread_cond_wait at gthr-default.h:865
   #1  inlined            in std::__condvar::wait at std_mutex.h:155
   #1  0x00007fa54dbd7390 in std::condition_variable::wait()+15 in /data/app/nads-rankingdump/lib/libstdc++.so.6 at condition_variable.cc:41
   #2  0x000000000084eb2c in _ZN4wifi2ad7ranking7service18RankingServiceImpl8ScanListEv!()+395 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000083a95b in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+762 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #6  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5305f2700, lwp: 1965, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa530df3700, lwp: 1964, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5315f4700, lwp: 1962, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa531df5700, lwp: 1961, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5325f6700, lwp: 1960, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa532df7700, lwp: 1959, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5335f8700, lwp: 1958, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa533df9700, lwp: 1957, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5345fa700, lwp: 1956, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa534dfb700, lwp: 1955, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5355fc700, lwp: 1954, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa535dfd700, lwp: 1953, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5365fe700, lwp: 1952, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa536dff700, lwp: 1951, type: 1
   #0  0x00007fa54d5dd680 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libc.so.6 at nanosleep.c:28
   #1  0x00007fa54d5dd55a in __sleep()+57 in /lib/x86_64-linux-gnu/libc.so.6 at sleep.c:55
   #2  0x000000000085578a in _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4wifi2ad7ranking7feature13FeatureClient17InitFeatureClientEvEUlvE_EEEEE6_M_runEv!()+57 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54dbdcf64 in std::execute_native_thread_routine()+19 in /data/app/nads-rankingdump/lib/libstdc++.so.6 at thread.cc:82
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa539ffc700, lwp: 1950, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa27b9a in _ZN7bthread9TaskGroup9wait_taskEPm!()+121 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa2ac4f in _ZN7bthread9TaskGroup13run_main_taskEv!()+126 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa53a7fd700, lwp: 1949, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa53b3fd700, lwp: 1948, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa53bdfe700, lwp: 1947, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa53c5ff700, lwp: 1946, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa27b9a in _ZN7bthread9TaskGroup9wait_taskEPm!()+121 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa2ac4f in _ZN7bthread9TaskGroup13run_main_taskEv!()+126 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa53d1fd700, lwp: 1945, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa53e3fe700, lwp: 1944, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa53ebff700, lwp: 1943, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa53f7f9700, lwp: 1942, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
   #11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
   #12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
   #13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa53fffa700, lwp: 1941, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa25895 in _ZN7bthread11TimerThread3runEv!()+3076 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa26299 in _ZN7bthread11TimerThread8run_thisEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
   #3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5407fb700, lwp: 1940, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8faa6f in _ZN4brpc6policyL24CallMethodInBackupThreadEPv!()+46 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8afa95 in _ZN4brpc18UserCodeBackupPool19UserCodeRunningLoopEv!()+276 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8b01c9 in _ZN4brpcL14UserCodeRunnerEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
   #10 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #11 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa540ffc700, lwp: 1939, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8faa6f in _ZN4brpc6policyL24CallMethodInBackupThreadEPv!()+46 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8afa95 in _ZN4brpc18UserCodeBackupPool19UserCodeRunningLoopEv!()+276 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8b01c9 in _ZN4brpcL14UserCodeRunnerEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
   #10 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #11 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5417fd700, lwp: 1938, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8faa6f in _ZN4brpc6policyL24CallMethodInBackupThreadEPv!()+46 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8afa95 in _ZN4brpc18UserCodeBackupPool19UserCodeRunningLoopEv!()+276 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8b01c9 in _ZN4brpcL14UserCodeRunnerEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
   #10 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #11 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa541ffe700, lwp: 1937, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8faa6f in _ZN4brpc6policyL24CallMethodInBackupThreadEPv!()+46 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8afa95 in _ZN4brpc18UserCodeBackupPool19UserCodeRunningLoopEv!()+276 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8b01c9 in _ZN4brpcL14UserCodeRunnerEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
   #10 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #11 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5427ff700, lwp: 1936, type: 1
   #0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
   #1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
   #2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
   #4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
   #5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a8faa6f in _ZN4brpc6policyL24CallMethodInBackupThreadEPv!()+46 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a8afa95 in _ZN4brpc18UserCodeBackupPool19UserCodeRunningLoopEv!()+276 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a8b01c9 in _ZN4brpcL14UserCodeRunnerEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
   #10 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #11 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5439fe700, lwp: 1935, type: 1
   #0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
   #1  0x000000000088a10b in do_completion!()+170 in /data/app/nads-rankingdump/bin/ranking
   #2  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #3  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5441ff700, lwp: 1934, type: 1
   #0  0x00007fa54d60dbb9 in __poll()+73 in /lib/x86_64-linux-gnu/libc.so.6 at poll.c:29
   #1  0x000000000088a354 in do_io!()+403 in /data/app/nads-rankingdump/bin/ranking
   #2  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #3  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa545c7f700, lwp: 1931, type: 1
   #0  0x00007fa54d60dbb9 in __poll()+73 in /lib/x86_64-linux-gnu/libc.so.6 at poll.c:29
   #1  0x00007fa54e916ef9 in <unknown>() in /usr/lib/x86_64-linux-gnu/libcurl.so.4
   #2  0x00007fa54e911212 in curl_multi_wait!()+817 in /usr/lib/x86_64-linux-gnu/libcurl.so.4
   #3  0x00007fa54e909970 in curl_easy_perform!()+207 in /usr/lib/x86_64-linux-gnu/libcurl.so.4
   #4  0x000000000a8156c2 in _ZN5nacos7HTTPCli16httpPostInternalERKSsRSt4listISsSaISsEES2_S2_l!()+625 in /data/app/nads-rankingdump/bin/ranking
   #5  0x000000000a815c90 in _ZN5nacos7HTTPCli8httpPostERKSsRSt4listISsSaISsEES6_S2_l!()+111 in /data/app/nads-rankingdump/bin/ranking
   #6  0x000000000a81a869 in _ZN5nacos16NoOpHttpDelegate8httpPostERKSsRSt4listISsSaISsEES6_S2_l!()+216 in /data/app/nads-rankingdump/bin/ranking
   #7  0x000000000a85deeb in _ZN5nacos11ConfigProxy6reqAPIEiRKSsRSt4listISsSaISsEES6_S2_l!()+2010 in /data/app/nads-rankingdump/bin/ranking
   #8  0x000000000a82288e in _ZN5nacos12ClientWorker17checkListenedKeysEv!()+1789 in /data/app/nads-rankingdump/bin/ranking
   #9  0x000000000a825163 in _ZN5nacos12ClientWorker12performWatchEv!()+82 in /data/app/nads-rankingdump/bin/ranking
   #10 0x000000000a826b4c in _ZN5nacos12ClientWorker14listenerThreadEPv!()+59 in /data/app/nads-rankingdump/bin/ranking
   #11 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #12 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0x7fa5475ff700, lwp: 1929, type: 1
   #0  0x00007fa54d5dd680 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libc.so.6 at nanosleep.c:28
   #1  0x00007fa54d610557 in usleep()+86 in /lib/x86_64-linux-gnu/libc.so.6 at usleep.c:32
   #2  0x000000000aa5be3a in _ZN4bvar6detail16SamplerCollector3runEv!()+601 in /data/app/nads-rankingdump/bin/ranking
   #3  0x000000000aa5c319 in _ZN4bvar6detail16SamplerCollector15sampling_threadEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
   #4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
   #5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95
   
   thread: 0, lwp: 0, type: 0
   #0  0x0000000000000000 no information for frame
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556751970

   可以在启动服务之后观察内置服务的bthread_count指标以及进程内存指标的变化,看是否会不断增长


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556881741

   不过就算done不自己管理,也会出现同样的状况,我已经测试过了


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556711159

   DynamicPartitionChannel 这个一次会并发访问多少个分片呢


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556804207

   看bthread_stack_count和内存占用都不大。
   我看你的堆栈里有CallMethodInBackupThread,是不是开启了usercode_in_pthread=true呢?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556649657

   https://github.com/apache/brpc/issues/1648 和这个问题应该是同一个问题


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "dandyhuang (via GitHub)" <gi...@apache.org>.
dandyhuang commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1643531105

   > DynamicPartitionChannel
   
   请教一下 ,有什么业务形态会用到DynamicPartitionChannel这个功能呢


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556864554

   ![image](https://github.com/apache/brpc/assets/8072296/066d00d1-5626-4764-9cab-7407a11cce6e)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556946363

   把DynamicPartitionChannel换成普通Channle后这个问题修复了,目前观察是这样的,再观察几天看看


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556714302

   8个分片


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556774629

   没有变化,一直是6


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556994283

   brpc问题实在太多了,有负盛名


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556789149

   ```
   bthread_count : 37
   bthread_group_status : "1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 "
   bthread_key_count : 0
   bthread_keytable_count : 0
   bthread_keytable_memory : 0
   bthread_signal_second : 987
   bthread_stack_count : 191
   bthread_switch_second : 2095
   bthread_timer_scheduled_second : 295
   bthread_timer_triggered_second : 14
   bthread_timer_usage : 0.000139691
   bthread_worker_count : 32
   bthread_worker_usage : 32.6478
   bvar_dump_interval : 10
   bvar_sampler_collector_usage : 0.00116603
   gcc_version : "11.3.0"
   iobuf_block_count : 215
   iobuf_block_count_hit_tls_threshold : 19283
   iobuf_block_memory : 1761280
   iobuf_newbigview_second : 1
   kernel_version : "Linux ad-nads-rankingdump7.wxxdc 5.4.193-0504193-generic #202205121146 SMP Thu May 12 12:05:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
   "
   pgrp : 28346
   pid : 28346
   ppid : 1966
   process_cmdline : "./bin/ranking
   "
   process_context_switches_involuntary_second : 5
   process_context_switches_voluntary_second : 2025
   process_cpu_usage : 0.658
   process_cpu_usage_system : 0.325
   process_cpu_usage_user : 0.333
   process_disk_read_bytes_second : 0
   process_disk_write_bytes_second : 250249
   process_faults_major : 0
   process_faults_minor_second : 270
   process_fd_count : 718
   process_inblocks_second : 0
   process_io_read_bytes_second : 712818
   process_io_read_second : 854
   process_io_write_bytes_second : 299397
   process_io_write_second : 967
   process_memory_data_and_stack : 1138036736
   process_memory_resident : 158076928
   process_memory_shared : 33538048
   process_memory_text : 215818240
   process_memory_virtual : 1470058496
   process_nice : 0
   process_outblocks_second : 489
   process_priority : 20
   process_thread_count : 65
   process_uptime : 313.753962
   process_username : "unknown (No such device or address)"
   process_work_dir : "/data/app/nads-rankingdump"
   ranking_ad_num_avg : 3
   ranking_cache_hit_rate : 9943
   ranking_cache_miss_num : 5
   ranking_channel_init_times : 0
   ranking_default_response_num : 0
   ranking_fast_fail_num : 0
   ranking_get_ctx_fea_count : 87498
   ranking_get_ctx_fea_latency : 0.007
   ranking_get_ctx_fea_latency_80 : 0
   ranking_get_ctx_fea_latency_90 : 0
   ranking_get_ctx_fea_latency_99 : 0
   ranking_get_ctx_fea_latency_999 : 0
   ranking_get_ctx_fea_latency_9999 : 0
   ranking_get_ctx_fea_max_latency : 1
   ranking_get_ctx_fea_qps : 284
   ranking_get_fea_count : 87487
   ranking_get_fea_latency : 59
   ranking_get_fea_latency_80 : 0
   ranking_get_fea_latency_90 : 0
   ranking_get_fea_latency_99 : 0
   ranking_get_fea_latency_999 : 0
   ranking_get_fea_latency_9999 : 0
   ranking_get_fea_max_latency : 170
   ranking_get_fea_qps : 286
   ranking_get_item_fea_count : 87492
   ranking_get_item_fea_latency : 0.038
   ranking_get_item_fea_latency_80 : 0
   ranking_get_item_fea_latency_90 : 0
   ranking_get_item_fea_latency_99 : 0
   ranking_get_item_fea_latency_999 : 0
   ranking_get_item_fea_latency_9999 : 0
   ranking_get_item_fea_max_latency : 1
   ranking_get_item_fea_qps : 287
   ranking_get_user_fea_count : 87500
   ranking_get_user_fea_latency : 42
   ranking_get_user_fea_latency_80 : 0
   ranking_get_user_fea_latency_90 : 0
   ranking_get_user_fea_latency_99 : 0
   ranking_get_user_fea_latency_999 : 0
   ranking_get_user_fea_latency_9999 : 0
   ranking_get_user_fea_max_latency : 117
   ranking_get_user_fea_qps : 283
   ranking_idea_not_equal : 0
   ranking_idea_num : 0
   ranking_index_empty_num : 0
   ranking_index_less_20_num : 0
   ranking_index_less_20_rate : 0.000
   ranking_item_fea_null : 0
   ranking_item_fea_num_error : 0
   ranking_miss_model_num : 0
   ranking_model_load_failed_num : 0
   ranking_parse_error_num : 0
   ranking_predict_count : 0
   ranking_predict_error : 0
   ranking_predict_fail_num : 0
   ranking_predict_latency : 0.000
   ranking_predict_latency_80 : 0
   ranking_predict_latency_90 : 0
   ranking_predict_latency_99 : 0
   ranking_predict_latency_999 : 0
   ranking_predict_latency_9999 : 0
   ranking_predict_max_latency : 0
   ranking_predict_qps : 0
   ranking_predict_sucess_num : 0
   ranking_prepare_count : 0
   ranking_prepare_error : 0
   ranking_prepare_latency : 0.000
   ranking_prepare_latency_80 : 0
   ranking_prepare_latency_90 : 0
   ranking_prepare_latency_99 : 0
   ranking_prepare_latency_999 : 0
   ranking_prepare_latency_9999 : 0
   ranking_prepare_max_latency : 0
   ranking_prepare_qps : 0
   ranking_rank_count : 87483
   ranking_rank_latency : 99
   ranking_rank_latency_80 : 0
   ranking_rank_latency_90 : 0
   ranking_rank_latency_99 : 0
   ranking_rank_latency_999 : 0
   ranking_rank_latency_9999 : 0
   ranking_rank_max_latency : 247
   ranking_rank_qps : 285
   ranking_reply_empty_num : 0
   ranking_reply_error_num : 0
   ranking_req_num : 283
   ranking_result_error : 0
   ranking_score_1_exception : 0
   ranking_score_high_exception : 0
   ranking_score_neg_exception : 0
   ranking_update_fail_num : 0
   ranking_user_fea_empty_req : 283
   ranking_user_fea_req : 283
   ranking_user_fea_req_empty_rate : 10035
   rpc_channel_connection_count : 8
   rpc_compressions : "snappy gzip zlib"
   rpc_controller_count : 40
   rpc_event_thread_second : 387
   rpc_health_check_count : 27
   rpc_keepwrite_second : 2
   rpc_load_balancer : "c_md5 wr c_murmurhash wrr rr c_ketama random la"
   rpc_naming_service : "https consul redis file list http discovery dlist nacos remotefile"
   rpc_profilers : "contention"
   rpc_protocols : "baidu_std streaming_rpc hulu_pbrpc sofa_pbrpc rtmp http public_pbrpc nova_pbrpc redis nshead mongo ubrpc_compack memcache nshead_mcpack ubrpc_mcpack2 esp h2"
   rpc_revision : "undefined"
   rpc_server_10003_builtin_service_count : 19
   rpc_server_10003_connection_count : 89
   rpc_server_10003_eps : 0
   rpc_server_10003_error : 0
   rpc_server_10003_service_count : 1
   rpc_server_10003_start_time : "2023/05/22-16:32:19"
   rpc_server_10003_uptime : 313.417985
   rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_concurrency : 31
   rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_count : 87480
   rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_eps : 0
   rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_error : 0
   rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_latency : 57759
   rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_latency_80 : 109728
   rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_latency_90 : 138437
   rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_latency_99 : 198365
   rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_latency_999 : 240210
   rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_latency_9999 : 250536
   rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_max_latency : 283137
   rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_qps : 282
   rpc_server_count : 1
   rpc_socket_count : 99
   rpc_waitepollout_count : 0
   rpc_waitepollout_second : 0
   rpcz_sampling_ratio : 1.000
   system_core_count : 32
   system_loadavg_15m : 0.420
   system_loadavg_1m : 0.910
   system_loadavg_5m : 0.660
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1557312738

   看起来是DynamicPartitionChannel的bug


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "dandyhuang (via GitHub)" <gi...@apache.org>.
dandyhuang commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1647694086

   > DynamicPartitionChannel
   
   启多个channel呢


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556780754

   会增长,也会下降,最多到50了,然后又下来了


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1557066564

   看堆栈里有brpc::policy::CallMethodInBackupThread:
   <img width="912" alt="image" src="https://github.com/apache/brpc/assets/3894631/71b63bc1-8cb9-4106-89b8-eb45086a31f8">
   
   
   根据brpc 1.3.0的代码,CallMethodInBackupThread会在EndRunningCallMethodInPool中使用:
   <img width="747" alt="image" src="https://github.com/apache/brpc/assets/3894631/2a4ee35c-0e4a-4714-a85a-5e8b397c743c">
   
   
   EndRunningCallMethodInPool只会在FLAGS_usercode_in_pthread为true的时候走到:
   <img width="573" alt="image" src="https://github.com/apache/brpc/assets/3894631/4ddb472a-2eab-4459-a26e-f807a7b9cf73">
   
   
   所以FLAGS_usercode_in_pthread的值应该为true,太奇怪了。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556694580

   vm.max_map_count = 262144 这个值也不小了吧


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556695520

   而且brpc_max_concurrency设置的为5000


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556632361

   能贴一下死锁时各个线程的堆栈吗?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556674314

   看起来和 #1648  #644 是类似的问题,由于并发bthread数量过大导致mmap数量超限,无法创建更多bthread,降级到pthread上执行,导致死锁。可以尝试调大内核参数vm.max_map_count来支持更多的bthread


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556864760

   没有开启


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org


Re: [I] brpc死锁 (brpc)

Posted by "xiedeacc (via GitHub)" <gi...@apache.org>.
xiedeacc commented on issue #2259:
URL: https://github.com/apache/brpc/issues/2259#issuecomment-1556872662

   我的done是自己管理的,有快速拒绝机制,可能会瞬间把自己管理的队列一下子全部释放掉(会调done->Run())


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org