You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Sumudu Madushanka <ng...@cse.mrt.ac.lk> on 2022/06/07 15:41:59 UTC
Impala RPC Reactor 227 Bottleneck
Hi all,
We are using kudu 1.15.0, and impala 3.4.0-RELEASE
In our impala select query, we are scanning *10 kudu partitions/tablets*
per impala daemon host in *6 node cluster* with *16 cores* per machine (
*c5.4xlarge*).
The kudu threads (around 9 threads) are working in parallel to do the
scanning as in given in fig_2. But, in Impala *only one reactor thread (rpc
reactor-227 thread)* is working and it’s at* 99.99%* which creates a
bottleneck(fig_1). There are other reactor threads at idle. Please note
that the query results are not cached to disk.
Here are our questions.
1. Why only one rpc reactor thread is working and creating a bottleneck
for multi-tablets scan query in an impala daemon host and why is it always *rpc
reactor-227*? (We also suspect that this could be the rpc reactor thread
that runs inside kudu client inside the impala daemon)
2. How can we solve this issue for other rpc reactor threads to work as
well?
This is creating a huge bottleneck in the query scanning. Could there be
other reasons why this behavior occurs?
Appreciate your help!
gstack 22788
Thread 1 (process 22788):
#0 0x00007f15a5ebeaeb in recv () from /usr/lib64/libpthread.so.0
#1 0x00007f15a42763b4 in kudu::Socket::Recv (this=0xd5588f0, buf=0x17324004
"\f\b\325\300\b\020", amt=1185345, nread=0x7f14c81d9940) at
/mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/util/net/socket.cc:517
#2 0x00007f15a419172e in kudu::rpc::InboundTransfer::ReceiveBuffer
(this=0xd507840, socket=...) at
/mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/transfer.cc:144
#3 0x00007f15a4182fd0 in ReadHandler (revents=<optimized out>, watcher=...,
this=0xdb56000) at
/mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/connection.cc:653
#4 ev::base<ev_io, ev::io>::method_thunk<kudu::rpc::Connection,
&kudu::rpc::Connection::ReadHandler> (loop=<optimized out>, w=<optimized
out>, revents=<optimized out>) at
/mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/ev++.h:479
#5 0x00007f15a43c1f1b in ev_invoke_pending (loop=0xd552480) at
/mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/src/libev-4.20/ev.c:3155
#6 0x00007f15a41611dd in kudu::rpc::ReactorThread::InvokePendingCb
(loop=0xd552480) at
/mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/reactor.cc:196
#7 0x00007f15a43c55f4 in ev_run (loop=0xd552480, flags=<optimized out>) at
/mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/src/libev-4.20/ev.c:3555
#8 0x00007f15a41617ab in run (flags=0, this=0x9c03088) at
/mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/ev++.h:211
#9 kudu::rpc::ReactorThread::RunThread (this=0x9c03080) at
/mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/reactor.cc:497
#10 0x00007f15a429df06 in operator() (this=0x9ffdb28) at
/mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:771
#11 kudu::Thread::SuperviseThread (arg=0x9ffdb00) at
/mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/util/thread.cc:675
#12 0x00007f15a5eb7ea5 in start_thread () from /usr/lib64/libpthread.so.0
#13 0x00007f15a252b9fd in clone () from /usr/lib64/libc.so.6
Thank you
Best Regards,
*Sumudu Madushanka*
Re: Impala RPC Reactor 227 Bottleneck
Posted by Sumudu Madushanka <ng...@cse.mrt.ac.lk>.
Hi all,
Link for fig_1 and fig_2:
https://drive.google.com/drive/folders/1w77EZTQp-vUcoYfIIaIBN3mZL4HS32M8?usp=sharing
Thank you,
Regards
*Sumudu Madushanka*
On Fri, 10 Jun 2022 at 09:01, Quanlong Huang <hu...@gmail.com>
wrote:
> Hi Sumudu,
>
> Thanks for sharing your findings! Unfortunately, the attachments (fig_1 and
> fig_2) are missing. They were probably ignored by the mailing list. Could
> you upload them to Google Drive and share the links?
>
> BTW, is it the same query you shared at this thread?
> https://lists.apache.org/thread/dstobv61xjhm18vddv264s66sx2oh22w
>
> I think we need some time to dig into the details. Not sure if the issue
> still occurs in 4.1.0.
>
> Thanks,
> Quanlong
>
> On Tue, Jun 7, 2022 at 11:42 PM Sumudu Madushanka <
> ngsmadushanka.16@cse.mrt.ac.lk> wrote:
>
> > Hi all,
> >
> > We are using kudu 1.15.0, and impala 3.4.0-RELEASE
> >
> > In our impala select query, we are scanning *10 kudu partitions/tablets*
> > per impala daemon host in *6 node cluster* with *16 cores* per machine (
> > *c5.4xlarge*).
> >
> > The kudu threads (around 9 threads) are working in parallel to do the
> > scanning as in given in fig_2. But, in Impala *only one reactor thread
> > (rpc reactor-227 thread)* is working and it’s at* 99.99%* which creates a
> > bottleneck(fig_1). There are other reactor threads at idle. Please note
> > that the query results are not cached to disk.
> >
> > Here are our questions.
> >
> > 1. Why only one rpc reactor thread is working and creating a bottleneck
> > for multi-tablets scan query in an impala daemon host and why is it
> always *rpc
> > reactor-227*? (We also suspect that this could be the rpc reactor thread
> > that runs inside kudu client inside the impala daemon)
> >
> > 2. How can we solve this issue for other rpc reactor threads to work as
> > well?
> >
> > This is creating a huge bottleneck in the query scanning. Could there be
> > other reasons why this behavior occurs?
> >
> > Appreciate your help!
> >
> > gstack 22788
> > Thread 1 (process 22788):
> > #0 0x00007f15a5ebeaeb in recv () from /usr/lib64/libpthread.so.0
> > #1 0x00007f15a42763b4 in kudu::Socket::Recv (this=0xd5588f0,
> > buf=0x17324004 "\f\b\325\300\b\020", amt=1185345, nread=0x7f14c81d9940)
> at
> > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/util/net/socket.cc:517
> > #2 0x00007f15a419172e in kudu::rpc::InboundTransfer::ReceiveBuffer
> > (this=0xd507840, socket=...) at
> > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/transfer.cc:144
> > #3 0x00007f15a4182fd0 in ReadHandler (revents=<optimized out>,
> > watcher=..., this=0xdb56000) at
> > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/connection.cc:653
> > #4 ev::base<ev_io, ev::io>::method_thunk<kudu::rpc::Connection,
> > &kudu::rpc::Connection::ReadHandler> (loop=<optimized out>, w=<optimized
> > out>, revents=<optimized out>) at
> >
> /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/ev++.h:479
> > #5 0x00007f15a43c1f1b in ev_invoke_pending (loop=0xd552480) at
> > /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/src/libev-4.20/ev.c:3155
> > #6 0x00007f15a41611dd in kudu::rpc::ReactorThread::InvokePendingCb
> > (loop=0xd552480) at
> > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/reactor.cc:196
> > #7 0x00007f15a43c55f4 in ev_run (loop=0xd552480, flags=<optimized out>)
> at
> > /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/src/libev-4.20/ev.c:3555
> > #8 0x00007f15a41617ab in run (flags=0, this=0x9c03088) at
> >
> /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/ev++.h:211
> > #9 kudu::rpc::ReactorThread::RunThread (this=0x9c03080) at
> > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/reactor.cc:497
> > #10 0x00007f15a429df06 in operator() (this=0x9ffdb28) at
> >
> /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:771
> > #11 kudu::Thread::SuperviseThread (arg=0x9ffdb00) at
> > /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/util/thread.cc:675
> > #12 0x00007f15a5eb7ea5 in start_thread () from /usr/lib64/libpthread.so.0
> > #13 0x00007f15a252b9fd in clone () from /usr/lib64/libc.so.6
> >
> >
> >
> > Thank you
> >
> >
> >
> > Best Regards,
> > *Sumudu Madushanka*
> >
>
Re: Impala RPC Reactor 227 Bottleneck
Posted by Quanlong Huang <hu...@gmail.com>.
Hi Sumudu,
Thanks for sharing your findings! Unfortunately, the attachments (fig_1 and
fig_2) are missing. They were probably ignored by the mailing list. Could
you upload them to Google Drive and share the links?
BTW, is it the same query you shared at this thread?
https://lists.apache.org/thread/dstobv61xjhm18vddv264s66sx2oh22w
I think we need some time to dig into the details. Not sure if the issue
still occurs in 4.1.0.
Thanks,
Quanlong
On Tue, Jun 7, 2022 at 11:42 PM Sumudu Madushanka <
ngsmadushanka.16@cse.mrt.ac.lk> wrote:
> Hi all,
>
> We are using kudu 1.15.0, and impala 3.4.0-RELEASE
>
> In our impala select query, we are scanning *10 kudu partitions/tablets*
> per impala daemon host in *6 node cluster* with *16 cores* per machine (
> *c5.4xlarge*).
>
> The kudu threads (around 9 threads) are working in parallel to do the
> scanning as in given in fig_2. But, in Impala *only one reactor thread
> (rpc reactor-227 thread)* is working and it’s at* 99.99%* which creates a
> bottleneck(fig_1). There are other reactor threads at idle. Please note
> that the query results are not cached to disk.
>
> Here are our questions.
>
> 1. Why only one rpc reactor thread is working and creating a bottleneck
> for multi-tablets scan query in an impala daemon host and why is it always *rpc
> reactor-227*? (We also suspect that this could be the rpc reactor thread
> that runs inside kudu client inside the impala daemon)
>
> 2. How can we solve this issue for other rpc reactor threads to work as
> well?
>
> This is creating a huge bottleneck in the query scanning. Could there be
> other reasons why this behavior occurs?
>
> Appreciate your help!
>
> gstack 22788
> Thread 1 (process 22788):
> #0 0x00007f15a5ebeaeb in recv () from /usr/lib64/libpthread.so.0
> #1 0x00007f15a42763b4 in kudu::Socket::Recv (this=0xd5588f0,
> buf=0x17324004 "\f\b\325\300\b\020", amt=1185345, nread=0x7f14c81d9940) at
> /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/util/net/socket.cc:517
> #2 0x00007f15a419172e in kudu::rpc::InboundTransfer::ReceiveBuffer
> (this=0xd507840, socket=...) at
> /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/transfer.cc:144
> #3 0x00007f15a4182fd0 in ReadHandler (revents=<optimized out>,
> watcher=..., this=0xdb56000) at
> /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/connection.cc:653
> #4 ev::base<ev_io, ev::io>::method_thunk<kudu::rpc::Connection,
> &kudu::rpc::Connection::ReadHandler> (loop=<optimized out>, w=<optimized
> out>, revents=<optimized out>) at
> /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/ev++.h:479
> #5 0x00007f15a43c1f1b in ev_invoke_pending (loop=0xd552480) at
> /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/src/libev-4.20/ev.c:3155
> #6 0x00007f15a41611dd in kudu::rpc::ReactorThread::InvokePendingCb
> (loop=0xd552480) at
> /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/reactor.cc:196
> #7 0x00007f15a43c55f4 in ev_run (loop=0xd552480, flags=<optimized out>) at
> /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/src/libev-4.20/ev.c:3555
> #8 0x00007f15a41617ab in run (flags=0, this=0x9c03088) at
> /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/ev++.h:211
> #9 kudu::rpc::ReactorThread::RunThread (this=0x9c03080) at
> /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/rpc/reactor.cc:497
> #10 0x00007f15a429df06 in operator() (this=0x9ffdb28) at
> /mnt/source/kudu/kudu-4ed0dbbd1/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:771
> #11 kudu::Thread::SuperviseThread (arg=0x9ffdb00) at
> /mnt/source/kudu/kudu-4ed0dbbd1/src/kudu/util/thread.cc:675
> #12 0x00007f15a5eb7ea5 in start_thread () from /usr/lib64/libpthread.so.0
> #13 0x00007f15a252b9fd in clone () from /usr/lib64/libc.so.6
>
>
>
> Thank you
>
>
>
> Best Regards,
> *Sumudu Madushanka*
>