You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Chen Wang <ch...@gmail.com> on 2014/11/05 23:01:45 UTC
Consumer lag keep increasing
Hey Guys,
I have a really simply storm topology with a kafka spout, reading from
kafka through high level consumer. Since the topic has 30 partitions, we
have 30 threads in the spout reading from it. However, it seems that the
lag keeps increasing even the thread only read the message and do nothing.
The largest message size are around 30KB, and the incoming rate can be as
hight as 14k/seconds. There are 3 brokers on some high config bare metal
machines. The client side config is like this:
kafka.config.fetch.message.max.bytes 3145728
kafka.config.group.id spout_readonly
kafka.config.rebalance.backoff.ms 6000
kafka.config.rebalance.max.retries 6
kafka.config.zookeeper.connect dare-broker00.sv.walmartlabs.com:2181,
dare-broker01.sv.walmartlabs.com:2181,dare-broker02.sv.walmartlabs.com:2181
kafka.config.zookeeper.session.timeout.ms 60000
what could possibly cause this huge lag? Will broker be a bottle neck, or
some config need to be adjusted? The server side config is like this:
replica.fetch.max.bytes=2097152
message.max.bytes=2097152
num.network.threads=4
num.io.threads=4
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=4194304
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=2097152
# The maximum size of a request that the socket server will accept
(protection against OOM)
socket.request.max.bytes=104857600
Any help appreciated!
Chen
Re: Consumer lag keep increasing
Posted by Neha Narkhede <ne...@gmail.com>.
Chen,
Consumers lag either due to an I/O or network bottleneck or due to slow
processing of messages by the user. To confirm that you are not hitting the
latter issue, you can run a console consumer on the same data and observe
the throughput that it provides and it's lag.
Thanks,
Neha
On Wed, Nov 5, 2014 at 3:31 PM, Chen Wang <ch...@gmail.com>
wrote:
> Guozhang,
> I can see message keep coming, meaning messages are being consumed, right?
> But the lag is pretty huge (average 30m messages behind) as you can see
> from the graph:
>
> https://www.dropbox.com/s/xli25zicxv5f2qa/Screenshot%202014-11-05%2015.23.05.png?dl=0
>
> My understanding is that for such light weight thread, the consumer should
> almost be at the same pace with the producer. I also checked the machine
> metrics, and nothing pegged there.
>
> I am also moving the testing application to a separate dev cluster. In your
> experience, what things might cause the slow reading? Is this more like a
> server side thing, or consumer side?
>
> Chen
>
> On Wed, Nov 5, 2014 at 3:10 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Chen,
> >
> > Your configs seems fine.
> >
> > Could you use ConsumerOffsetChecker tool to see if the offset is
> advancing
> > at all (i.e. messages are comsumed), and if yes get some thread dumps and
> > check if your consumer is blocked on some locks?
> >
> > Guozhang
> >
> > On Wed, Nov 5, 2014 at 2:01 PM, Chen Wang <ch...@gmail.com>
> > wrote:
> >
> > > Hey Guys,
> > > I have a really simply storm topology with a kafka spout, reading from
> > > kafka through high level consumer. Since the topic has 30 partitions,
> we
> > > have 30 threads in the spout reading from it. However, it seems that
> the
> > > lag keeps increasing even the thread only read the message and do
> > nothing.
> > > The largest message size are around 30KB, and the incoming rate can be
> > as
> > > hight as 14k/seconds. There are 3 brokers on some high config bare
> metal
> > > machines. The client side config is like this:
> > >
> > > kafka.config.fetch.message.max.bytes 3145728
> > > kafka.config.group.id spout_readonly
> > > kafka.config.rebalance.backoff.ms 6000
> > > kafka.config.rebalance.max.retries 6
> > > kafka.config.zookeeper.connect dare-broker00.sv.walmartlabs.com:2181,
> > > dare-broker01.sv.walmartlabs.com:2181,
> > > dare-broker02.sv.walmartlabs.com:2181
> > > kafka.config.zookeeper.session.timeout.ms 60000
> > >
> > > what could possibly cause this huge lag? Will broker be a bottle neck,
> or
> > > some config need to be adjusted? The server side config is like this:
> > >
> > > replica.fetch.max.bytes=2097152
> > > message.max.bytes=2097152
> > > num.network.threads=4
> > > num.io.threads=4
> > >
> > > # The send buffer (SO_SNDBUF) used by the socket server
> > > socket.send.buffer.bytes=4194304
> > >
> > > # The receive buffer (SO_RCVBUF) used by the socket server
> > > socket.receive.buffer.bytes=2097152
> > >
> > > # The maximum size of a request that the socket server will accept
> > > (protection against OOM)
> > > socket.request.max.bytes=104857600
> > >
> > > Any help appreciated!
> > > Chen
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
Re: Consumer lag keep increasing
Posted by Chen Wang <ch...@gmail.com>.
Guozhang,
I can see message keep coming, meaning messages are being consumed, right?
But the lag is pretty huge (average 30m messages behind) as you can see
from the graph:
https://www.dropbox.com/s/xli25zicxv5f2qa/Screenshot%202014-11-05%2015.23.05.png?dl=0
My understanding is that for such light weight thread, the consumer should
almost be at the same pace with the producer. I also checked the machine
metrics, and nothing pegged there.
I am also moving the testing application to a separate dev cluster. In your
experience, what things might cause the slow reading? Is this more like a
server side thing, or consumer side?
Chen
On Wed, Nov 5, 2014 at 3:10 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Chen,
>
> Your configs seems fine.
>
> Could you use ConsumerOffsetChecker tool to see if the offset is advancing
> at all (i.e. messages are comsumed), and if yes get some thread dumps and
> check if your consumer is blocked on some locks?
>
> Guozhang
>
> On Wed, Nov 5, 2014 at 2:01 PM, Chen Wang <ch...@gmail.com>
> wrote:
>
> > Hey Guys,
> > I have a really simply storm topology with a kafka spout, reading from
> > kafka through high level consumer. Since the topic has 30 partitions, we
> > have 30 threads in the spout reading from it. However, it seems that the
> > lag keeps increasing even the thread only read the message and do
> nothing.
> > The largest message size are around 30KB, and the incoming rate can be
> as
> > hight as 14k/seconds. There are 3 brokers on some high config bare metal
> > machines. The client side config is like this:
> >
> > kafka.config.fetch.message.max.bytes 3145728
> > kafka.config.group.id spout_readonly
> > kafka.config.rebalance.backoff.ms 6000
> > kafka.config.rebalance.max.retries 6
> > kafka.config.zookeeper.connect dare-broker00.sv.walmartlabs.com:2181,
> > dare-broker01.sv.walmartlabs.com:2181,
> > dare-broker02.sv.walmartlabs.com:2181
> > kafka.config.zookeeper.session.timeout.ms 60000
> >
> > what could possibly cause this huge lag? Will broker be a bottle neck, or
> > some config need to be adjusted? The server side config is like this:
> >
> > replica.fetch.max.bytes=2097152
> > message.max.bytes=2097152
> > num.network.threads=4
> > num.io.threads=4
> >
> > # The send buffer (SO_SNDBUF) used by the socket server
> > socket.send.buffer.bytes=4194304
> >
> > # The receive buffer (SO_RCVBUF) used by the socket server
> > socket.receive.buffer.bytes=2097152
> >
> > # The maximum size of a request that the socket server will accept
> > (protection against OOM)
> > socket.request.max.bytes=104857600
> >
> > Any help appreciated!
> > Chen
> >
>
>
>
> --
> -- Guozhang
>
Re: Consumer lag keep increasing
Posted by Guozhang Wang <wa...@gmail.com>.
Chen,
Your configs seems fine.
Could you use ConsumerOffsetChecker tool to see if the offset is advancing
at all (i.e. messages are comsumed), and if yes get some thread dumps and
check if your consumer is blocked on some locks?
Guozhang
On Wed, Nov 5, 2014 at 2:01 PM, Chen Wang <ch...@gmail.com>
wrote:
> Hey Guys,
> I have a really simply storm topology with a kafka spout, reading from
> kafka through high level consumer. Since the topic has 30 partitions, we
> have 30 threads in the spout reading from it. However, it seems that the
> lag keeps increasing even the thread only read the message and do nothing.
> The largest message size are around 30KB, and the incoming rate can be as
> hight as 14k/seconds. There are 3 brokers on some high config bare metal
> machines. The client side config is like this:
>
> kafka.config.fetch.message.max.bytes 3145728
> kafka.config.group.id spout_readonly
> kafka.config.rebalance.backoff.ms 6000
> kafka.config.rebalance.max.retries 6
> kafka.config.zookeeper.connect dare-broker00.sv.walmartlabs.com:2181,
> dare-broker01.sv.walmartlabs.com:2181,
> dare-broker02.sv.walmartlabs.com:2181
> kafka.config.zookeeper.session.timeout.ms 60000
>
> what could possibly cause this huge lag? Will broker be a bottle neck, or
> some config need to be adjusted? The server side config is like this:
>
> replica.fetch.max.bytes=2097152
> message.max.bytes=2097152
> num.network.threads=4
> num.io.threads=4
>
> # The send buffer (SO_SNDBUF) used by the socket server
> socket.send.buffer.bytes=4194304
>
> # The receive buffer (SO_RCVBUF) used by the socket server
> socket.receive.buffer.bytes=2097152
>
> # The maximum size of a request that the socket server will accept
> (protection against OOM)
> socket.request.max.bytes=104857600
>
> Any help appreciated!
> Chen
>
--
-- Guozhang