You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Sybrandy, Casey" <Ca...@Six3Systems.com> on 2013/08/01 16:30:16 UTC
Consumer pauses when running many threads
Hello,
We're seeing an issue running 0.7.0 where one or more of our consumers are pausing after about an hour when we have a lot of threads configured. Our setup is as follows:
* 4 brokers configured for 32 threads and 32 partitions on each broker.
* 2 consumers each processing 40 streams (24 and 16).
* Zookeeper server is a CDH version that's at least 3.3.4.
We were also seeing this with 3 consumers running 18 threads each. As you can tell, the hardware is quite beefy and the brokers are described as being "bored."
Outside of upgrading to 0.7.2, which we are planning on doing but can't yet, what else can we look into to try to resolve this or at least determine what's happening?
Thanks.
Casey
Re: Consumer pauses when running many threads
Posted by Jun Rao <ju...@gmail.com>.
That's right. In 0.7, # partitions is per broker. However, in 0.8, #
partitions is for the whole cluster.
Thanks,
Jun
On Fri, Aug 2, 2013 at 8:13 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> Yes, we have. Our SA where this is occurring has been monitoring this.
> When the consumers went down, we could see that things were lagging.
> Yesterday, they lowered the number of threads for the consumers to six
> each and they haven't shut down yet. There appears to still be some lag,
> but since the consumers are running, it's decreasing.
>
> A test was run with each broker configured to have 32 partitions each and
> when the number of threads across the consumers exceeds 32, then we have
> issues. My understanding from the documentation is that when you set the
> number of partitions on a broker, it's just for that broker, correct?
> Therefore, if we set each broker to have 32 partitions, across 4 brokers
> we should have 128 partitions per topic, correct? In which case, we should
> be able to run 128 consumer threads with ease.
>
> Casey
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Thursday, August 01, 2013 11:13 AM
> To: users@kafka.apache.org
> Subject: Re: Consumer pauses when running many threads
>
> Have you looked at
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F
> ?
>
> Thanks,
>
> Jun
>
>
> On Thu, Aug 1, 2013 at 7:30 AM, Sybrandy, Casey <
> Casey.Sybrandy@six3systems.com> wrote:
>
> > Hello,
> >
> > We're seeing an issue running 0.7.0 where one or more of our consumers
> > are pausing after about an hour when we have a lot of threads
> > configured. Our setup is as follows:
> >
> >
> > * 4 brokers configured for 32 threads and 32 partitions on each
> > broker.
> >
> > * 2 consumers each processing 40 streams (24 and 16).
> >
> > * Zookeeper server is a CDH version that's at least 3.3.4.
> >
> > We were also seeing this with 3 consumers running 18 threads each. As
> > you can tell, the hardware is quite beefy and the brokers are
> > described as being "bored."
> >
> > Outside of upgrading to 0.7.2, which we are planning on doing but
> > can't yet, what else can we look into to try to resolve this or at
> > least determine what's happening?
> >
> > Thanks.
> >
> > Casey
> >
>
RE: Consumer pauses when running many threads
Posted by "Sybrandy, Casey" <Ca...@Six3Systems.com>.
Yes, we have. Our SA where this is occurring has been monitoring this. When the consumers went down, we could see that things were lagging. Yesterday, they lowered the number of threads for the consumers to six each and they haven't shut down yet. There appears to still be some lag, but since the consumers are running, it's decreasing.
A test was run with each broker configured to have 32 partitions each and when the number of threads across the consumers exceeds 32, then we have issues. My understanding from the documentation is that when you set the number of partitions on a broker, it's just for that broker, correct? Therefore, if we set each broker to have 32 partitions, across 4 brokers we should have 128 partitions per topic, correct? In which case, we should be able to run 128 consumer threads with ease.
Casey
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Thursday, August 01, 2013 11:13 AM
To: users@kafka.apache.org
Subject: Re: Consumer pauses when running many threads
Have you looked at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F?
Thanks,
Jun
On Thu, Aug 1, 2013 at 7:30 AM, Sybrandy, Casey < Casey.Sybrandy@six3systems.com> wrote:
> Hello,
>
> We're seeing an issue running 0.7.0 where one or more of our consumers
> are pausing after about an hour when we have a lot of threads
> configured. Our setup is as follows:
>
>
> * 4 brokers configured for 32 threads and 32 partitions on each
> broker.
>
> * 2 consumers each processing 40 streams (24 and 16).
>
> * Zookeeper server is a CDH version that's at least 3.3.4.
>
> We were also seeing this with 3 consumers running 18 threads each. As
> you can tell, the hardware is quite beefy and the brokers are
> described as being "bored."
>
> Outside of upgrading to 0.7.2, which we are planning on doing but
> can't yet, what else can we look into to try to resolve this or at
> least determine what's happening?
>
> Thanks.
>
> Casey
>
Re: Consumer pauses when running many threads
Posted by Jun Rao <ju...@gmail.com>.
Have you looked at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F?
Thanks,
Jun
On Thu, Aug 1, 2013 at 7:30 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:
> Hello,
>
> We're seeing an issue running 0.7.0 where one or more of our consumers are
> pausing after about an hour when we have a lot of threads configured. Our
> setup is as follows:
>
>
> * 4 brokers configured for 32 threads and 32 partitions on each
> broker.
>
> * 2 consumers each processing 40 streams (24 and 16).
>
> * Zookeeper server is a CDH version that's at least 3.3.4.
>
> We were also seeing this with 3 consumers running 18 threads each. As you
> can tell, the hardware is quite beefy and the brokers are described as
> being "bored."
>
> Outside of upgrading to 0.7.2, which we are planning on doing but can't
> yet, what else can we look into to try to resolve this or at least
> determine what's happening?
>
> Thanks.
>
> Casey
>