Re: Problem consuming from broker 1.1.0

[SenthilKumar K <se...@gmail.com>]

Hi Craig Ching,

Reg. *We did end up turning on debug logs for the console consumer and
found that one broker seemed to be having problems, it would lead to
timeouts communicating with it.  After restarting that broker, things
sorted themselves out.*

We had similar problem on prod cluster and i'm trying to figure out the
root cause for why broker stopped responding ? Pls check my email subject :
*"**Kafka Broker Not Responding" *where i described problem in detail.

Curious to know , Were you able to figure out reason for broker failure ?.
Of Course , turning off/on is not the ideal solution.

--Senthil


On Thu, Jun 14, 2018 at 9:48 AM, Craig Ching <cr...@gmail.com> wrote:

> Hi Manikumar!
>
> Thanks for responding!  Sorry it took me so long to get back!
>
> We did end up turning on debug logs for the console consumer and found
> that one broker seemed to be having problems, it would lead to timeouts
> communicating with it.  After restarting that broker, things sorted
> themselves out. However, I always hate the “turn off/ turn on” solution ;)
> It’s interesting to me that the 1.1.0 consumer, though it reported timeouts
> in the logs, never had a problem, it seemed able to recover.  Whereas the
> 1.0.1 consumer (talking to a 1.1.0 cluster remember) couldn’t recover.
> Does any of this make sense?  I’m happy to provide more details and logs if
> necessary as I’d like to understand the root problem here.
>
> Thanks again!
>
> Cheers,
> Craig
>
> > On Jun 13, 2018, at 12:43 AM, Manikumar <ma...@gmail.com>
> wrote:
> >
> > Can you post consumer debug logs?
> > You can enable console consumer debug logs here:
> > kafka/config/tools-log4j.properties
> >
> > On Wed, Jun 13, 2018 at 9:55 AM Craig Ching <cr...@gmail.com>
> wrote:
> >
> >> Hi!
> >>
> >> We’re having a problem with a new kafka cluster at 1.1.0.  The problem
> is,
> >> in general, that consumers can’t consume from the different broker (old
> >> broker was 0.11 I think).  The easiest recipe I have for reproducing the
> >> problem is that downloading kafka 1.0.1 and running console consumer
> can’t
> >> consume from the 1.1.0 cluster while a 1.1.0 console consumer can.
> We’re
> >> invoking console consumer like this:
> >>
> >> bin/kafka-console-consumer.sh \
> >>    —bootstrap-server [kafka server] \
> >>    —topic [our topic] \
> >>    —max-messages 3
> >>
> >> That works for a 1.1.0 console consumer, but not for 1.0.1.  However, if
> >> we change that to:
> >>
> >> bin/kafka-console-consumer.sh \
> >>    —zookeeper [zookeeper server] \
> >>    —topic [our topic] \
> >>    —max-messages 3
> >>
> >> Then it works for 1.0.1.
> >>
> >> I was wondering, is the zookeeper schema published for 1.1.0?  I have a
> >> feeling that maybe something is wrong in zookeeper and I know earlier
> >> versions of kafka used to publish the zk schema, could there be a
> problem
> >> in zk?  if so, what might I look for?
> >>
> >> Any help is greatly appreciated!
> >>
> >> Cheers,
> >> Craig
>
>

Re: Problem consuming from broker 1.1.0

[Craig Ching <cr...@gmail.com>]

Hi Senthil,

We never did get to the root cause unfortunately.  Our kafka cluster is
currently owned by a different team and it wasn't a priority for them to
figure that out, or at least communicate the problem to us :(

Cheers,
Craig

On Thu, Jul 19, 2018 at 8:13 AM SenthilKumar K <se...@gmail.com>
wrote:

> Hi Craig Ching,
>
> Reg. *We did end up turning on debug logs for the console consumer and
> found that one broker seemed to be having problems, it would lead to
> timeouts communicating with it.  After restarting that broker, things
> sorted themselves out.*
>
> We had similar problem on prod cluster and i'm trying to figure out the
> root cause for why broker stopped responding ? Pls check my email subject :
> *"**Kafka Broker Not Responding" *where i described problem in detail.
>
> Curious to know , Were you able to figure out reason for broker failure ?.
> Of Course , turning off/on is not the ideal solution.
>
> --Senthil
>
>
> On Thu, Jun 14, 2018 at 9:48 AM, Craig Ching <cr...@gmail.com> wrote:
>
> > Hi Manikumar!
> >
> > Thanks for responding!  Sorry it took me so long to get back!
> >
> > We did end up turning on debug logs for the console consumer and found
> > that one broker seemed to be having problems, it would lead to timeouts
> > communicating with it.  After restarting that broker, things sorted
> > themselves out. However, I always hate the “turn off/ turn on” solution
> ;)
> > It’s interesting to me that the 1.1.0 consumer, though it reported
> timeouts
> > in the logs, never had a problem, it seemed able to recover.  Whereas the
> > 1.0.1 consumer (talking to a 1.1.0 cluster remember) couldn’t recover.
> > Does any of this make sense?  I’m happy to provide more details and logs
> if
> > necessary as I’d like to understand the root problem here.
> >
> > Thanks again!
> >
> > Cheers,
> > Craig
> >
> > > On Jun 13, 2018, at 12:43 AM, Manikumar <ma...@gmail.com>
> > wrote:
> > >
> > > Can you post consumer debug logs?
> > > You can enable console consumer debug logs here:
> > > kafka/config/tools-log4j.properties
> > >
> > > On Wed, Jun 13, 2018 at 9:55 AM Craig Ching <cr...@gmail.com>
> > wrote:
> > >
> > >> Hi!
> > >>
> > >> We’re having a problem with a new kafka cluster at 1.1.0.  The problem
> > is,
> > >> in general, that consumers can’t consume from the different broker
> (old
> > >> broker was 0.11 I think).  The easiest recipe I have for reproducing
> the
> > >> problem is that downloading kafka 1.0.1 and running console consumer
> > can’t
> > >> consume from the 1.1.0 cluster while a 1.1.0 console consumer can.
> > We’re
> > >> invoking console consumer like this:
> > >>
> > >> bin/kafka-console-consumer.sh \
> > >>    —bootstrap-server [kafka server] \
> > >>    —topic [our topic] \
> > >>    —max-messages 3
> > >>
> > >> That works for a 1.1.0 console consumer, but not for 1.0.1.  However,
> if
> > >> we change that to:
> > >>
> > >> bin/kafka-console-consumer.sh \
> > >>    —zookeeper [zookeeper server] \
> > >>    —topic [our topic] \
> > >>    —max-messages 3
> > >>
> > >> Then it works for 1.0.1.
> > >>
> > >> I was wondering, is the zookeeper schema published for 1.1.0?  I have
> a
> > >> feeling that maybe something is wrong in zookeeper and I know earlier
> > >> versions of kafka used to publish the zk schema, could there be a
> > problem
> > >> in zk?  if so, what might I look for?
> > >>
> > >> Any help is greatly appreciated!
> > >>
> > >> Cheers,
> > >> Craig
> >
> >
>