You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Srinivas, Kaushik (Nokia - IN/Bangalore)" <ka...@nokia.com> on 2019/06/03 11:52:13 UTC
kafka connect stops consuming data when kafka broker goes down
Hello kafka dev,
We are encountering an issue when kafka connect is running hdfs sink connector pulling data from kafka and writing to hdfs location.
In between when the data is flowing in the pipeline from producer -> kafka topic -> kafka connect hdfs sink connector -> hdfs,
If even one of the kafka broker goes down, the connect framework stops responding. Stops consuming records and REST API also becomes not interactive.
Until the kafka connect framework is restarted, it would not pull the data from kafka and REST api remains inactive. Nothing is coming in the logs as well.
Checked the topics in kafka used by connect, everything has been reassigned to another broker and has the leader available.
Has anyone encountered this issue ? what would be the expected behavior ?
Thanks in advance
Kaushik
RE: kafka connect stops consuming data when kafka broker goes down
Posted by "Srinivas, Kaushik (Nokia - IN/Bangalore)" <ka...@nokia.com>.
Hi Paul,
We tried patching out kafka connect as well referencing the PR which you had shared. But no improvement in our case.
Have updated more details in the blocker issue which I have created for the same.
https://issues.apache.org/jira/browse/KAFKA-8485
But the issue seems to be coming from the internal producers/consumers getting into hung state after the broker restarts.
Any other lead/inputs would be useful.
This issue is always reproducible even with/without data streaming in progress.
Thanks,
Kaushik
-----Original Message-----
From: Paul Whalen <pg...@gmail.com>
Sent: Wednesday, June 05, 2019 5:17 PM
To: dev@kafka.apache.org
Cc: Basil Brito, Aldan (Nokia - IN/Bangalore) <al...@nokia.com>
Subject: Re: kafka connect stops consuming data when kafka broker goes down
It’s not totally clear, but this may be https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-7941
For which there is a fix that is very nearly approved: https://github.com/apache/kafka/pull/6283
Paul
> On Jun 5, 2019, at 1:26 AM, Srinivas, Kaushik (Nokia - IN/Bangalore) <ka...@nokia.com> wrote:
>
> Hello,
> Anyone has any information on this issue.
> Created a critical ticket for the same, since this is a major stability issue for connect framework.
> https://issues.apache.org/jira/browse/KAFKA-8485?filter=-2
>
> Thanks.
> Kaushik,
> NOKIA
>
> From: Srinivas, Kaushik (Nokia - IN/Bangalore)
> Sent: Monday, June 03, 2019 5:22 PM
> To: dev@kafka.apache.org
> Cc: Basil Brito, Aldan (Nokia - IN/Bangalore)
> <al...@nokia.com>
> Subject: kafka connect stops consuming data when kafka broker goes
> down
>
> Hello kafka dev,
>
> We are encountering an issue when kafka connect is running hdfs sink connector pulling data from kafka and writing to hdfs location.
> In between when the data is flowing in the pipeline from producer ->
> kafka topic -> kafka connect hdfs sink connector -> hdfs, If even one of the kafka broker goes down, the connect framework stops responding. Stops consuming records and REST API also becomes not interactive.
>
> Until the kafka connect framework is restarted, it would not pull the data from kafka and REST api remains inactive. Nothing is coming in the logs as well.
> Checked the topics in kafka used by connect, everything has been reassigned to another broker and has the leader available.
>
> Has anyone encountered this issue ? what would be the expected behavior ?
>
> Thanks in advance
> Kaushik
Re: kafka connect stops consuming data when kafka broker goes down
Posted by Paul Whalen <pg...@gmail.com>.
It’s not totally clear, but this may be https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-7941
For which there is a fix that is very nearly approved: https://github.com/apache/kafka/pull/6283
Paul
> On Jun 5, 2019, at 1:26 AM, Srinivas, Kaushik (Nokia - IN/Bangalore) <ka...@nokia.com> wrote:
>
> Hello,
> Anyone has any information on this issue.
> Created a critical ticket for the same, since this is a major stability issue for connect framework.
> https://issues.apache.org/jira/browse/KAFKA-8485?filter=-2
>
> Thanks.
> Kaushik,
> NOKIA
>
> From: Srinivas, Kaushik (Nokia - IN/Bangalore)
> Sent: Monday, June 03, 2019 5:22 PM
> To: dev@kafka.apache.org
> Cc: Basil Brito, Aldan (Nokia - IN/Bangalore) <al...@nokia.com>
> Subject: kafka connect stops consuming data when kafka broker goes down
>
> Hello kafka dev,
>
> We are encountering an issue when kafka connect is running hdfs sink connector pulling data from kafka and writing to hdfs location.
> In between when the data is flowing in the pipeline from producer -> kafka topic -> kafka connect hdfs sink connector -> hdfs,
> If even one of the kafka broker goes down, the connect framework stops responding. Stops consuming records and REST API also becomes not interactive.
>
> Until the kafka connect framework is restarted, it would not pull the data from kafka and REST api remains inactive. Nothing is coming in the logs as well.
> Checked the topics in kafka used by connect, everything has been reassigned to another broker and has the leader available.
>
> Has anyone encountered this issue ? what would be the expected behavior ?
>
> Thanks in advance
> Kaushik
RE: kafka connect stops consuming data when kafka broker goes down
Posted by "Srinivas, Kaushik (Nokia - IN/Bangalore)" <ka...@nokia.com>.
Hello,
Anyone has any information on this issue.
Created a critical ticket for the same, since this is a major stability issue for connect framework.
https://issues.apache.org/jira/browse/KAFKA-8485?filter=-2
Thanks.
Kaushik,
NOKIA
From: Srinivas, Kaushik (Nokia - IN/Bangalore)
Sent: Monday, June 03, 2019 5:22 PM
To: dev@kafka.apache.org
Cc: Basil Brito, Aldan (Nokia - IN/Bangalore) <al...@nokia.com>
Subject: kafka connect stops consuming data when kafka broker goes down
Hello kafka dev,
We are encountering an issue when kafka connect is running hdfs sink connector pulling data from kafka and writing to hdfs location.
In between when the data is flowing in the pipeline from producer -> kafka topic -> kafka connect hdfs sink connector -> hdfs,
If even one of the kafka broker goes down, the connect framework stops responding. Stops consuming records and REST API also becomes not interactive.
Until the kafka connect framework is restarted, it would not pull the data from kafka and REST api remains inactive. Nothing is coming in the logs as well.
Checked the topics in kafka used by connect, everything has been reassigned to another broker and has the leader available.
Has anyone encountered this issue ? what would be the expected behavior ?
Thanks in advance
Kaushik