You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Woschitz, Janosch" <Ja...@thinkbiganalytics.com> on 2017/11/14 10:01:01 UTC

Current state of Kafka processors

Hi,

I saw the following notice on the current documentation for the PublishKafka and ConsumeKafka processors:

"Please note there are cases where the publisher can get into an indefinite stuck state. We are closely monitoring how this evolves in the Kafka community and will take advantage of those fixes as soon as we can. In the mean time it is possible to enter states where the only resolution will be to restart the JVM NiFi runs on.”

I was unable to find related and unresolved JIRA issues and was wondering if this is still an issue in NiFi 1.4.0?

Could somebody provide more insights into the current state?

Thanks,
Janosch

Re: Current state of Kafka processors

Posted by "Woschitz, Janosch" <Ja...@thinkbiganalytics.com>.
Hi Joe,

Thanks a lot for the explanation, this was also my assumption by looking at the code, though I assumed I missed something.

I filed https://issues.apache.org/jira/browse/NIFI-4623 and added a simple PR which just removes the warning in the doc comments for the newer Kafka processors.

Regards,
Janosch




On 21.11.17, 00:24, "Joe Witt" <jo...@gmail.com> wrote:

>Janosch
>
>Sorry for the lack of response.  It was there previously because it
>was observed that timeout conditions could cause threads to stay in a
>hung state and as the caller of the client we weren't in a position to
>resolve it.  However, newer client libraries are better and I no
>longer have observed these issues personally.  For the 0_10 or newer
>clients I think we can adjust the docs to indicate that these should
>work just fine.  Please feel free to file a JIRA if you dont mind.
>
>Thanks
>
>On Mon, Nov 20, 2017 at 6:22 PM, Woschitz, Janosch
><Ja...@thinkbiganalytics.com> wrote:
>> Hi everyone,
>>
>> Has anyone an idea what's the technical background regarding the warning in the Kafka processors documentation?
>>
>> It is not clear to me how the processor can end up in a "indefinite stuck state”.
>>
>> Thanks,
>> Janosch
>>
>>
>>
>> On 14.11.17, 11:01, "Woschitz, Janosch" <Ja...@thinkbiganalytics.com> wrote:
>>
>>>Hi,
>>>
>>>I saw the following notice on the current documentation for the PublishKafka and ConsumeKafka processors:
>>>
>>>"Please note there are cases where the publisher can get into an indefinite stuck state. We are closely monitoring how this evolves in the Kafka community and will take advantage of those fixes as soon as we can. In the mean time it is possible to enter states where the only resolution will be to restart the JVM NiFi runs on.”
>>>
>>>I was unable to find related and unresolved JIRA issues and was wondering if this is still an issue in NiFi 1.4.0?
>>>
>>>Could somebody provide more insights into the current state?
>>>
>>>Thanks,
>>>Janosch
>>>

Re: Current state of Kafka processors

Posted by Joe Witt <jo...@gmail.com>.
Janosch

Sorry for the lack of response.  It was there previously because it
was observed that timeout conditions could cause threads to stay in a
hung state and as the caller of the client we weren't in a position to
resolve it.  However, newer client libraries are better and I no
longer have observed these issues personally.  For the 0_10 or newer
clients I think we can adjust the docs to indicate that these should
work just fine.  Please feel free to file a JIRA if you dont mind.

Thanks

On Mon, Nov 20, 2017 at 6:22 PM, Woschitz, Janosch
<Ja...@thinkbiganalytics.com> wrote:
> Hi everyone,
>
> Has anyone an idea what's the technical background regarding the warning in the Kafka processors documentation?
>
> It is not clear to me how the processor can end up in a "indefinite stuck state”.
>
> Thanks,
> Janosch
>
>
>
> On 14.11.17, 11:01, "Woschitz, Janosch" <Ja...@thinkbiganalytics.com> wrote:
>
>>Hi,
>>
>>I saw the following notice on the current documentation for the PublishKafka and ConsumeKafka processors:
>>
>>"Please note there are cases where the publisher can get into an indefinite stuck state. We are closely monitoring how this evolves in the Kafka community and will take advantage of those fixes as soon as we can. In the mean time it is possible to enter states where the only resolution will be to restart the JVM NiFi runs on.”
>>
>>I was unable to find related and unresolved JIRA issues and was wondering if this is still an issue in NiFi 1.4.0?
>>
>>Could somebody provide more insights into the current state?
>>
>>Thanks,
>>Janosch
>>

Re: Current state of Kafka processors

Posted by "Woschitz, Janosch" <Ja...@thinkbiganalytics.com>.
Hi everyone,

Has anyone an idea what's the technical background regarding the warning in the Kafka processors documentation?

It is not clear to me how the processor can end up in a "indefinite stuck state”.

Thanks,
Janosch



On 14.11.17, 11:01, "Woschitz, Janosch" <Ja...@thinkbiganalytics.com> wrote:

>Hi,
>
>I saw the following notice on the current documentation for the PublishKafka and ConsumeKafka processors:
>
>"Please note there are cases where the publisher can get into an indefinite stuck state. We are closely monitoring how this evolves in the Kafka community and will take advantage of those fixes as soon as we can. In the mean time it is possible to enter states where the only resolution will be to restart the JVM NiFi runs on.”
>
>I was unable to find related and unresolved JIRA issues and was wondering if this is still an issue in NiFi 1.4.0?
>
>Could somebody provide more insights into the current state?
>
>Thanks,
>Janosch
>