[jira] [Commented] (KAFKA-7657) Invalid reporting of stream state in Kafka streams application

["Guozhang Wang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-7657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16695186#comment-16695186 ] 

Guozhang Wang commented on KAFKA-7657:
--------------------------------------

Hi [~tscrowley] Thanks for reporting this issue. The described situation does sound like a bug to me. And since you are already running on a pretty new version (2.0.1) this bug may well be existing in trunk as well.

At the moment if the Streams state is not reliable reporting the status of your application, I'd suggest using the consumer lag as a side-indicator of your application's "healthness". And to investigate this issue, could you share with me steps to reproduce this issue, and also try to turn off EOS and see if the issue goes away? I doubt the transformer has anything to do with the bug, and hence the one that would be my suspicion is EOS config. And if turning off EOS can resolve the issue, it can help us narrowing down the investigation scope.

> Invalid reporting of stream state in Kafka streams application
> --------------------------------------------------------------
>
>                 Key: KAFKA-7657
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7657
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 2.0.1
>            Reporter: Thomas Crowley
>            Priority: Minor
>
> We have a streams application with 3 instances running, two of which are reporting the state of REBALANCING even after they have been running for days. Restarting the application has no effect on the stream state.
> This seems suspect because each instance appears to be processing messages, and the kafka-consumer-groups CLI tool reports hardly any offset lag in any of the partitions assigned to the REBALANCING consumers. Each partition seems to be processing an equal amount of records too.
> Inspecting the state.dir on disk, it looks like the RocksDB state has been built and hovers at the expected size on disk.
> This problem has persisted for us after we rebuilt our Kafka cluster and reset topics + consumer groups in our dev environment.
> There is nothing in the logs (with level set to DEBUG) in both the broker or the application that suggests something exceptional has happened causing the application to be stuck REBALANCING.
> We are also running multiple streaming applications where this problem does not exist.
> Two differences between this application and our other streaming applications are:
>  * We have processing.guarantee set to exactly_once
>  * We are using a ValueTransformer which fetches from and puts data on a windowed state store
> The REBALANCING state is returned from both polling the state method of our KafkaStreams instance, and our custom metric which is derived from some logic in a KafkaStreams.StateListener class attached via the setStateListener method.
>  
> While I have provided a bit of context, before I reply with some reproducible code - is there a simple way in which I can determine that my streams application is in a RUNNING state without relying on the same mechanisms as used above?
> Further, given that it seems like my application is actually running - could this perhaps be a bug to do with how the stream state is being reported (in the context of a transactional stream using the processor API)?
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)