You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Debraj Manna <su...@gmail.com> on 2023/10/02 19:11:44 UTC
Re: Kafka Streams reaching ERROR state during rolling upgrade / restart of brokers
Are you suggesting to check the Kafka broker logs? I do not see any other
errors logs on the client / application side.
On Fri, 29 Sep, 2023, 22:01 Matthias J. Sax, <mj...@apache.org> wrote:
> In general, Kafka Streams should keep running.
>
> Can you inspect the logs to figure out why it's going into ERROR state
> to begin with? Maybe you need to increase/change some timeouts/retries
> configs.
>
> The stack trace you shared, is a symptom, but not the root cause.
>
> -Matthias
>
> On 9/21/23 12:56 AM, Debraj Manna wrote:
> > I am using Kafka broker 2.8.1 (from AWS MSK) with Kafka clients and Kafka
> > stream 3.5.1.
> >
> > I am observing that whenever some rolling upgrade is done on AWS MSK our
> > stream application reaches an error state. I get the below exception on
> > trying to query the state store
> >
> > caused by: java.lang.IllegalStateException: KafkaStreams is not running.
> > State is ERROR.
> > at
> >
> org.apache.kafka.streams.KafkaStreams.validateIsRunningOrRebalancing(KafkaStreams.java:381)
> > at
> >
> org.apache.kafka.streams.KafkaStreams.queryMetadataForKey(KafkaStreams.java:1663)
> > at
> >
> org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$2(InteractiveQueryService.java:227)
> >
> > Can someone let me know what the recommended way we can keep the stream
> > application running whenever some rolling upgrade/restart of brokers is
> > done in the background?
> >
>
Re: Kafka Streams reaching ERROR state during rolling upgrade / restart of brokers
Posted by Debraj Manna <su...@gmail.com>.
Matthias
It happened again yesterday during another rolling update. The first error
log I can see on the client side is below. It was there in PENDING_ERROR
state for sometime and then went into ERROR state.
Caused by: java.lang.IllegalStateException: KafkaStreams is not running.
State is PENDING_ERROR. at
org.apache.kafka.streams.KafkaStreams.validateIsRunningOrRebalancing(KafkaStreams.java:381)
at
org.apache.kafka.streams.KafkaStreams.queryMetadataForKey(KafkaStreams.java:1663)
at
org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$2(InteractiveQueryService.java:227)
at
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.HashMap$KeySpliterator.tryAdvance(HashMap.java:1728)
at
java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
at
java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at
java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:647)
at
org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$3(InteractiveQueryService.ja
On Tue, Oct 3, 2023 at 8:50 AM Matthias J. Sax <mj...@apache.org> wrote:
> I did mean client side... If KS goes into ERROR state, it should log
> the reason.
>
> If the logs are indeed empty, try to register an
> uncaught-exception-handler via
>
> KafkaStreamssetUncaughtExceptionHandler(...)
>
>
> -Matthias
>
> On 10/2/23 12:11 PM, Debraj Manna wrote:
> > Are you suggesting to check the Kafka broker logs? I do not see any other
> > errors logs on the client / application side.
> >
> > On Fri, 29 Sep, 2023, 22:01 Matthias J. Sax, <mj...@apache.org> wrote:
> >
> >> In general, Kafka Streams should keep running.
> >>
> >> Can you inspect the logs to figure out why it's going into ERROR state
> >> to begin with? Maybe you need to increase/change some timeouts/retries
> >> configs.
> >>
> >> The stack trace you shared, is a symptom, but not the root cause.
> >>
> >> -Matthias
> >>
> >> On 9/21/23 12:56 AM, Debraj Manna wrote:
> >>> I am using Kafka broker 2.8.1 (from AWS MSK) with Kafka clients and
> Kafka
> >>> stream 3.5.1.
> >>>
> >>> I am observing that whenever some rolling upgrade is done on AWS MSK
> our
> >>> stream application reaches an error state. I get the below exception on
> >>> trying to query the state store
> >>>
> >>> caused by: java.lang.IllegalStateException: KafkaStreams is not
> running.
> >>> State is ERROR.
> >>> at
> >>>
> >>
> org.apache.kafka.streams.KafkaStreams.validateIsRunningOrRebalancing(KafkaStreams.java:381)
> >>> at
> >>>
> >>
> org.apache.kafka.streams.KafkaStreams.queryMetadataForKey(KafkaStreams.java:1663)
> >>> at
> >>>
> >>
> org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$2(InteractiveQueryService.java:227)
> >>>
> >>> Can someone let me know what the recommended way we can keep the stream
> >>> application running whenever some rolling upgrade/restart of brokers is
> >>> done in the background?
> >>>
> >>
> >
>
Re: Kafka Streams reaching ERROR state during rolling upgrade / restart of brokers
Posted by "Matthias J. Sax" <mj...@apache.org>.
I did mean client side... If KS goes into ERROR state, it should log
the reason.
If the logs are indeed empty, try to register an
uncaught-exception-handler via
KafkaStreamssetUncaughtExceptionHandler(...)
-Matthias
On 10/2/23 12:11 PM, Debraj Manna wrote:
> Are you suggesting to check the Kafka broker logs? I do not see any other
> errors logs on the client / application side.
>
> On Fri, 29 Sep, 2023, 22:01 Matthias J. Sax, <mj...@apache.org> wrote:
>
>> In general, Kafka Streams should keep running.
>>
>> Can you inspect the logs to figure out why it's going into ERROR state
>> to begin with? Maybe you need to increase/change some timeouts/retries
>> configs.
>>
>> The stack trace you shared, is a symptom, but not the root cause.
>>
>> -Matthias
>>
>> On 9/21/23 12:56 AM, Debraj Manna wrote:
>>> I am using Kafka broker 2.8.1 (from AWS MSK) with Kafka clients and Kafka
>>> stream 3.5.1.
>>>
>>> I am observing that whenever some rolling upgrade is done on AWS MSK our
>>> stream application reaches an error state. I get the below exception on
>>> trying to query the state store
>>>
>>> caused by: java.lang.IllegalStateException: KafkaStreams is not running.
>>> State is ERROR.
>>> at
>>>
>> org.apache.kafka.streams.KafkaStreams.validateIsRunningOrRebalancing(KafkaStreams.java:381)
>>> at
>>>
>> org.apache.kafka.streams.KafkaStreams.queryMetadataForKey(KafkaStreams.java:1663)
>>> at
>>>
>> org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$2(InteractiveQueryService.java:227)
>>>
>>> Can someone let me know what the recommended way we can keep the stream
>>> application running whenever some rolling upgrade/restart of brokers is
>>> done in the background?
>>>
>>
>