You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Dimitry Lvovsky <dl...@gmail.com> on 2019/09/03 11:37:45 UTC
Failed to rebalance -- Unexpected error in InitProducerIdResponse
Hi all,
We have a streaming app running for over a year now without much incident
until recently. Suddenly the process keeps failing after a minute or so
with a failed to rebalance exception caused by Unexpected error in
InitProducerIdResponse; The server experienced an unexpected error when
processing the request. Please see stack trace at end of the email.
I've hunted on Google to some insights into the problem but to no avail.
We're running Kafka 2.0.0. The stream app/client is running on java 1.8.,
spring-kafka 2.1.10.Release and related kafa libs at 2.0.0 .
Your help is much appreciated. Many thanks in advance.
Stack trace below:
13:13:30.448 [ERROR]
[filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31]
[StreamThread$RebalanceListener:onPartitionsAssigned:273] - stream-thread
[filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31] Error
caught during partition assignment, will abort the current process and
re-throw at the end of rebalance: {}
org.apache.kafka.common.KafkaException: Unexpected error in
InitProducerIdResponse; The server experienced an unexpected error when
processing the request
at
org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:982)
at
org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:907)
at
org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
at
org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:532)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:524)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
at java.lang.Thread.run(Thread.java:748)
Exception in thread
"filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31"
org.apache.kafka.streams.errors.StreamsException: stream-thread
[filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31]
Failed to rebalance.
at
org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:870)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:810)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:767)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:736)
Caused by: org.apache.kafka.common.KafkaException: Unexpected error in
InitProducerIdResponse; The server experienced an unexpected error when
processing the request
at
org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:982)
at
org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:907)
at
org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
at
org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:532)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:524)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
at java.lang.Thread.run(Thread.java:748)
Re: Failed to rebalance -- Unexpected error in InitProducerIdResponse
Posted by Dimitry Lvovsky <dl...@gmail.com>.
It’s only limited to this one app. I’ll keep digging and let you know.
More suggestions always welcome.
Thanks.
On Tue, Sep 3, 2019 at 13:53 Jonathan Santilli <jo...@gmail.com>
wrote:
> Np,
>
> are other Apps working correctly? (if that the case)
> Seems like the hints are around "org.apache.kafka.common.KafkaException:
> Unexpected error in InitProducerIdResponse; The server experienced an
> unexpected error when processing the request"
> I would start checking the Broker's health and if this particular issue is
> just affecting the filter-dps-app App.
> Maybe other apps are experiencing diff issues around the same time.
> Without more info, if difficult to triage.
>
> Hope that helps,
> --
> Jonathan
>
>
>
>
> On Tue, Sep 3, 2019 at 12:46 PM Dimitry Lvovsky <dl...@gmail.com>
> wrote:
>
> > Thanks, Jonathan for the quick response. I have, but I didn't see any
> > errors.
> >
> >
> >
> > On Tue, Sep 3, 2019 at 1:43 PM Jonathan Santilli <
> > jonathansantilli@gmail.com>
> > wrote:
> >
> > > Hello, have you check the Brokers Logs? maybe there is something there
> at
> > > that particular time when the error occurred in the App side.
> > >
> > > Cheers!
> > >
> > >
> > >
> > > On Tue, Sep 3, 2019 at 12:38 PM Dimitry Lvovsky <dl...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > > We have a streaming app running for over a year now without much
> > incident
> > > > until recently. Suddenly the process keeps failing after a minute
> or
> > so
> > > > with a failed to rebalance exception caused by Unexpected error in
> > > > InitProducerIdResponse; The server experienced an unexpected error
> when
> > > > processing the request. Please see stack trace at end of the email.
> > > > I've hunted on Google to some insights into the problem but to no
> > avail.
> > > >
> > > > We're running Kafka 2.0.0. The stream app/client is running on java
> > > 1.8.,
> > > > spring-kafka 2.1.10.Release and related kafa libs at 2.0.0 .
> > > >
> > > > Your help is much appreciated. Many thanks in advance.
> > > >
> > > > Stack trace below:
> > > >
> > > >
> > > > 13:13:30.448 [ERROR]
> > > > [filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31]
> > > > [StreamThread$RebalanceListener:onPartitionsAssigned:273] -
> > stream-thread
> > > > [filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31]
> > > Error
> > > > caught during partition assignment, will abort the current process
> and
> > > > re-throw at the end of rebalance: {}
> > > > org.apache.kafka.common.KafkaException: Unexpected error in
> > > > InitProducerIdResponse; The server experienced an unexpected error
> when
> > > > processing the request
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:982)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:907)
> > > > at
> > > >
> > >
> >
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:532)
> > > > at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:524)
> > > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> > > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > Exception in thread
> > > > "filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31"
> > > > org.apache.kafka.streams.errors.StreamsException: stream-thread
> > > > [filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31]
> > > > Failed to rebalance.
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:870)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:810)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:767)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:736)
> > > > Caused by: org.apache.kafka.common.KafkaException: Unexpected error
> in
> > > > InitProducerIdResponse; The server experienced an unexpected error
> when
> > > > processing the request
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:982)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:907)
> > > > at
> > > >
> > >
> >
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:532)
> > > > at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:524)
> > > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> > > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
> > > > at java.lang.Thread.run(Thread.java:748)
> > > >
> > >
> > >
> > > --
> > > Santilli Jonathan
> > >
> >
>
>
> --
> Santilli Jonathan
>
Re: Failed to rebalance -- Unexpected error in InitProducerIdResponse
Posted by Jonathan Santilli <jo...@gmail.com>.
Np,
are other Apps working correctly? (if that the case)
Seems like the hints are around "org.apache.kafka.common.KafkaException:
Unexpected error in InitProducerIdResponse; The server experienced an
unexpected error when processing the request"
I would start checking the Broker's health and if this particular issue is
just affecting the filter-dps-app App.
Maybe other apps are experiencing diff issues around the same time.
Without more info, if difficult to triage.
Hope that helps,
--
Jonathan
On Tue, Sep 3, 2019 at 12:46 PM Dimitry Lvovsky <dl...@gmail.com> wrote:
> Thanks, Jonathan for the quick response. I have, but I didn't see any
> errors.
>
>
>
> On Tue, Sep 3, 2019 at 1:43 PM Jonathan Santilli <
> jonathansantilli@gmail.com>
> wrote:
>
> > Hello, have you check the Brokers Logs? maybe there is something there at
> > that particular time when the error occurred in the App side.
> >
> > Cheers!
> >
> >
> >
> > On Tue, Sep 3, 2019 at 12:38 PM Dimitry Lvovsky <dl...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > > We have a streaming app running for over a year now without much
> incident
> > > until recently. Suddenly the process keeps failing after a minute or
> so
> > > with a failed to rebalance exception caused by Unexpected error in
> > > InitProducerIdResponse; The server experienced an unexpected error when
> > > processing the request. Please see stack trace at end of the email.
> > > I've hunted on Google to some insights into the problem but to no
> avail.
> > >
> > > We're running Kafka 2.0.0. The stream app/client is running on java
> > 1.8.,
> > > spring-kafka 2.1.10.Release and related kafa libs at 2.0.0 .
> > >
> > > Your help is much appreciated. Many thanks in advance.
> > >
> > > Stack trace below:
> > >
> > >
> > > 13:13:30.448 [ERROR]
> > > [filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31]
> > > [StreamThread$RebalanceListener:onPartitionsAssigned:273] -
> stream-thread
> > > [filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31]
> > Error
> > > caught during partition assignment, will abort the current process and
> > > re-throw at the end of rebalance: {}
> > > org.apache.kafka.common.KafkaException: Unexpected error in
> > > InitProducerIdResponse; The server experienced an unexpected error when
> > > processing the request
> > > at
> > >
> > >
> >
> org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:982)
> > > at
> > >
> > >
> >
> org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:907)
> > > at
> > >
> >
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
> > > at
> > >
> > >
> >
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:532)
> > > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:524)
> > > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> > > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
> > > at java.lang.Thread.run(Thread.java:748)
> > > Exception in thread
> > > "filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31"
> > > org.apache.kafka.streams.errors.StreamsException: stream-thread
> > > [filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31]
> > > Failed to rebalance.
> > > at
> > >
> > >
> >
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:870)
> > > at
> > >
> > >
> >
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:810)
> > > at
> > >
> > >
> >
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:767)
> > > at
> > >
> > >
> >
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:736)
> > > Caused by: org.apache.kafka.common.KafkaException: Unexpected error in
> > > InitProducerIdResponse; The server experienced an unexpected error when
> > > processing the request
> > > at
> > >
> > >
> >
> org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:982)
> > > at
> > >
> > >
> >
> org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:907)
> > > at
> > >
> >
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
> > > at
> > >
> > >
> >
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:532)
> > > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:524)
> > > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> > > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
> > > at java.lang.Thread.run(Thread.java:748)
> > >
> >
> >
> > --
> > Santilli Jonathan
> >
>
--
Santilli Jonathan
Re: Failed to rebalance -- Unexpected error in InitProducerIdResponse
Posted by Dimitry Lvovsky <dl...@gmail.com>.
Thanks, Jonathan for the quick response. I have, but I didn't see any
errors.
On Tue, Sep 3, 2019 at 1:43 PM Jonathan Santilli <jo...@gmail.com>
wrote:
> Hello, have you check the Brokers Logs? maybe there is something there at
> that particular time when the error occurred in the App side.
>
> Cheers!
>
>
>
> On Tue, Sep 3, 2019 at 12:38 PM Dimitry Lvovsky <dl...@gmail.com>
> wrote:
>
> > Hi all,
> > We have a streaming app running for over a year now without much incident
> > until recently. Suddenly the process keeps failing after a minute or so
> > with a failed to rebalance exception caused by Unexpected error in
> > InitProducerIdResponse; The server experienced an unexpected error when
> > processing the request. Please see stack trace at end of the email.
> > I've hunted on Google to some insights into the problem but to no avail.
> >
> > We're running Kafka 2.0.0. The stream app/client is running on java
> 1.8.,
> > spring-kafka 2.1.10.Release and related kafa libs at 2.0.0 .
> >
> > Your help is much appreciated. Many thanks in advance.
> >
> > Stack trace below:
> >
> >
> > 13:13:30.448 [ERROR]
> > [filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31]
> > [StreamThread$RebalanceListener:onPartitionsAssigned:273] - stream-thread
> > [filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31]
> Error
> > caught during partition assignment, will abort the current process and
> > re-throw at the end of rebalance: {}
> > org.apache.kafka.common.KafkaException: Unexpected error in
> > InitProducerIdResponse; The server experienced an unexpected error when
> > processing the request
> > at
> >
> >
> org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:982)
> > at
> >
> >
> org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:907)
> > at
> >
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
> > at
> >
> >
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:532)
> > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:524)
> > at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> > at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
> > at java.lang.Thread.run(Thread.java:748)
> > Exception in thread
> > "filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31"
> > org.apache.kafka.streams.errors.StreamsException: stream-thread
> > [filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31]
> > Failed to rebalance.
> > at
> >
> >
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:870)
> > at
> >
> >
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:810)
> > at
> >
> >
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:767)
> > at
> >
> >
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:736)
> > Caused by: org.apache.kafka.common.KafkaException: Unexpected error in
> > InitProducerIdResponse; The server experienced an unexpected error when
> > processing the request
> > at
> >
> >
> org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:982)
> > at
> >
> >
> org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:907)
> > at
> >
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
> > at
> >
> >
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:532)
> > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:524)
> > at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> > at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
> > at java.lang.Thread.run(Thread.java:748)
> >
>
>
> --
> Santilli Jonathan
>
Re: Failed to rebalance -- Unexpected error in InitProducerIdResponse
Posted by Jonathan Santilli <jo...@gmail.com>.
Hello, have you check the Brokers Logs? maybe there is something there at
that particular time when the error occurred in the App side.
Cheers!
On Tue, Sep 3, 2019 at 12:38 PM Dimitry Lvovsky <dl...@gmail.com> wrote:
> Hi all,
> We have a streaming app running for over a year now without much incident
> until recently. Suddenly the process keeps failing after a minute or so
> with a failed to rebalance exception caused by Unexpected error in
> InitProducerIdResponse; The server experienced an unexpected error when
> processing the request. Please see stack trace at end of the email.
> I've hunted on Google to some insights into the problem but to no avail.
>
> We're running Kafka 2.0.0. The stream app/client is running on java 1.8.,
> spring-kafka 2.1.10.Release and related kafa libs at 2.0.0 .
>
> Your help is much appreciated. Many thanks in advance.
>
> Stack trace below:
>
>
> 13:13:30.448 [ERROR]
> [filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31]
> [StreamThread$RebalanceListener:onPartitionsAssigned:273] - stream-thread
> [filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31] Error
> caught during partition assignment, will abort the current process and
> re-throw at the end of rebalance: {}
> org.apache.kafka.common.KafkaException: Unexpected error in
> InitProducerIdResponse; The server experienced an unexpected error when
> processing the request
> at
>
> org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:982)
> at
>
> org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:907)
> at
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
> at
>
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:532)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:524)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
> at java.lang.Thread.run(Thread.java:748)
> Exception in thread
> "filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31"
> org.apache.kafka.streams.errors.StreamsException: stream-thread
> [filter-dps-app-b86df7a2-da04-4093-9b09-45d1ea5dc7fb-StreamThread-31]
> Failed to rebalance.
> at
>
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:870)
> at
>
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:810)
> at
>
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:767)
> at
>
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:736)
> Caused by: org.apache.kafka.common.KafkaException: Unexpected error in
> InitProducerIdResponse; The server experienced an unexpected error when
> processing the request
> at
>
> org.apache.kafka.clients.producer.internals.TransactionManager$InitProducerIdHandler.handleResponse(TransactionManager.java:982)
> at
>
> org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:907)
> at
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
> at
>
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:532)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:524)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
> at java.lang.Thread.run(Thread.java:748)
>
--
Santilli Jonathan