You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by detharon <de...@gmail.com> on 2018/03/07 14:07:42 UTC
Kafka Streams - "state store may have migrated to another instance"
Hello, I'm experiencing issues accessing the state stores outside the Kafka
stream. My application queries the state store every n seconds using the
.all() method to retrieve all key value pairs. I know that the state store
might not be available, so I guard against the InvalidStateStoreException
and in case it's caught, I simply invoke the .store method on my stream, in
order to get a new store. The problem is, that for some reason the store
never becomes available.
Some facts:
- During that time stream processing works correctly, and it successfully
puts and gets data to and from the store.
- Stream is in "running" state. I've started logging that because under
normal circumstances this exception is being thrown when the stream is in
"rebalancing" phase, but after a while it's gone, as expected, so I can
distinguish between these two cases.
- It only happens when I run my app in multiple instances.
I've set the log level to debug, but I can't see anything suspicious in the
logs, but maybe there's something particular I should pay attention to?
I access the store from inside an Akka actor, to which I pass the reference
to KafkaStreams object, if that matters.
I ran out of ideas what might have caused that behavior, so any help will
be greatly appreciated.
Re: Kafka Streams - "state store may have migrated to another instance"
Posted by Damian Guy <da...@gmail.com>.
If you run multiple instances of your app you may not be able to access the
state store you are trying to access from the instance you are trying from,
i.e., it may be on another instance. If streams is in the RUNNING state,
this would seem to be the issue.
On Wed, 7 Mar 2018 at 15:56 detharon <de...@gmail.com> wrote:
> I'm afraid that's not what I'm looking for, as I'm just trying to retrieve
> the local data, from inside my application (but from outside the stream
> topology), and in some cases it becomes impossible. That is, the stream
> changes its state from "rebalancing" to "running", but the store is remains
> inaccessible.
>
> I don't want to have access to state stores located on other instances.
>
> In pseudocode:
> streams.start(); <- here's where the stream starts
> loopOverLocalData(streams) <- periodical, asynchronous calls to
> streams.store, which, if the problem occurs, always result in exceptions
> being thrown. I expected to retrieve the local data this way.
>
> On 7 March 2018 at 16:20, Damian Guy <da...@gmail.com> wrote:
>
> > If you have multiple streams instances then the store might only be
> > available on one of the instances. Using `KafkaStreams.store(..)` will
> only
> > locate stores that are currently accessible by that instance. If you need
> > to be able to locate stores on other instances, then you should probably
> > have a read of:
> > https://kafka.apache.org/10/documentation/streams/
> > developer-guide/interactive-queries.html#querying-remote-
> > state-stores-for-the-entire-app
> >
> > There is also a convenient 3rd party lib that can help you with this:
> > https://github.com/lightbend/kafka-streams-query
> >
> > On Wed, 7 Mar 2018 at 14:07 detharon <de...@gmail.com> wrote:
> >
> > > Hello, I'm experiencing issues accessing the state stores outside the
> > Kafka
> > > stream. My application queries the state store every n seconds using
> the
> > > .all() method to retrieve all key value pairs. I know that the state
> > store
> > > might not be available, so I guard against the
> InvalidStateStoreException
> > > and in case it's caught, I simply invoke the .store method on my
> stream,
> > in
> > > order to get a new store. The problem is, that for some reason the
> store
> > > never becomes available.
> > >
> > > Some facts:
> > > - During that time stream processing works correctly, and it
> successfully
> > > puts and gets data to and from the store.
> > > - Stream is in "running" state. I've started logging that because under
> > > normal circumstances this exception is being thrown when the stream is
> in
> > > "rebalancing" phase, but after a while it's gone, as expected, so I can
> > > distinguish between these two cases.
> > > - It only happens when I run my app in multiple instances.
> > >
> > > I've set the log level to debug, but I can't see anything suspicious in
> > the
> > > logs, but maybe there's something particular I should pay attention to?
> > > I access the store from inside an Akka actor, to which I pass the
> > reference
> > > to KafkaStreams object, if that matters.
> > >
> > > I ran out of ideas what might have caused that behavior, so any help
> will
> > > be greatly appreciated.
> > >
> >
>
Re: Kafka Streams - "state store may have migrated to another instance"
Posted by detharon <de...@gmail.com>.
I'm afraid that's not what I'm looking for, as I'm just trying to retrieve
the local data, from inside my application (but from outside the stream
topology), and in some cases it becomes impossible. That is, the stream
changes its state from "rebalancing" to "running", but the store is remains
inaccessible.
I don't want to have access to state stores located on other instances.
In pseudocode:
streams.start(); <- here's where the stream starts
loopOverLocalData(streams) <- periodical, asynchronous calls to
streams.store, which, if the problem occurs, always result in exceptions
being thrown. I expected to retrieve the local data this way.
On 7 March 2018 at 16:20, Damian Guy <da...@gmail.com> wrote:
> If you have multiple streams instances then the store might only be
> available on one of the instances. Using `KafkaStreams.store(..)` will only
> locate stores that are currently accessible by that instance. If you need
> to be able to locate stores on other instances, then you should probably
> have a read of:
> https://kafka.apache.org/10/documentation/streams/
> developer-guide/interactive-queries.html#querying-remote-
> state-stores-for-the-entire-app
>
> There is also a convenient 3rd party lib that can help you with this:
> https://github.com/lightbend/kafka-streams-query
>
> On Wed, 7 Mar 2018 at 14:07 detharon <de...@gmail.com> wrote:
>
> > Hello, I'm experiencing issues accessing the state stores outside the
> Kafka
> > stream. My application queries the state store every n seconds using the
> > .all() method to retrieve all key value pairs. I know that the state
> store
> > might not be available, so I guard against the InvalidStateStoreException
> > and in case it's caught, I simply invoke the .store method on my stream,
> in
> > order to get a new store. The problem is, that for some reason the store
> > never becomes available.
> >
> > Some facts:
> > - During that time stream processing works correctly, and it successfully
> > puts and gets data to and from the store.
> > - Stream is in "running" state. I've started logging that because under
> > normal circumstances this exception is being thrown when the stream is in
> > "rebalancing" phase, but after a while it's gone, as expected, so I can
> > distinguish between these two cases.
> > - It only happens when I run my app in multiple instances.
> >
> > I've set the log level to debug, but I can't see anything suspicious in
> the
> > logs, but maybe there's something particular I should pay attention to?
> > I access the store from inside an Akka actor, to which I pass the
> reference
> > to KafkaStreams object, if that matters.
> >
> > I ran out of ideas what might have caused that behavior, so any help will
> > be greatly appreciated.
> >
>
Re: Kafka Streams - "state store may have migrated to another instance"
Posted by Damian Guy <da...@gmail.com>.
If you have multiple streams instances then the store might only be
available on one of the instances. Using `KafkaStreams.store(..)` will only
locate stores that are currently accessible by that instance. If you need
to be able to locate stores on other instances, then you should probably
have a read of:
https://kafka.apache.org/10/documentation/streams/developer-guide/interactive-queries.html#querying-remote-state-stores-for-the-entire-app
There is also a convenient 3rd party lib that can help you with this:
https://github.com/lightbend/kafka-streams-query
On Wed, 7 Mar 2018 at 14:07 detharon <de...@gmail.com> wrote:
> Hello, I'm experiencing issues accessing the state stores outside the Kafka
> stream. My application queries the state store every n seconds using the
> .all() method to retrieve all key value pairs. I know that the state store
> might not be available, so I guard against the InvalidStateStoreException
> and in case it's caught, I simply invoke the .store method on my stream, in
> order to get a new store. The problem is, that for some reason the store
> never becomes available.
>
> Some facts:
> - During that time stream processing works correctly, and it successfully
> puts and gets data to and from the store.
> - Stream is in "running" state. I've started logging that because under
> normal circumstances this exception is being thrown when the stream is in
> "rebalancing" phase, but after a while it's gone, as expected, so I can
> distinguish between these two cases.
> - It only happens when I run my app in multiple instances.
>
> I've set the log level to debug, but I can't see anything suspicious in the
> logs, but maybe there's something particular I should pay attention to?
> I access the store from inside an Akka actor, to which I pass the reference
> to KafkaStreams object, if that matters.
>
> I ran out of ideas what might have caused that behavior, so any help will
> be greatly appreciated.
>