Reg : kafka state store HA

[Penumarthi Durga Prasad Chowdary <pr...@gmail.com>]

Hi ,
   Working on kafka-streams to process data & stored into
state-store(stored into local file system & changelog topic as well).Two
streams instances are started for HA & performance improvement ,then
streams are re-balanced the topic partition to process. Here we are facing
issue with state-store data , here two stream instances have their own
individual state-stores and those are not in sync(local file system), but
in sync with changelog topic . So *when querying on state-store data, it's
not giving results*.
I read this article
https://tech.transferwise.com/achieving-high-availability-with-kafka-streams/
,
added  num.standby.replicas  as 1 & 2 , but it's couldn't resolve the issue.
Is there a way to overcome this issue?

Thanks & Regards
Durga Prasad


-- 


Thank's&Regard's,
Prasad,
91-9030546248.

Re: Reg : kafka state store HA

["Matthias J. Sax" <ma...@confluent.io>]

Not exactly sure what your code does. But note, that state stores are
sharded and you need to query the right shard (ie, instance).

I would recommend to read:
https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/

There is also a blog post that explains it:
https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/

Last, there is a Kafka Summit talk:
https://www.confluent.io/kafka-summit-sf17/Interactive-Queries-in-Kafka-Streams-API


Hope this helps.


-Matthias



On 11/17/19 8:49 PM, Penumarthi Durga Prasad Chowdary wrote:
> Hi ,
>    Working on kafka-streams to process data & stored into
> state-store(stored into local file system & changelog topic as well).Two
> streams instances are started for HA & performance improvement ,then
> streams are re-balanced the topic partition to process. Here we are facing
> issue with state-store data , here two stream instances have their own
> individual state-stores and those are not in sync(local file system), but
> in sync with changelog topic . So *when querying on state-store data, it's
> not giving results*.
> I read this article
> https://tech.transferwise.com/achieving-high-availability-with-kafka-streams/
> ,
> added  num.standby.replicas  as 1 & 2 , but it's couldn't resolve the issue.
> Is there a way to overcome this issue?
> 
> Thanks & Regards
> Durga Prasad
> 
>