You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Druhin Sagar Goel <dr...@arrcus.com> on 2018/07/18 06:12:16 UTC
Kafka Streams: Share state store across processors
Hi,
I am new to the Kafka Streams framework. I have the following streams use case:
State store A
State store B
Processor A
Processor B
State store A is only written to by Processor A but also needs to be read by Processor B. State store B needs to be written to by both Processor A and Processor B. Processor A and Processor B run concurrently.
What is the best way for me to be able to share state stores across processors such that they can be written to by multiple processors?
Thanks,
Druhin
Re: Kafka Streams: Share state store across processors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
If you connect both stores to both processor, there will be only one
thread for both processors. Thus, a concurrent access can never happen.
-Matthias
On 7/18/18 10:51 AM, Druhin Sagar Goel wrote:
> Hi Matthias,
>
>
> I was under the impression that state stores are not thread safe and so two processors writing to the same store at the same time would not work. I understood this from your reply in this post: https://groups.google.com/forum/#!topic/confluent-platform/JTKyDE231y8. Is that not the case?
>
>
> Thanks,
>
> Druhin
>
> [http://www.google.com/images/icons/product/groups-128.png]<https://groups.google.com/forum/#!topic/confluent-platform/JTKyDE231y8>
>
> Kafka streams with global state store for NUM_STREAM_THREADS_CONFIG=3<https://groups.google.com/forum/#!topic/confluent-platform/JTKyDE231y8>
> groups.google.com
> Posted 11/13/17 2:50 AM, 8 messages
>
>
> ________________________________
> From: Matthias J. Sax <ma...@confluent.io>
> Sent: Wednesday, July 18, 2018 10:20:02 AM
> To: users@kafka.apache.org
> Subject: Re: Kafka Streams: Share state store across processors
>
> You can connect both stores to both processor for this.
>
> -Matthias
>
> On 7/17/18 11:12 PM, Druhin Sagar Goel wrote:
>> Hi,
>>
>> I am new to the Kafka Streams framework. I have the following streams use case:
>>
>> State store A
>> State store B
>>
>> Processor A
>> Processor B
>>
>> State store A is only written to by Processor A but also needs to be read by Processor B. State store B needs to be written to by both Processor A and Processor B. Processor A and Processor B run concurrently.
>>
>> What is the best way for me to be able to share state stores across processors such that they can be written to by multiple processors?
>>
>>
>> Thanks,
>> Druhin
>>
>
>
Re: Kafka Streams: Share state store across processors
Posted by Druhin Sagar Goel <dr...@arrcus.com>.
Hi Matthias,
I was under the impression that state stores are not thread safe and so two processors writing to the same store at the same time would not work. I understood this from your reply in this post: https://groups.google.com/forum/#!topic/confluent-platform/JTKyDE231y8. Is that not the case?
Thanks,
Druhin
[http://www.google.com/images/icons/product/groups-128.png]<https://groups.google.com/forum/#!topic/confluent-platform/JTKyDE231y8>
Kafka streams with global state store for NUM_STREAM_THREADS_CONFIG=3<https://groups.google.com/forum/#!topic/confluent-platform/JTKyDE231y8>
groups.google.com
Posted 11/13/17 2:50 AM, 8 messages
________________________________
From: Matthias J. Sax <ma...@confluent.io>
Sent: Wednesday, July 18, 2018 10:20:02 AM
To: users@kafka.apache.org
Subject: Re: Kafka Streams: Share state store across processors
You can connect both stores to both processor for this.
-Matthias
On 7/17/18 11:12 PM, Druhin Sagar Goel wrote:
> Hi,
>
> I am new to the Kafka Streams framework. I have the following streams use case:
>
> State store A
> State store B
>
> Processor A
> Processor B
>
> State store A is only written to by Processor A but also needs to be read by Processor B. State store B needs to be written to by both Processor A and Processor B. Processor A and Processor B run concurrently.
>
> What is the best way for me to be able to share state stores across processors such that they can be written to by multiple processors?
>
>
> Thanks,
> Druhin
>
Re: Kafka Streams: Share state store across processors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
You can connect both stores to both processor for this.
-Matthias
On 7/17/18 11:12 PM, Druhin Sagar Goel wrote:
> Hi,
>
> I am new to the Kafka Streams framework. I have the following streams use case:
>
> State store A
> State store B
>
> Processor A
> Processor B
>
> State store A is only written to by Processor A but also needs to be read by Processor B. State store B needs to be written to by both Processor A and Processor B. Processor A and Processor B run concurrently.
>
> What is the best way for me to be able to share state stores across processors such that they can be written to by multiple processors?
>
>
> Thanks,
> Druhin
>