You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yoshimo <yo...@protonmail.com.INVALID> on 2018/11/30 08:57:15 UTC
Partial updates in Kafka Streams stores
Hello Kafka users,
I am currently building a Kafka Streams application and I am using a transform step with two KeyValue stores, one containing the transformed message and the second one some metadata about the message.
In the transform step I first update the metadata store and then store the message in the second store. My question is, is it possible to see an update in the metada store without seeing the according transformation in the second store when reading from the above stores during failover scenarios? If yes, is there a way to update the two stores in a transactional way?
Thank you,
Yianns
Sent with [ProtonMail](https://protonmail.com) Secure Email.
Re: Partial updates in Kafka Streams stores
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Yes, that could happen. If you write into a store, the write _might_ go
directly into the changelog topic, too, and for this case, the instance
that takes over would see this write and add it to it's store during
recovery.
Idempotent updates would solve this issue.
-Matthias
On 11/30/18 12:17 PM, Yoshimo wrote:
> Hello Matthias,
>
> thank you for the answer. What if the operations on the stores are idempotent? I am only worried for the scenario where after doing a put in the first store and before updating the second one, the application crashes or there is a network outage. Will the node that picks up the partition see the first update?
>
> Thank you
>
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Friday, 30 November 2018 21:23, Matthias J. Sax <ma...@confluent.io> wrote:
>
>> You can enable exactly-once processing guarantees to guard against
>> inconsistent stores in fail over scenario.
>>
>> -Matthias
>>
>> On 11/30/18 12:57 AM, Yoshimo wrote:
>>
>>> Hello Kafka users,
>>> I am currently building a Kafka Streams application and I am using a transform step with two KeyValue stores, one containing the transformed message and the second one some metadata about the message.
>>> In the transform step I first update the metadata store and then store the message in the second store. My question is, is it possible to see an update in the metada store without seeing the according transformation in the second store when reading from the above stores during failover scenarios? If yes, is there a way to update the two stores in a transactional way?
>>> Thank you,
>>> Yianns
>>> Sent with ProtonMail Secure Email.
>
>
Re: Partial updates in Kafka Streams stores
Posted by Yoshimo <yo...@protonmail.com.INVALID>.
Hello Matthias,
thank you for the answer. What if the operations on the stores are idempotent? I am only worried for the scenario where after doing a put in the first store and before updating the second one, the application crashes or there is a network outage. Will the node that picks up the partition see the first update?
Thank you
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, 30 November 2018 21:23, Matthias J. Sax <ma...@confluent.io> wrote:
> You can enable exactly-once processing guarantees to guard against
> inconsistent stores in fail over scenario.
>
> -Matthias
>
> On 11/30/18 12:57 AM, Yoshimo wrote:
>
> > Hello Kafka users,
> > I am currently building a Kafka Streams application and I am using a transform step with two KeyValue stores, one containing the transformed message and the second one some metadata about the message.
> > In the transform step I first update the metadata store and then store the message in the second store. My question is, is it possible to see an update in the metada store without seeing the according transformation in the second store when reading from the above stores during failover scenarios? If yes, is there a way to update the two stores in a transactional way?
> > Thank you,
> > Yianns
> > Sent with ProtonMail Secure Email.
Re: Partial updates in Kafka Streams stores
Posted by "Matthias J. Sax" <ma...@confluent.io>.
You can enable exactly-once processing guarantees to guard against
inconsistent stores in fail over scenario.
-Matthias
On 11/30/18 12:57 AM, Yoshimo wrote:
> Hello Kafka users,
>
> I am currently building a Kafka Streams application and I am using a transform step with two KeyValue stores, one containing the transformed message and the second one some metadata about the message.
>
> In the transform step I first update the metadata store and then store the message in the second store. My question is, is it possible to see an update in the metada store without seeing the according transformation in the second store when reading from the above stores during failover scenarios? If yes, is there a way to update the two stores in a transactional way?
>
> Thank you,
> Yianns
>
> Sent with [ProtonMail](https://protonmail.com) Secure Email.
>