​Inconsistent state in Account Debit - Credit Application Using Kafka Transaction

[Ankur Saxena <an...@timesinternet.in.INVALID>]

Hi ,
 Please address my issue

Account A sends 500 bucks to Account B , We are using a topic Account
having multiple partitions to record these events

Producer ->
Transaction starts
Account A  BalanceA - 500 to topic Account , partition p0
Account B  BalanceB + 500 to topic Account , partition p1
Transaction Ends

on Consumer Side we have single stream thread that handle these partition ,
and update its global state store(Global K Table) accordingly , consumers
consuming these partitions consume these message at different poll  and
hanse inconsistent state produces

Subtract 500 from Account A in global state store  on some poll
consuming some non transactional data from other partitions
Add 500 to Account B in global state store   -- on another poll

During  we have inconsistent state in which Account A debited but Account B
is not credited

How can We consume transactional data atomically o in application using Low
level Kafka Stream Api to update its global State Store(Global K Table) in
order to avoid inconsistent state at any point of time


-- 
*   Thanks*
*  Ankur Saxena*
*+91 9717002303*

-- 


**Did You Know you get rewarded everytime you read news on ToI.in? ******



Each moment you spend reading news & watching videos on TOI, NewsPoint, 
NBT you earn TIMESPOINTS. These TimesPoints can be redeemed for exclusive 
offers from your favorite brands including Amazon, BigBasket, BigBazaar, 
Uber, BookMyShow, Paytm and many others. Login to TimesPoints.com  
<https://www.timespoints.com/products?utm_source=TimesPoints&utm_medium=EmailFooter>and 
start redeeming NOW!

Re: ​Inconsistent state in Account Debit - Credit Application Using Kafka Transaction

["Matthias J. Sax" <ma...@confluent.io>]

That is not possible.

On read, it's not even known which records belong to the same "write
transaction". Note, that this is how transactions work in general:
assume you have a database and a transactions updates two rows. After
the transaction commits, you can read those updates, but you don't know
which rows were updated in a single transaction.


-Matthias

On 3/13/19 2:32 AM, Ankur Saxena wrote:
> Hi ,
>  Please address my issue
> 
> Account A sends 500 bucks to Account B , We are using a topic Account
> having multiple partitions to record these events
> 
> Producer ->
> Transaction starts
> Account A  BalanceA - 500 to topic Account , partition p0
> Account B  BalanceB + 500 to topic Account , partition p1
> Transaction Ends
> 
> on Consumer Side we have single stream thread that handle these partition ,
> and update its global state store(Global K Table) accordingly , consumers
> consuming these partitions consume these message at different poll  and
> hanse inconsistent state produces
> 
> Subtract 500 from Account A in global state store  on some poll
> consuming some non transactional data from other partitions
> Add 500 to Account B in global state store   -- on another poll
> 
> During  we have inconsistent state in which Account A debited but Account B
> is not credited
> 
> How can We consume transactional data atomically o in application using Low
> level Kafka Stream Api to update its global State Store(Global K Table) in
> order to avoid inconsistent state at any point of time
> 
>