Transactional Consistency beyond KIP-98 (transactional consumer)

["Daehn, Werner" <we...@sap.com>]

While the new transactional consistency feature is definitely a move into the right direction, I find the current design limitations quite severe. Understandable but severe.

First is the offset being the add-time, not the commit time. The consequence of that is, if one transactional producer adds rows every once a while and does commit minutes or hours later, all consumers with read=committed will not see any other producer's data until this one did the commit. Will decrease the stability of the overall solution, won't it?
Would have been better to use the commit time as the offset value. Harder to implement but more stable.

The other issue is at the consumer side. Current implementation does not provide transaction guarantees on the consumers. Again, for logical reasons and to simplify the implementation, but as a user you want to have both - when data is committed together you want to read the entire transaction as one unit and complete.

Are there any plans to go beyond the current implementation?

Thanks in advance

Re: Transactional Consistency beyond KIP-98 (transactional consumer)

[Guozhang Wang <wa...@gmail.com>]

Hello Werner,

1) Regarding the long transaction, we did discuss about this in the design
doc (
https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#bookmark=id.z9b0a6k2a3zm),
and as stated the main reason we chose to go with the current design is to
preserve offset ordering; and if it is becoming a critical feature request,
we may consider allow users to read either in offset ordering or
transaction ordering, in latter case a single long transaction will not
stall others.

2) Regarding the transactional consumption behavior, my understanding is
you want to have `consumer.poll()` return all records committed in a single
transaction right? My take is that this is again depending on the
transaction ordering mentioned above, since a single topic partition may
have multiple producers sending to it, resulting in multiple transactions
interleaving with each other on that log. In order to return a single
completed transaction's all messages you are likely to get
"non-consecutive" messages. But again if we decide transaction ordering
based consumption is common and necessary we may likely just to this while
implementing the feature.


Guozhang



On Tue, Jul 4, 2017 at 4:29 AM, Daehn, Werner <we...@sap.com> wrote:

> While the new transactional consistency feature is definitely a move into
> the right direction, I find the current design limitations quite severe.
> Understandable but severe.
>
> First is the offset being the add-time, not the commit time. The
> consequence of that is, if one transactional producer adds rows every once
> a while and does commit minutes or hours later, all consumers with
> read=committed will not see any other producer's data until this one did
> the commit. Will decrease the stability of the overall solution, won't it?
> Would have been better to use the commit time as the offset value. Harder
> to implement but more stable.
>
> The other issue is at the consumer side. Current implementation does not
> provide transaction guarantees on the consumers. Again, for logical reasons
> and to simplify the implementation, but as a user you want to have both -
> when data is committed together you want to read the entire transaction as
> one unit and complete.
>
> Are there any plans to go beyond the current implementation?
>
> Thanks in advance
>



-- 
-- Guozhang