Re: [DISCUSS] KIP-890 Server Side Defense

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Justine,

Thanks for the update and sorry for the late reply.

120. I am wondering what value is used for
ClientTransactionProtocolVersion. Is it the version of the EndTxnRequest?

121. Earlier, you made the change to set lastProducerId in PREPARE to
indicate that the marker is written for the new client. With the new
ClientTransactionProtocolVersion field, it seems this is no longer
necessary.

Jun

On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan <jo...@confluent.io.invalid>
wrote:

> Hi there -- another update!
>
> When looking into the implementation for the safe epoch bumps I realized
> that we are already populating previousProducerID in memory as part of
> KIP-360.
> If we are to start using flexible fields, it is better to always use this
> information and have an explicit (tagged) field to indicate whether the
> client supports KIP-890 part 2.
>
> I've included the extra field and how it is set in the KIP. I've also
> updated the KIP to explain that we will be setting the tagged fields when
> they are available for all transitions.
>
> Finally, I added clearer text about the transaction protocol versions
> included with this KIP. 1 for flexible transaction state records and 2 for
> KIP-890 part 2 enablement.
>
> Justine
>
> On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <jo...@confluent.io>
> wrote:
>
> > Hey there -- small update to the KIP,
> >
> > The KIP mentioned introducing ABORTABLE_ERROR and bumping TxnOffsetCommit
> > and Produce requests. I've changed the name in the KIP to
> > ABORTABLE_TRANSACTION and the corresponding exception
> > AbortableTransactionException to match the pattern we had for other
> errors.
> > I also mentioned bumping all 6 transactional APIs so we can future
> > proof/support the error on the client going forward. If a future change
> > wants to have an error scenario that requires us to abort the
> transaction,
> > we can rely on the 3.8+ clients to support it. We ran into issues finding
> > good/generic error codes that older clients could support while working
> on
> > this KIP, so this should help in the future.
> >
> > The features discussion is still ongoing in KIP-1022. Will update again
> > here when that concludes.
> >
> > Justine
> >
> > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <jo...@confluent.io>
> > wrote:
> >
> >> I don't think AddPartitions is a good example since we currenly don't
> >> gate the version on TV or MV. (We only set a different flag depending on
> >> the TV)
> >>
> >> Even if we did want to gate it on TV, I think the idea is to move away
> >> from MV gating inter broker protocols. Ideally we can get to a state
> where
> >> MV is just used for metadata changes.
> >>
> >> I think some of this discussion might fit more with the feature version
> >> KIP, so I can try to open that up soon. Until we settle that, some of
> the
> >> work in KIP-890 is blocked.
> >>
> >> Justine
> >>
> >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao <ju...@confluent.io.invalid>
> wrote:
> >>
> >>> Hi, Justine,
> >>>
> >>> Thanks for the reply.
> >>>
> >>> Since AddPartitions is an inter broker request, will its version be
> gated
> >>> only by TV or other features like MV too? For example, if we need to
> >>> change
> >>> the protocol for AddPartitions for reasons other than txn verification
> in
> >>> the future, will the new version be gated by a new MV? If so, does
> >>> downgrading a TV imply potential downgrade of MV too?
> >>>
> >>> Jun
> >>>
> >>>
> >>>
> >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> >>> <jo...@confluent.io.invalid>
> >>> wrote:
> >>>
> >>> > One TV gates the flexible feature version (no rpcs involved, only the
> >>> > transactional records that should only be gated by TV)
> >>> > Another TV gates the ability to turn on kip-890 part 2. This would
> >>> gate the
> >>> > version of Produce and EndTxn (likely only used by transactions), and
> >>> > specifies a flag in AddPartitionsToTxn though the version is already
> >>> used
> >>> > without TV.
> >>> >
> >>> > I think the only concern is the Produce request and we could consider
> >>> work
> >>> > arounds similar to the AddPartitionsToTxn call.
> >>> >
> >>> > Justine
> >>> >
> >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao <ju...@confluent.io.invalid>
> >>> wrote:
> >>> >
> >>> > > Hi, Justine,
> >>> > >
> >>> > > Which PRC/record protocols will TV guard? Going forward, will those
> >>> > > PRC/record protocols only be guarded by TV and not by other
> features
> >>> like
> >>> > > MV?
> >>> > >
> >>> > > Thanks,
> >>> > >
> >>> > > Jun
> >>> > >
> >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> >>> > <jolshan@confluent.io.invalid
> >>> > > >
> >>> > > wrote:
> >>> > >
> >>> > > > Hi Jun,
> >>> > > >
> >>> > > > Sorry I think I misunderstood your question or answered
> >>> incorrectly.
> >>> > The
> >>> > > TV
> >>> > > > version should ideally be fully independent from MV.
> >>> > > > At least for the changes I proposed, TV should not affect MV and
> MV
> >>> > > should
> >>> > > > not affect TV/
> >>> > > >
> >>> > > > I think if we downgrade TV, only that feature should downgrade.
> >>> > Likewise
> >>> > > > the same with MV. The finalizedFeatures should just reflect the
> >>> feature
> >>> > > > downgrade we made.
> >>> > > >
> >>> > > > I also plan to write a new KIP for managing the disk format and
> >>> upgrade
> >>> > > > tool as we will need new flags to support these features. That
> >>> should
> >>> > > help
> >>> > > > clarify some things.
> >>> > > >
> >>> > > > Justine
> >>> > > >
> >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao <jun@confluent.io.invalid
> >
> >>> > > wrote:
> >>> > > >
> >>> > > > > Hi, Justine,
> >>> > > > >
> >>> > > > > Thanks for the reply.
> >>> > > > >
> >>> > > > > So, if we downgrade TV, we could implicitly downgrade another
> >>> feature
> >>> > > > (say
> >>> > > > > MV) that has dependency (e.g. RPC). What would we return for
> >>> > > > > FinalizedFeatures for MV in ApiVersionsResponse in that case?
> >>> > > > >
> >>> > > > > Thanks,
> >>> > > > >
> >>> > > > > Jun
> >>> > > > >
> >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> >>> > > > <jolshan@confluent.io.invalid
> >>> > > > > >
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > > > Hey Jun,
> >>> > > > > >
> >>> > > > > > Yes, the idea is that if we downgrade TV (transaction
> version)
> >>> we
> >>> > > will
> >>> > > > > stop
> >>> > > > > > using the add partitions to txn optimization and stop writing
> >>> the
> >>> > > > > flexible
> >>> > > > > > feature version of the log.
> >>> > > > > > In the compatibility section I included some explanations on
> >>> how
> >>> > this
> >>> > > > is
> >>> > > > > > done.
> >>> > > > > >
> >>> > > > > > Thanks,
> >>> > > > > > Justine
> >>> > > > > >
> >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> >>> <ju...@confluent.io.invalid>
> >>> > > > > wrote:
> >>> > > > > >
> >>> > > > > > > Hi, Justine,
> >>> > > > > > >
> >>> > > > > > > Thanks for the update.
> >>> > > > > > >
> >>> > > > > > > If we ever downgrade the transaction feature, any feature
> >>> > depending
> >>> > > > on
> >>> > > > > > > changes on top of those RPC/record
> >>> > > > > > > (AddPartitionsToTxnRequest/TransactionLogValue) changes
> made
> >>> in
> >>> > > > KIP-890
> >>> > > > > > > will be automatically downgraded too?
> >>> > > > > > >
> >>> > > > > > > Jun
> >>> > > > > > >
> >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine Olshan
> >>> > > > > > > <jo...@confluent.io.invalid>
> >>> > > > > > > wrote:
> >>> > > > > > >
> >>> > > > > > > > Hey Jun,
> >>> > > > > > > >
> >>> > > > > > > > I wanted to get back to you about your questions about
> >>> MV/IBP.
> >>> > > > > > > >
> >>> > > > > > > > Looking at the options, I think it makes the most sense
> to
> >>> > > create a
> >>> > > > > > > > separate feature for transactions and use that to version
> >>> gate
> >>> > > the
> >>> > > > > > > features
> >>> > > > > > > > we need to version gate (flexible transactional state
> >>> records
> >>> > and
> >>> > > > > using
> >>> > > > > > > the
> >>> > > > > > > > new protocol)
> >>> > > > > > > > I've updated the KIP to include this change. Hopefully
> >>> that's
> >>> > > > > > everything
> >>> > > > > > > we
> >>> > > > > > > > need for this KIP :)
> >>> > > > > > > >
> >>> > > > > > > > Justine
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine Olshan <
> >>> > > > jolshan@confluent.io
> >>> > > > > >
> >>> > > > > > > > wrote:
> >>> > > > > > > >
> >>> > > > > > > > > Thanks Jun,
> >>> > > > > > > > >
> >>> > > > > > > > > I will update the KIP with the prev field for prepare
> as
> >>> > well.
> >>> > > > > > > > >
> >>> > > > > > > > > PREPARE
> >>> > > > > > > > > producerId: x
> >>> > > > > > > > > previous/lastProducerId (tagged field): x
> >>> > > > > > > > > nextProducerId (tagged field): empty or z if y will
> >>> overflow
> >>> > > > > > > > > producerEpoch: y + 1
> >>> > > > > > > > >
> >>> > > > > > > > > COMPLETE
> >>> > > > > > > > > producerId: x or z if y overflowed
> >>> > > > > > > > > previous/lastProducerId (tagged field): x
> >>> > > > > > > > > nextProducerId (tagged field): empty
> >>> > > > > > > > > producerEpoch: y + 1 or 0 if we overflowed
> >>> > > > > > > > >
> >>> > > > > > > > > Thanks again,
> >>> > > > > > > > > Justine
> >>> > > > > > > > >
> >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun Rao
> >>> > > <jun@confluent.io.invalid
> >>> > > > >
> >>> > > > > > > > wrote:
> >>> > > > > > > > >
> >>> > > > > > > > >> Hi, Justine,
> >>> > > > > > > > >>
> >>> > > > > > > > >> 101.3 Thanks for the explanation.
> >>> > > > > > > > >> (1) My point was that the coordinator could fail right
> >>> after
> >>> > > > > writing
> >>> > > > > > > the
> >>> > > > > > > > >> prepare marker. When the new txn coordinator generates
> >>> the
> >>> > > > > complete
> >>> > > > > > > > marker
> >>> > > > > > > > >> after the failover, it needs some field from the
> prepare
> >>> > > marker
> >>> > > > to
> >>> > > > > > > > >> determine whether it's written by the new client.
> >>> > > > > > > > >>
> >>> > > > > > > > >> (2) The changing of the behavior sounds good to me. We
> >>> only
> >>> > > want
> >>> > > > > to
> >>> > > > > > > > return
> >>> > > > > > > > >> success if the prepare state is written by the new
> >>> client.
> >>> > So,
> >>> > > > in
> >>> > > > > > the
> >>> > > > > > > > >> non-overflow case, it seems that we also need sth in
> the
> >>> > > prepare
> >>> > > > > > > marker
> >>> > > > > > > > to
> >>> > > > > > > > >> tell us whether it's written by the new client.
> >>> > > > > > > > >>
> >>> > > > > > > > >> 112. Thanks for the explanation. That sounds good to
> me.
> >>> > > > > > > > >>
> >>> > > > > > > > >> Jun
> >>> > > > > > > > >>
> >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM Justine Olshan
> >>> > > > > > > > >> <jo...@confluent.io.invalid> wrote:
> >>> > > > > > > > >>
> >>> > > > > > > > >> > 101.3 I realized that I actually have two questions.
> >>> > > > > > > > >> > > (1) In the non-overflow case, we need to write the
> >>> > > previous
> >>> > > > > > > produce
> >>> > > > > > > > Id
> >>> > > > > > > > >> > tagged field in the end maker so that we know if the
> >>> > marker
> >>> > > is
> >>> > > > > > from
> >>> > > > > > > > the
> >>> > > > > > > > >> new
> >>> > > > > > > > >> > client. Since the end maker is derived from the
> >>> prepare
> >>> > > > marker,
> >>> > > > > > > should
> >>> > > > > > > > >> we
> >>> > > > > > > > >> > write the previous produce Id in the prepare marker
> >>> field
> >>> > > too?
> >>> > > > > > > > >> Otherwise,
> >>> > > > > > > > >> > we will lose this information when deriving the end
> >>> > marker.
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > The "previous" producer ID is in the normal producer
> >>> ID
> >>> > > field.
> >>> > > > > So
> >>> > > > > > > yes,
> >>> > > > > > > > >> we
> >>> > > > > > > > >> > need it in prepare and that was always the plan.
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > Maybe it is a bit unclear so I will enumerate the
> >>> fields
> >>> > and
> >>> > > > add
> >>> > > > > > > them
> >>> > > > > > > > to
> >>> > > > > > > > >> > the KIP if that helps.
> >>> > > > > > > > >> > Say we have producer ID x and epoch y. When we
> >>> overflow
> >>> > > epoch
> >>> > > > y
> >>> > > > > we
> >>> > > > > > > get
> >>> > > > > > > > >> > producer ID Z.
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > PREPARE
> >>> > > > > > > > >> > producerId: x
> >>> > > > > > > > >> > previous/lastProducerId (tagged field): empty
> >>> > > > > > > > >> > nextProducerId (tagged field): empty or z if y will
> >>> > overflow
> >>> > > > > > > > >> > producerEpoch: y + 1
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > COMPLETE
> >>> > > > > > > > >> > producerId: x or z if y overflowed
> >>> > > > > > > > >> > previous/lastProducerId (tagged field): x
> >>> > > > > > > > >> > nextProducerId (tagged field): empty
> >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we overflowed
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > (2) In the prepare phase, if we retry and see epoch
> -
> >>> 1 +
> >>> > ID
> >>> > > > in
> >>> > > > > > last
> >>> > > > > > > > >> seen
> >>> > > > > > > > >> > fields and are issuing the same command (ie commit
> not
> >>> > > abort),
> >>> > > > > we
> >>> > > > > > > > return
> >>> > > > > > > > >> > success. The logic before KIP-890 seems to return
> >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> >>> > > > > > > > >> > in this case. Are we intentionally making this
> change?
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > Hmm -- we would fence the producer if the epoch is
> >>> bumped
> >>> > > and
> >>> > > > we
> >>> > > > > > > get a
> >>> > > > > > > > >> > lower epoch. Yes -- we are intentionally adding this
> >>> to
> >>> > > > prevent
> >>> > > > > > > > fencing.
> >>> > > > > > > > >> >
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > 112. We already merged the code that adds the
> >>> VerifyOnly
> >>> > > field
> >>> > > > > in
> >>> > > > > > > > >> > AddPartitionsToTxnRequest, which is an inter broker
> >>> > request.
> >>> > > > It
> >>> > > > > > > seems
> >>> > > > > > > > >> that
> >>> > > > > > > > >> > we didn't bump up the IBP for that. Do you know why?
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > We no longer need IBP for all interbroker requests
> as
> >>> > > > > ApiVersions
> >>> > > > > > > > should
> >>> > > > > > > > >> > correctly gate versioning.
> >>> > > > > > > > >> > We also handle unsupported version errors correctly
> >>> if we
> >>> > > > > receive
> >>> > > > > > > them
> >>> > > > > > > > >> in
> >>> > > > > > > > >> > edge cases like upgrades/downgrades.
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > Justine
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM Jun Rao
> >>> > > > > <jun@confluent.io.invalid
> >>> > > > > > >
> >>> > > > > > > > >> wrote:
> >>> > > > > > > > >> >
> >>> > > > > > > > >> > > Hi, Justine,
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > Thanks for the reply.
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > 101.3 I realized that I actually have two
> questions.
> >>> > > > > > > > >> > > (1) In the non-overflow case, we need to write the
> >>> > > previous
> >>> > > > > > > produce
> >>> > > > > > > > Id
> >>> > > > > > > > >> > > tagged field in the end maker so that we know if
> the
> >>> > > marker
> >>> > > > is
> >>> > > > > > > from
> >>> > > > > > > > >> the
> >>> > > > > > > > >> > new
> >>> > > > > > > > >> > > client. Since the end maker is derived from the
> >>> prepare
> >>> > > > > marker,
> >>> > > > > > > > >> should we
> >>> > > > > > > > >> > > write the previous produce Id in the prepare
> marker
> >>> > field
> >>> > > > too?
> >>> > > > > > > > >> Otherwise,
> >>> > > > > > > > >> > > we will lose this information when deriving the
> end
> >>> > > marker.
> >>> > > > > > > > >> > > (2) In the prepare phase, if we retry and see
> epoch
> >>> - 1
> >>> > +
> >>> > > ID
> >>> > > > > in
> >>> > > > > > > last
> >>> > > > > > > > >> seen
> >>> > > > > > > > >> > > fields and are issuing the same command (ie commit
> >>> not
> >>> > > > abort),
> >>> > > > > > we
> >>> > > > > > > > >> return
> >>> > > > > > > > >> > > success. The logic before KIP-890 seems to return
> >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> >>> > > > > > > > >> > > in this case. Are we intentionally making this
> >>> change?
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > 112. We already merged the code that adds the
> >>> VerifyOnly
> >>> > > > field
> >>> > > > > > in
> >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which is an inter
> broker
> >>> > > request.
> >>> > > > > It
> >>> > > > > > > > seems
> >>> > > > > > > > >> > that
> >>> > > > > > > > >> > > we didn't bump up the IBP for that. Do you know
> why?
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > Jun
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM Justine Olshan
> >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> >>> > > > > > > > >> > > wrote:
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> > > > Hi Jun,
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > 101.3 I can change "last seen" to "current
> >>> producer id
> >>> > > and
> >>> > > > > > > epoch"
> >>> > > > > > > > if
> >>> > > > > > > > >> > that
> >>> > > > > > > > >> > > > was the part that was confusing
> >>> > > > > > > > >> > > > 110 I can mention this
> >>> > > > > > > > >> > > > 111 I can do that
> >>> > > > > > > > >> > > > 112 We still need it. But I am still finalizing
> >>> the
> >>> > > > design.
> >>> > > > > I
> >>> > > > > > > will
> >>> > > > > > > > >> > update
> >>> > > > > > > > >> > > > the KIP once I get the information finalized.
> >>> Sorry
> >>> > for
> >>> > > > the
> >>> > > > > > > > delays.
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > Justine
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at 10:50 AM Jun Rao
> >>> > > > > > > <jun@confluent.io.invalid
> >>> > > > > > > > >
> >>> > > > > > > > >> > > wrote:
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > > Hi, Justine,
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > > > Thanks for the reply.
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > > > 101.3 In the non-overflow case, the previous
> ID
> >>> is
> >>> > the
> >>> > > > > same
> >>> > > > > > as
> >>> > > > > > > > the
> >>> > > > > > > > >> > > > produce
> >>> > > > > > > > >> > > > > ID for the complete marker too, but we set the
> >>> > > previous
> >>> > > > ID
> >>> > > > > > in
> >>> > > > > > > > the
> >>> > > > > > > > >> > > > complete
> >>> > > > > > > > >> > > > > marker. Earlier you mentioned that this is to
> >>> know
> >>> > > that
> >>> > > > > the
> >>> > > > > > > > >> marker is
> >>> > > > > > > > >> > > > > written by the new client so that we could
> >>> return
> >>> > > > success
> >>> > > > > on
> >>> > > > > > > > >> retried
> >>> > > > > > > > >> > > > > endMarker requests. I was trying to understand
> >>> why
> >>> > > this
> >>> > > > is
> >>> > > > > > not
> >>> > > > > > > > >> needed
> >>> > > > > > > > >> > > for
> >>> > > > > > > > >> > > > > the prepare marker since retry can happen in
> the
> >>> > > prepare
> >>> > > > > > state
> >>> > > > > > > > >> too.
> >>> > > > > > > > >> > Is
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > reason that in the prepare state, we return
> >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> >>> > > > > > > > >> > > > instead
> >>> > > > > > > > >> > > > > of success on retried endMaker requests? If
> so,
> >>> > should
> >>> > > > we
> >>> > > > > > > change
> >>> > > > > > > > >> "If
> >>> > > > > > > > >> > we
> >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID in last seen
> >>> fields and
> >>> > > are
> >>> > > > > > > issuing
> >>> > > > > > > > >> the
> >>> > > > > > > > >> > > same
> >>> > > > > > > > >> > > > > command (ie commit not abort) we can return
> >>> (with
> >>> > the
> >>> > > > new
> >>> > > > > > > > epoch)"
> >>> > > > > > > > >> > > > > accordingly?
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > > > 110. Yes, without this KIP, a delayed endMaker
> >>> > request
> >>> > > > > > carries
> >>> > > > > > > > the
> >>> > > > > > > > >> > same
> >>> > > > > > > > >> > > > > epoch and won't be fenced. This can
> >>> commit/abort a
> >>> > > > future
> >>> > > > > > > > >> transaction
> >>> > > > > > > > >> > > > > unexpectedly. I am not sure if we have seen
> >>> this in
> >>> > > > > practice
> >>> > > > > > > > >> though.
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > > > 111. Sounds good. It would be useful to make
> it
> >>> > clear
> >>> > > > that
> >>> > > > > > we
> >>> > > > > > > > can
> >>> > > > > > > > >> now
> >>> > > > > > > > >> > > > > populate the lastSeen field from the log
> >>> reliably.
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> >>> > AddPartitionsToTxnRequest
> >>> > > > > since
> >>> > > > > > > > it's
> >>> > > > > > > > >> > > called
> >>> > > > > > > > >> > > > > across brokers and we are changing its schema.
> >>> Are
> >>> > you
> >>> > > > > > saying
> >>> > > > > > > we
> >>> > > > > > > > >> > don't
> >>> > > > > > > > >> > > > need
> >>> > > > > > > > >> > > > > it any more? I thought that we already
> >>> implemented
> >>> > the
> >>> > > > > > server
> >>> > > > > > > > side
> >>> > > > > > > > >> > > > > verification logic based on
> >>> > AddPartitionsToTxnRequest
> >>> > > > > across
> >>> > > > > > > > >> brokers.
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > > > Jun
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at 5:05 PM Justine Olshan
> >>> > > > > > > > >> > > > > <jo...@confluent.io.invalid>
> >>> > > > > > > > >> > > > > wrote:
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > > > > Hey Jun,
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > 101.3 We don't set the previous ID in the
> >>> Prepare
> >>> > > > field
> >>> > > > > > > since
> >>> > > > > > > > we
> >>> > > > > > > > >> > > don't
> >>> > > > > > > > >> > > > > need
> >>> > > > > > > > >> > > > > > it. It is the same producer ID as the main
> >>> > producer
> >>> > > ID
> >>> > > > > > > field.
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to reread your
> message
> >>> > about
> >>> > > > > > delayed
> >>> > > > > > > > >> > markers.
> >>> > > > > > > > >> > > > If
> >>> > > > > > > > >> > > > > we
> >>> > > > > > > > >> > > > > > receive a delayed endTxn marker after the
> >>> > > transaction
> >>> > > > is
> >>> > > > > > > > already
> >>> > > > > > > > >> > > > > complete?
> >>> > > > > > > > >> > > > > > So we will commit the next transaction early
> >>> > without
> >>> > > > the
> >>> > > > > > > fixes
> >>> > > > > > > > >> in
> >>> > > > > > > > >> > > part
> >>> > > > > > > > >> > > > 2?
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > 111 Yes -- this terminology was used in a
> >>> previous
> >>> > > KIP
> >>> > > > > and
> >>> > > > > > > > never
> >>> > > > > > > > >> > > > > > implemented it in the log -- only in memory
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > 112 Hmm -- which interbroker protocol are
> you
> >>> > > > referring
> >>> > > > > > to?
> >>> > > > > > > I
> >>> > > > > > > > am
> >>> > > > > > > > >> > > > working
> >>> > > > > > > > >> > > > > on
> >>> > > > > > > > >> > > > > > the design for the work to remove the extra
> >>> add
> >>> > > > > partitions
> >>> > > > > > > > call
> >>> > > > > > > > >> > and I
> >>> > > > > > > > >> > > > > right
> >>> > > > > > > > >> > > > > > now the design bumps MV. I have yet to
> update
> >>> that
> >>> > > > > section
> >>> > > > > > > as
> >>> > > > > > > > I
> >>> > > > > > > > >> > > > finalize
> >>> > > > > > > > >> > > > > > the design so please stay tuned. Was there
> >>> > anything
> >>> > > > else
> >>> > > > > > you
> >>> > > > > > > > >> > thought
> >>> > > > > > > > >> > > > > needed
> >>> > > > > > > > >> > > > > > MV bump?
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > Justine
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at 3:07 PM Jun Rao
> >>> > > > > > > > >> <ju...@confluent.io.invalid>
> >>> > > > > > > > >> > > > > wrote:
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > > Hi, Justine,
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > I don't see this create any issue. It just
> >>> makes
> >>> > > it
> >>> > > > a
> >>> > > > > > bit
> >>> > > > > > > > >> hard to
> >>> > > > > > > > >> > > > > explain
> >>> > > > > > > > >> > > > > > > what this non-tagged produce id field
> >>> means. We
> >>> > > are
> >>> > > > > > > > >> essentially
> >>> > > > > > > > >> > > > trying
> >>> > > > > > > > >> > > > > to
> >>> > > > > > > > >> > > > > > > combine two actions (completing a txn and
> >>> init a
> >>> > > new
> >>> > > > > > > produce
> >>> > > > > > > > >> Id)
> >>> > > > > > > > >> > > in a
> >>> > > > > > > > >> > > > > > > single record. But, this may be fine too.
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > A few other follow up comments.
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > 101.3 I guess the reason that we only set
> >>> the
> >>> > > > previous
> >>> > > > > > > > >> produce id
> >>> > > > > > > > >> > > > > tagged
> >>> > > > > > > > >> > > > > > > field in the complete marker, but not in
> the
> >>> > > prepare
> >>> > > > > > > marker,
> >>> > > > > > > > >> is
> >>> > > > > > > > >> > > that
> >>> > > > > > > > >> > > > in
> >>> > > > > > > > >> > > > > > the
> >>> > > > > > > > >> > > > > > > prepare state, we always return
> >>> > > > > CONCURRENT_TRANSACTIONS
> >>> > > > > > on
> >>> > > > > > > > >> > retried
> >>> > > > > > > > >> > > > > > endMaker
> >>> > > > > > > > >> > > > > > > requests?
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > 110. "I believe your second point is
> >>> mentioned
> >>> > in
> >>> > > > the
> >>> > > > > > > KIP. I
> >>> > > > > > > > >> can
> >>> > > > > > > > >> > > add
> >>> > > > > > > > >> > > > > more
> >>> > > > > > > > >> > > > > > > text on
> >>> > > > > > > > >> > > > > > > this if it is helpful.
> >>> > > > > > > > >> > > > > > > > The delayed message case can also
> violate
> >>> EOS
> >>> > if
> >>> > > > the
> >>> > > > > > > > delayed
> >>> > > > > > > > >> > > > message
> >>> > > > > > > > >> > > > > > > comes in after the next addPartitionsToTxn
> >>> > request
> >>> > > > > comes
> >>> > > > > > > in.
> >>> > > > > > > > >> > > > > Effectively
> >>> > > > > > > > >> > > > > > we
> >>> > > > > > > > >> > > > > > > may see a message from a previous
> (aborted)
> >>> > > > > transaction
> >>> > > > > > > > become
> >>> > > > > > > > >> > part
> >>> > > > > > > > >> > > > of
> >>> > > > > > > > >> > > > > > the
> >>> > > > > > > > >> > > > > > > next transaction."
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > The above is the case when a delayed
> >>> message is
> >>> > > > > appended
> >>> > > > > > > to
> >>> > > > > > > > >> the
> >>> > > > > > > > >> > > data
> >>> > > > > > > > >> > > > > > > partition. What I mentioned is a slightly
> >>> > > different
> >>> > > > > case
> >>> > > > > > > > when
> >>> > > > > > > > >> a
> >>> > > > > > > > >> > > > delayed
> >>> > > > > > > > >> > > > > > > marker is appended to the transaction log
> >>> > > partition.
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > 111. The KIP says "Once we move past the
> >>> Prepare
> >>> > > and
> >>> > > > > > > > Complete
> >>> > > > > > > > >> > > states,
> >>> > > > > > > > >> > > > > we
> >>> > > > > > > > >> > > > > > > don’t need to worry about lastSeen fields
> >>> and
> >>> > > clear
> >>> > > > > > them,
> >>> > > > > > > > just
> >>> > > > > > > > >> > > handle
> >>> > > > > > > > >> > > > > > state
> >>> > > > > > > > >> > > > > > > transitions as normal.". Is the lastSeen
> >>> field
> >>> > the
> >>> > > > > same
> >>> > > > > > as
> >>> > > > > > > > the
> >>> > > > > > > > >> > > > previous
> >>> > > > > > > > >> > > > > > > Produce Id tagged field in
> >>> TransactionLogValue?
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > 112. Since the kip changes the
> inter-broker
> >>> > > > protocol,
> >>> > > > > > > should
> >>> > > > > > > > >> we
> >>> > > > > > > > >> > > bump
> >>> > > > > > > > >> > > > up
> >>> > > > > > > > >> > > > > > the
> >>> > > > > > > > >> > > > > > > MV/IBP version? Is this feature only for
> the
> >>> > KRaft
> >>> > > > > mode?
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > Thanks,
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > Jun
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at 11:13 AM Justine
> >>> Olshan
> >>> > > > > > > > >> > > > > > > <jo...@confluent.io.invalid> wrote:
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > > > > Hey Jun,
> >>> > > > > > > > >> > > > > > > >
> >>> > > > > > > > >> > > > > > > > I'm glad we are getting to convergence
> on
> >>> the
> >>> > > > > design.
> >>> > > > > > :)
> >>> > > > > > > > >> > > > > > > >
> >>> > > > > > > > >> > > > > > > > While I understand it seems a little
> >>> "weird".
> >>> > > I'm
> >>> > > > > not
> >>> > > > > > > sure
> >>> > > > > > > > >> what
> >>> > > > > > > > >> > > the
> >>> > > > > > > > >> > > > > > > benefit
> >>> > > > > > > > >> > > > > > > > of writing an extra record to the log.
> >>> > > > > > > > >> > > > > > > > Is the concern a tool to describe
> >>> transactions
> >>> > > > won't
> >>> > > > > > > work
> >>> > > > > > > > >> (ie,
> >>> > > > > > > > >> > > the
> >>> > > > > > > > >> > > > > > > complete
> >>> > > > > > > > >> > > > > > > > state is needed to calculate the time
> >>> since
> >>> > the
> >>> > > > > > > > transaction
> >>> > > > > > > > >> > > > > completed?)
> >>> > > > > > > > >> > > > > > > > If we have a reason like this, it is
> >>> enough to
> >>> > > > > > convince
> >>> > > > > > > me
> >>> > > > > > > > >> we
> >>> > > > > > > > >> > > need
> >>> > > > > > > > >> > > > > such
> >>> > > > > > > > >> > > > > > > an
> >>> > > > > > > > >> > > > > > > > extra record. It seems like it would be
> >>> > > replacing
> >>> > > > > the
> >>> > > > > > > > record
> >>> > > > > > > > >> > > > written
> >>> > > > > > > > >> > > > > on
> >>> > > > > > > > >> > > > > > > > InitProducerId. Is this correct?
> >>> > > > > > > > >> > > > > > > >
> >>> > > > > > > > >> > > > > > > > Thanks,
> >>> > > > > > > > >> > > > > > > > Justine
> >>> > > > > > > > >> > > > > > > >
> >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024 at 5:14 PM Jun Rao
> >>> > > > > > > > >> > <jun@confluent.io.invalid
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > > > > > > wrote:
> >>> > > > > > > > >> > > > > > > >
> >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > Thanks for the explanation. I
> >>> understand the
> >>> > > > > > intention
> >>> > > > > > > > >> now.
> >>> > > > > > > > >> > In
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > > > > overflow
> >>> > > > > > > > >> > > > > > > > > case, we set the non-tagged field to
> >>> the old
> >>> > > pid
> >>> > > > > > (and
> >>> > > > > > > > the
> >>> > > > > > > > >> max
> >>> > > > > > > > >> > > > > epoch)
> >>> > > > > > > > >> > > > > > in
> >>> > > > > > > > >> > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > prepare marker so that we could
> >>> correctly
> >>> > > write
> >>> > > > > the
> >>> > > > > > > > >> marker to
> >>> > > > > > > > >> > > the
> >>> > > > > > > > >> > > > > > data
> >>> > > > > > > > >> > > > > > > > > partition if the broker downgrades.
> When
> >>> > > writing
> >>> > > > > the
> >>> > > > > > > > >> complete
> >>> > > > > > > > >> > > > > marker,
> >>> > > > > > > > >> > > > > > > we
> >>> > > > > > > > >> > > > > > > > > know the marker has already been
> >>> written to
> >>> > > the
> >>> > > > > data
> >>> > > > > > > > >> > partition.
> >>> > > > > > > > >> > > > We
> >>> > > > > > > > >> > > > > > set
> >>> > > > > > > > >> > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > non-tagged field to the new pid to
> avoid
> >>> > > > > > > > >> > > > InvalidPidMappingException
> >>> > > > > > > > >> > > > > > in
> >>> > > > > > > > >> > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > client if the broker downgrades.
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > The above seems to work. It's just a
> bit
> >>> > > > > > inconsistent
> >>> > > > > > > > for
> >>> > > > > > > > >> a
> >>> > > > > > > > >> > > > prepare
> >>> > > > > > > > >> > > > > > > > marker
> >>> > > > > > > > >> > > > > > > > > and a complete marker to use different
> >>> pids
> >>> > in
> >>> > > > > this
> >>> > > > > > > > >> special
> >>> > > > > > > > >> > > case.
> >>> > > > > > > > >> > > > > If
> >>> > > > > > > > >> > > > > > we
> >>> > > > > > > > >> > > > > > > > > downgrade with the complete marker, it
> >>> seems
> >>> > > > that
> >>> > > > > we
> >>> > > > > > > > will
> >>> > > > > > > > >> > never
> >>> > > > > > > > >> > > > be
> >>> > > > > > > > >> > > > > > able
> >>> > > > > > > > >> > > > > > > > to
> >>> > > > > > > > >> > > > > > > > > write the complete marker with the old
> >>> pid.
> >>> > > Not
> >>> > > > > sure
> >>> > > > > > > if
> >>> > > > > > > > it
> >>> > > > > > > > >> > > causes
> >>> > > > > > > > >> > > > > any
> >>> > > > > > > > >> > > > > > > > > issue, but it seems a bit weird.
> >>> Instead of
> >>> > > > > writing
> >>> > > > > > > the
> >>> > > > > > > > >> > > complete
> >>> > > > > > > > >> > > > > > marker
> >>> > > > > > > > >> > > > > > > > > with the new pid, could we write two
> >>> > records:
> >>> > > a
> >>> > > > > > > complete
> >>> > > > > > > > >> > marker
> >>> > > > > > > > >> > > > > with
> >>> > > > > > > > >> > > > > > > the
> >>> > > > > > > > >> > > > > > > > > old pid followed by a
> >>> TransactionLogValue
> >>> > with
> >>> > > > the
> >>> > > > > > new
> >>> > > > > > > > pid
> >>> > > > > > > > >> > and
> >>> > > > > > > > >> > > an
> >>> > > > > > > > >> > > > > > empty
> >>> > > > > > > > >> > > > > > > > > state? We could make the two records
> in
> >>> the
> >>> > > same
> >>> > > > > > batch
> >>> > > > > > > > so
> >>> > > > > > > > >> > that
> >>> > > > > > > > >> > > > they
> >>> > > > > > > > >> > > > > > > will
> >>> > > > > > > > >> > > > > > > > be
> >>> > > > > > > > >> > > > > > > > > added to the log atomically.
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > Thanks,
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > Jun
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024 at 5:40 PM
> Justine
> >>> > Olshan
> >>> > > > > > > > >> > > > > > > > > <jo...@confluent.io.invalid>
> >>> > > > > > > > >> > > > > > > > > wrote:
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > (1) the prepare marker is written,
> >>> but the
> >>> > > > > endTxn
> >>> > > > > > > > >> response
> >>> > > > > > > > >> > is
> >>> > > > > > > > >> > > > not
> >>> > > > > > > > >> > > > > > > > > received
> >>> > > > > > > > >> > > > > > > > > > by the client when the server
> >>> downgrades
> >>> > > > > > > > >> > > > > > > > > > (2)  the prepare marker is written,
> >>> the
> >>> > > endTxn
> >>> > > > > > > > response
> >>> > > > > > > > >> is
> >>> > > > > > > > >> > > > > received
> >>> > > > > > > > >> > > > > > > by
> >>> > > > > > > > >> > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > client when the server downgrades.
> >>> > > > > > > > >> > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > I think I am still a little
> confused.
> >>> In
> >>> > > both
> >>> > > > of
> >>> > > > > > > these
> >>> > > > > > > > >> > cases,
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > > > > > > transaction log has the old producer
> >>> ID.
> >>> > We
> >>> > > > > don't
> >>> > > > > > > > write
> >>> > > > > > > > >> the
> >>> > > > > > > > >> > > new
> >>> > > > > > > > >> > > > > > > > producer
> >>> > > > > > > > >> > > > > > > > > ID
> >>> > > > > > > > >> > > > > > > > > > in the prepare marker's non tagged
> >>> fields.
> >>> > > > > > > > >> > > > > > > > > > If the server downgrades now, it
> would
> >>> > read
> >>> > > > the
> >>> > > > > > > > records
> >>> > > > > > > > >> not
> >>> > > > > > > > >> > > in
> >>> > > > > > > > >> > > > > > tagged
> >>> > > > > > > > >> > > > > > > > > > fields and the complete marker will
> >>> also
> >>> > > have
> >>> > > > > the
> >>> > > > > > > old
> >>> > > > > > > > >> > > producer
> >>> > > > > > > > >> > > > > ID.
> >>> > > > > > > > >> > > > > > > > > > (If we had used the new producer ID,
> >>> we
> >>> > > would
> >>> > > > > not
> >>> > > > > > > have
> >>> > > > > > > > >> > > > > > transactional
> >>> > > > > > > > >> > > > > > > > > > correctness since the producer id
> >>> doesn't
> >>> > > > match
> >>> > > > > > the
> >>> > > > > > > > >> > > transaction
> >>> > > > > > > > >> > > > > and
> >>> > > > > > > > >> > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > state would not be correct on the
> data
> >>> > > > > partition.)
> >>> > > > > > > > >> > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > In the overflow case, I'd expect the
> >>> > > following
> >>> > > > > to
> >>> > > > > > > > >> happen on
> >>> > > > > > > > >> > > the
> >>> > > > > > > > >> > > > > > > client
> >>> > > > > > > > >> > > > > > > > > side
> >>> > > > > > > > >> > > > > > > > > > Case 1  -- we retry EndTxn -- it is
> >>> the
> >>> > same
> >>> > > > > > > producer
> >>> > > > > > > > ID
> >>> > > > > > > > >> > and
> >>> > > > > > > > >> > > > > epoch
> >>> > > > > > > > >> > > > > > -
> >>> > > > > > > > >> > > > > > > 1
> >>> > > > > > > > >> > > > > > > > > this
> >>> > > > > > > > >> > > > > > > > > > would fence the producer
> >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't retry EndTxn and
> >>> use
> >>> > the
> >>> > > > new
> >>> > > > > > > > >> producer id
> >>> > > > > > > > >> > > > which
> >>> > > > > > > > >> > > > > > > would
> >>> > > > > > > > >> > > > > > > > > > result in InvalidPidMappingException
> >>> > > > > > > > >> > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > Maybe we can have special handling
> for
> >>> > when
> >>> > > a
> >>> > > > > > server
> >>> > > > > > > > >> > > > downgrades.
> >>> > > > > > > > >> > > > > > When
> >>> > > > > > > > >> > > > > > > > it
> >>> > > > > > > > >> > > > > > > > > > reconnects we could get an API
> version
> >>> > > request
> >>> > > > > > > showing
> >>> > > > > > > > >> > > KIP-890
> >>> > > > > > > > >> > > > > > part 2
> >>> > > > > > > > >> > > > > > > > is
> >>> > > > > > > > >> > > > > > > > > > not supported. In that case, we can
> >>> call
> >>> > > > > > > > initProducerId
> >>> > > > > > > > >> to
> >>> > > > > > > > >> > > > abort
> >>> > > > > > > > >> > > > > > the
> >>> > > > > > > > >> > > > > > > > > > transaction. (In the overflow case,
> >>> this
> >>> > > > > correctly
> >>> > > > > > > > gives
> >>> > > > > > > > >> > us a
> >>> > > > > > > > >> > > > new
> >>> > > > > > > > >> > > > > > > > > producer
> >>> > > > > > > > >> > > > > > > > > > ID)
> >>> > > > > > > > >> > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > I guess the corresponding case would
> >>> be
> >>> > > where
> >>> > > > > the
> >>> > > > > > > > >> *complete
> >>> > > > > > > > >> > > > > marker
> >>> > > > > > > > >> > > > > > > *is
> >>> > > > > > > > >> > > > > > > > > > written but the endTxn is not
> >>> received by
> >>> > > the
> >>> > > > > > client
> >>> > > > > > > > and
> >>> > > > > > > > >> > the
> >>> > > > > > > > >> > > > > server
> >>> > > > > > > > >> > > > > > > > > > downgrades? This would result in the
> >>> > > > transaction
> >>> > > > > > > > >> > coordinator
> >>> > > > > > > > >> > > > > having
> >>> > > > > > > > >> > > > > > > the
> >>> > > > > > > > >> > > > > > > > > new
> >>> > > > > > > > >> > > > > > > > > > ID and not the old one.  If the
> client
> >>> > > > retries,
> >>> > > > > it
> >>> > > > > > > > will
> >>> > > > > > > > >> > > receive
> >>> > > > > > > > >> > > > > an
> >>> > > > > > > > >> > > > > > > > > > InvalidPidMappingException. The
> >>> > > InitProducerId
> >>> > > > > > > > scenario
> >>> > > > > > > > >> > above
> >>> > > > > > > > >> > > > > would
> >>> > > > > > > > >> > > > > > > > help
> >>> > > > > > > > >> > > > > > > > > > here too.
> >>> > > > > > > > >> > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > To be clear, my compatibility story
> is
> >>> > meant
> >>> > > > to
> >>> > > > > > > > support
> >>> > > > > > > > >> > > > > downgrades
> >>> > > > > > > > >> > > > > > > > server
> >>> > > > > > > > >> > > > > > > > > > side in keeping the transactional
> >>> > > correctness.
> >>> > > > > > > Keeping
> >>> > > > > > > > >> the
> >>> > > > > > > > >> > > > client
> >>> > > > > > > > >> > > > > > > from
> >>> > > > > > > > >> > > > > > > > > > fencing itself is not the priority.
> >>> > > > > > > > >> > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > Hope this helps. I can also add text
> >>> in
> >>> > the
> >>> > > > KIP
> >>> > > > > > > about
> >>> > > > > > > > >> > > > > > InitProducerId
> >>> > > > > > > > >> > > > > > > if
> >>> > > > > > > > >> > > > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > think that fixes some edge cases.
> >>> > > > > > > > >> > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > Justine
> >>> > > > > > > > >> > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12, 2024 at 4:10 PM Jun
> >>> Rao
> >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > > > > > > wrote:
> >>> > > > > > > > >> > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> >>> > > > > > > > >> > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > Thanks for the reply.
> >>> > > > > > > > >> > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > I agree that we don't need to
> >>> optimize
> >>> > for
> >>> > > > > > fencing
> >>> > > > > > > > >> during
> >>> > > > > > > > >> > > > > > > downgrades.
> >>> > > > > > > > >> > > > > > > > > > > Regarding consistency, there are
> two
> >>> > > > possible
> >>> > > > > > > cases:
> >>> > > > > > > > >> (1)
> >>> > > > > > > > >> > > the
> >>> > > > > > > > >> > > > > > > prepare
> >>> > > > > > > > >> > > > > > > > > > marker
> >>> > > > > > > > >> > > > > > > > > > > is written, but the endTxn
> response
> >>> is
> >>> > not
> >>> > > > > > > received
> >>> > > > > > > > by
> >>> > > > > > > > >> > the
> >>> > > > > > > > >> > > > > client
> >>> > > > > > > > >> > > > > > > > when
> >>> > > > > > > > >> > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > server downgrades; (2)  the
> prepare
> >>> > marker
> >>> > > > is
> >>> > > > > > > > written,
> >>> > > > > > > > >> > the
> >>> > > > > > > > >> > > > > endTxn
> >>> > > > > > > > >> > > > > > > > > > response
> >>> > > > > > > > >> > > > > > > > > > > is received by the client when the
> >>> > server
> >>> > > > > > > > downgrades.
> >>> > > > > > > > >> In
> >>> > > > > > > > >> > > (1),
> >>> > > > > > > > >> > > > > the
> >>> > > > > > > > >> > > > > > > > > client
> >>> > > > > > > > >> > > > > > > > > > > will have the old produce Id and
> in
> >>> (2),
> >>> > > the
> >>> > > > > > > client
> >>> > > > > > > > >> will
> >>> > > > > > > > >> > > have
> >>> > > > > > > > >> > > > > the
> >>> > > > > > > > >> > > > > > > new
> >>> > > > > > > > >> > > > > > > > > > > produce Id. If we downgrade right
> >>> after
> >>> > > the
> >>> > > > > > > prepare
> >>> > > > > > > > >> > marker,
> >>> > > > > > > > >> > > > we
> >>> > > > > > > > >> > > > > > > can't
> >>> > > > > > > > >> > > > > > > > be
> >>> > > > > > > > >> > > > > > > > > > > consistent to both (1) and (2)
> >>> since we
> >>> > > can
> >>> > > > > only
> >>> > > > > > > put
> >>> > > > > > > > >> one
> >>> > > > > > > > >> > > > value
> >>> > > > > > > > >> > > > > in
> >>> > > > > > > > >> > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > existing produce Id field. It's
> >>> also not
> >>> > > > clear
> >>> > > > > > > which
> >>> > > > > > > > >> case
> >>> > > > > > > > >> > > is
> >>> > > > > > > > >> > > > > more
> >>> > > > > > > > >> > > > > > > > > likely.
> >>> > > > > > > > >> > > > > > > > > > > So we could probably be consistent
> >>> with
> >>> > > > either
> >>> > > > > > > case.
> >>> > > > > > > > >> By
> >>> > > > > > > > >> > > > putting
> >>> > > > > > > > >> > > > > > the
> >>> > > > > > > > >> > > > > > > > new
> >>> > > > > > > > >> > > > > > > > > > > producer Id in the prepare marker,
> >>> we
> >>> > are
> >>> > > > > > > consistent
> >>> > > > > > > > >> with
> >>> > > > > > > > >> > > > case
> >>> > > > > > > > >> > > > > > (2)
> >>> > > > > > > > >> > > > > > > > and
> >>> > > > > > > > >> > > > > > > > > it
> >>> > > > > > > > >> > > > > > > > > > > also has the slight benefit that
> the
> >>> > > produce
> >>> > > > > > field
> >>> > > > > > > > in
> >>> > > > > > > > >> the
> >>> > > > > > > > >> > > > > prepare
> >>> > > > > > > > >> > > > > > > and
> >>> > > > > > > > >> > > > > > > > > > > complete marker are consistent in
> >>> the
> >>> > > > overflow
> >>> > > > > > > case.
> >>> > > > > > > > >> > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > Jun
> >>> > > > > > > > >> > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12, 2024 at 3:11 PM
> >>> Justine
> >>> > > > Olshan
> >>> > > > > > > > >> > > > > > > > > > > <jo...@confluent.io.invalid>
> >>> > > > > > > > >> > > > > > > > > > > wrote:
> >>> > > > > > > > >> > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> >>> > > > > > > > >> > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > In the case you describe, we
> would
> >>> > need
> >>> > > to
> >>> > > > > > have
> >>> > > > > > > a
> >>> > > > > > > > >> > delayed
> >>> > > > > > > > >> > > > > > > request,
> >>> > > > > > > > >> > > > > > > > > > send a
> >>> > > > > > > > >> > > > > > > > > > > > successful EndTxn, and a
> >>> successful
> >>> > > > > > > > >> AddPartitionsToTxn
> >>> > > > > > > > >> > > and
> >>> > > > > > > > >> > > > > then
> >>> > > > > > > > >> > > > > > > > have
> >>> > > > > > > > >> > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn request go
> through
> >>> for
> >>> > a
> >>> > > > > given
> >>> > > > > > > > >> producer.
> >>> > > > > > > > >> > > > > > > > > > > > I'm trying to figure out if it
> is
> >>> > > possible
> >>> > > > > for
> >>> > > > > > > the
> >>> > > > > > > > >> > client
> >>> > > > > > > > >> > > > to
> >>> > > > > > > > >> > > > > > > > > transition
> >>> > > > > > > > >> > > > > > > > > > > if
> >>> > > > > > > > >> > > > > > > > > > > > a previous request is delayed
> >>> > somewhere.
> >>> > > > But
> >>> > > > > > > yes,
> >>> > > > > > > > in
> >>> > > > > > > > >> > this
> >>> > > > > > > > >> > > > > case
> >>> > > > > > > > >> > > > > > I
> >>> > > > > > > > >> > > > > > > > > think
> >>> > > > > > > > >> > > > > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > > > would fence the client.
> >>> > > > > > > > >> > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > Not for the overflow case. In
> the
> >>> > > overflow
> >>> > > > > > case,
> >>> > > > > > > > the
> >>> > > > > > > > >> > > > producer
> >>> > > > > > > > >> > > > > > ID
> >>> > > > > > > > >> > > > > > > > and
> >>> > > > > > > > >> > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > epoch are different on the
> marker
> >>> and
> >>> > on
> >>> > > > the
> >>> > > > > > new
> >>> > > > > > > > >> > > > transaction.
> >>> > > > > > > > >> > > > > > So
> >>> > > > > > > > >> > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > want
> >>> > > > > > > > >> > > > > > > > > > > > the marker to use the max epoch
> >>> but
> >>> > the
> >>> > > > new
> >>> > > > > > > > >> > transaction
> >>> > > > > > > > >> > > > > should
> >>> > > > > > > > >> > > > > > > > start
> >>> > > > > > > > >> > > > > > > > > > > with
> >>> > > > > > > > >> > > > > > > > > > > > the new ID and epoch 0 in the
> >>> > > > transactional
> >>> > > > > > > state.
> >>> > > > > > > > >> > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > In the server downgrade case, we
> >>> want
> >>> > to
> >>> > > > see
> >>> > > > > > the
> >>> > > > > > > > >> > producer
> >>> > > > > > > > >> > > > ID
> >>> > > > > > > > >> > > > > as
> >>> > > > > > > > >> > > > > > > > that
> >>> > > > > > > > >> > > > > > > > > is
> >>> > > > > > > > >> > > > > > > > > > > > what the client will have. If we
> >>> > > complete
> >>> > > > > the
> >>> > > > > > > > >> commit,
> >>> > > > > > > > >> > and
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > > > > > > transaction
> >>> > > > > > > > >> > > > > > > > > > > > state is reloaded, we need the
> new
> >>> > > > producer
> >>> > > > > ID
> >>> > > > > > > in
> >>> > > > > > > > >> the
> >>> > > > > > > > >> > > state
> >>> > > > > > > > >> > > > > so
> >>> > > > > > > > >> > > > > > > > there
> >>> > > > > > > > >> > > > > > > > > > > isn't
> >>> > > > > > > > >> > > > > > > > > > > > an invalid producer ID mapping.
> >>> > > > > > > > >> > > > > > > > > > > > The server downgrade cases are
> >>> > > considering
> >>> > > > > > > > >> > transactional
> >>> > > > > > > > >> > > > > > > > correctness
> >>> > > > > > > > >> > > > > > > > > > and
> >>> > > > > > > > >> > > > > > > > > > > > not regressing from previous
> >>> behavior
> >>> > --
> >>> > > > and
> >>> > > > > > are
> >>> > > > > > > > not
> >>> > > > > > > > >> > > > > concerned
> >>> > > > > > > > >> > > > > > > > about
> >>> > > > > > > > >> > > > > > > > > > > > supporting the safety from
> fencing
> >>> > > retries
> >>> > > > > (as
> >>> > > > > > > we
> >>> > > > > > > > >> have
> >>> > > > > > > > >> > > > > > downgraded
> >>> > > > > > > > >> > > > > > > > so
> >>> > > > > > > > >> > > > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > > > don't need to support). Perhaps
> >>> this
> >>> > is
> >>> > > a
> >>> > > > > > trade
> >>> > > > > > > > off,
> >>> > > > > > > > >> > but
> >>> > > > > > > > >> > > I
> >>> > > > > > > > >> > > > > > think
> >>> > > > > > > > >> > > > > > > it
> >>> > > > > > > > >> > > > > > > > > is
> >>> > > > > > > > >> > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > right one.
> >>> > > > > > > > >> > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > (If the client downgrades, it
> will
> >>> > have
> >>> > > > > > > restarted
> >>> > > > > > > > >> and
> >>> > > > > > > > >> > it
> >>> > > > > > > > >> > > is
> >>> > > > > > > > >> > > > > ok
> >>> > > > > > > > >> > > > > > > for
> >>> > > > > > > > >> > > > > > > > it
> >>> > > > > > > > >> > > > > > > > > > to
> >>> > > > > > > > >> > > > > > > > > > > > have a new producer ID too).
> >>> > > > > > > > >> > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > Justine
> >>> > > > > > > > >> > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan 12, 2024 at 11:42 AM
> >>> Jun
> >>> > Rao
> >>> > > > > > > > >> > > > > > > <jun@confluent.io.invalid
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > wrote:
> >>> > > > > > > > >> > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> >>> > > > > > > > >> > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > Thanks for the reply.
> >>> > > > > > > > >> > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If the marker is
> written
> >>> by
> >>> > the
> >>> > > > new
> >>> > > > > > > > >> client, we
> >>> > > > > > > > >> > > can
> >>> > > > > > > > >> > > > > as
> >>> > > > > > > > >> > > > > > I
> >>> > > > > > > > >> > > > > > > > > > > mentioned
> >>> > > > > > > > >> > > > > > > > > > > > in
> >>> > > > > > > > >> > > > > > > > > > > > > the last email guarantee that
> >>> any
> >>> > > EndTxn
> >>> > > > > > > > requests
> >>> > > > > > > > >> > with
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > > same
> >>> > > > > > > > >> > > > > > > > > epoch
> >>> > > > > > > > >> > > > > > > > > > > are
> >>> > > > > > > > >> > > > > > > > > > > > > from the same producer and the
> >>> same
> >>> > > > > > > transaction.
> >>> > > > > > > > >> Then
> >>> > > > > > > > >> > > we
> >>> > > > > > > > >> > > > > > don't
> >>> > > > > > > > >> > > > > > > > have
> >>> > > > > > > > >> > > > > > > > > > to
> >>> > > > > > > > >> > > > > > > > > > > > > return a fenced error but can
> >>> handle
> >>> > > > > > > gracefully
> >>> > > > > > > > as
> >>> > > > > > > > >> > > > > described
> >>> > > > > > > > >> > > > > > in
> >>> > > > > > > > >> > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > KIP."
> >>> > > > > > > > >> > > > > > > > > > > > > When a delayed EndTnx request
> is
> >>> > > > > processed,
> >>> > > > > > > the
> >>> > > > > > > > >> txn
> >>> > > > > > > > >> > > state
> >>> > > > > > > > >> > > > > > could
> >>> > > > > > > > >> > > > > > > > be
> >>> > > > > > > > >> > > > > > > > > > > > ongoing
> >>> > > > > > > > >> > > > > > > > > > > > > for the next txn. I guess in
> >>> this
> >>> > case
> >>> > > > we
> >>> > > > > > > still
> >>> > > > > > > > >> > return
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > > > fenced
> >>> > > > > > > > >> > > > > > > > > > error
> >>> > > > > > > > >> > > > > > > > > > > > for
> >>> > > > > > > > >> > > > > > > > > > > > > the delayed request?
> >>> > > > > > > > >> > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry, my question was
> >>> > > inaccurate.
> >>> > > > > What
> >>> > > > > > > you
> >>> > > > > > > > >> > > > described
> >>> > > > > > > > >> > > > > is
> >>> > > > > > > > >> > > > > > > > > > accurate.
> >>> > > > > > > > >> > > > > > > > > > > > > "The downgrade compatibility I
> >>> > mention
> >>> > > > is
> >>> > > > > > that
> >>> > > > > > > > we
> >>> > > > > > > > >> > keep
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > > same
> >>> > > > > > > > >> > > > > > > > > > > producer
> >>> > > > > > > > >> > > > > > > > > > > > ID
> >>> > > > > > > > >> > > > > > > > > > > > > and epoch in the main
> >>> (non-tagged)
> >>> > > > fields
> >>> > > > > as
> >>> > > > > > > we
> >>> > > > > > > > >> did
> >>> > > > > > > > >> > > > before
> >>> > > > > > > > >> > > > > > the
> >>> > > > > > > > >> > > > > > > > code
> >>> > > > > > > > >> > > > > > > > > > on
> >>> > > > > > > > >> > > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > server side." If we want to do
> >>> this,
> >>> > > it
> >>> > > > > > seems
> >>> > > > > > > > >> that we
> >>> > > > > > > > >> > > > > should
> >>> > > > > > > > >> > > > > > > use
> >>> > > > > > > > >> > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > current produce Id and max
> >>> epoch in
> >>> > > the
> >>> > > > > > > existing
> >>> > > > > > > > >> > > > producerId
> >>> > > > > > > > >> > > > > > and
> >>> > > > > > > > >> > > > > > > > > > > > > producerEpoch fields for both
> >>> the
> >>> > > > prepare
> >>> > > > > > and
> >>> > > > > > > > the
> >>> > > > > > > > >> > > > complete
> >>> > > > > > > > >> > > > > > > > marker,
> >>> > > > > > > > >> > > > > > > > > > > right?
> >>> > > > > > > > >> > > > > > > > > > > > > The downgrade can happen after
> >>> the
> >>> > > > > complete
> >>> > > > > > > > >> marker is
> >>> > > > > > > > >> > > > > > written.
> >>> > > > > > > > >> > > > > > > > With
> >>> > > > > > > > >> > > > > > > > > > > what
> >>> > > > > > > > >> > > > > > > > > > > > > you described, the downgraded
> >>> > > > coordinator
> >>> > > > > > will
> >>> > > > > > > > see
> >>> > > > > > > > >> > the
> >>> > > > > > > > >> > > > new
> >>> > > > > > > > >> > > > > > > > produce
> >>> > > > > > > > >> > > > > > > > > Id
> >>> > > > > > > > >> > > > > > > > > > > > > instead of the old one.
> >>> > > > > > > > >> > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > Jun
> >>> > > > > > > > >> > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan 12, 2024 at
> 10:44 AM
> >>> > > Justine
> >>> > > > > > > Olshan
> >>> > > > > > > > >> > > > > > > > > > > > > <jolshan@confluent.io.invalid
> >
> >>> > wrote:
> >>> > > > > > > > >> > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> >>> > > > > > > > >> > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > I can update the
> description.
> >>> > > > > > > > >> > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > I believe your second point
> is
> >>> > > > mentioned
> >>> > > > > > in
> >>> > > > > > > > the
> >>> > > > > > > > >> > KIP.
> >>> > > > > > > > >> > > I
> >>> > > > > > > > >> > > > > can
> >>> > > > > > > > >> > > > > > > add
> >>> > > > > > > > >> > > > > > > > > more
> >>> > > > > > > > >> > > > > > > > > > > > text
> >>> > > > > > > > >> > > > > > > > > > > > > on
> >>> > > > > > > > >> > > > > > > > > > > > > > this if it is helpful.
> >>> > > > > > > > >> > > > > > > > > > > > > > > The delayed message case
> can
> >>> > also
> >>> > > > > > violate
> >>> > > > > > > > EOS
> >>> > > > > > > > >> if
> >>> > > > > > > > >> > > the
> >>> > > > > > > > >> > > > > > > delayed
> >>> > > > > > > > >> > > > > > > > > > > message
> >>> > > > > > > > >> > > > > > > > > > > > > > comes in after the next
> >>> > > > > addPartitionsToTxn
> >>> > > > > > > > >> request
> >>> > > > > > > > >> > > > comes
> >>> > > > > > > > >> > > > > > in.
> >>> > > > > > > > >> > > > > > > > > > > > Effectively
> >>> > > > > > > > >> > > > > > > > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > > > > > may see a message from a
> >>> previous
> >>> > > > > > (aborted)
> >>> > > > > > > > >> > > transaction
> >>> > > > > > > > >> > > > > > > become
> >>> > > > > > > > >> > > > > > > > > part
> >>> > > > > > > > >> > > > > > > > > > > of
> >>> > > > > > > > >> > > > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > next transaction.
> >>> > > > > > > > >> > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > If the marker is written by
> >>> the
> >>> > new
> >>> > > > > > client,
> >>> > > > > > > we
> >>> > > > > > > > >> can
> >>> > > > > > > > >> > > as I
> >>> > > > > > > > >> > > > > > > > mentioned
> >>> > > > > > > > >> > > > > > > > > > in
> >>> > > > > > > > >> > > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > last email guarantee that
> any
> >>> > EndTxn
> >>> > > > > > > requests
> >>> > > > > > > > >> with
> >>> > > > > > > > >> > > the
> >>> > > > > > > > >> > > > > same
> >>> > > > > > > > >> > > > > > > > epoch
> >>> > > > > > > > >> > > > > > > > > > are
> >>> > > > > > > > >> > > > > > > > > > > > > from
> >>> > > > > > > > >> > > > > > > > > > > > > > the same producer and the
> same
> >>> > > > > > transaction.
> >>> > > > > > > > >> Then we
> >>> > > > > > > > >> > > > don't
> >>> > > > > > > > >> > > > > > > have
> >>> > > > > > > > >> > > > > > > > to
> >>> > > > > > > > >> > > > > > > > > > > > return
> >>> > > > > > > > >> > > > > > > > > > > > > a
> >>> > > > > > > > >> > > > > > > > > > > > > > fenced error but can handle
> >>> > > gracefully
> >>> > > > > as
> >>> > > > > > > > >> described
> >>> > > > > > > > >> > > in
> >>> > > > > > > > >> > > > > the
> >>> > > > > > > > >> > > > > > > KIP.
> >>> > > > > > > > >> > > > > > > > > > > > > > I don't think a boolean is
> >>> useful
> >>> > > > since
> >>> > > > > it
> >>> > > > > > > is
> >>> > > > > > > > >> > > directly
> >>> > > > > > > > >> > > > > > > encoded
> >>> > > > > > > > >> > > > > > > > by
> >>> > > > > > > > >> > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > existence or lack of the
> >>> tagged
> >>> > > field
> >>> > > > > > being
> >>> > > > > > > > >> > written.
> >>> > > > > > > > >> > > > > > > > > > > > > > In the prepare marker we
> will
> >>> have
> >>> > > the
> >>> > > > > > same
> >>> > > > > > > > >> > producer
> >>> > > > > > > > >> > > ID
> >>> > > > > > > > >> > > > > in
> >>> > > > > > > > >> > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> >>> > > > > > > > >> > > > > > > > > > > > > > field. In the Complete state
> >>> we
> >>> > may
> >>> > > > not.
> >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure why the ongoing
> >>> state
> >>> > > > > matters
> >>> > > > > > > for
> >>> > > > > > > > >> this
> >>> > > > > > > > >> > > > KIP.
> >>> > > > > > > > >> > > > > It
> >>> > > > > > > > >> > > > > > > > does
> >>> > > > > > > > >> > > > > > > > > > > matter
> >>> > > > > > > > >> > > > > > > > > > > > > for
> >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> >>> > > > > > > > >> > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure what you are
> >>> > referring
> >>> > > to
> >>> > > > > > about
> >>> > > > > > > > >> > writing
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > > > > previous
> >>> > > > > > > > >> > > > > > > > > > > > > producer
> >>> > > > > > > > >> > > > > > > > > > > > > > ID in the prepare marker.
> >>> This is
> >>> > > not
> >>> > > > in
> >>> > > > > > the
> >>> > > > > > > > >> KIP.
> >>> > > > > > > > >> > > > > > > > > > > > > > In the overflow case, we
> >>> write the
> >>> > > > > > > > >> nextProducerId
> >>> > > > > > > > >> > in
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > > > > prepare
> >>> > > > > > > > >> > > > > > > > > > > state.
> >>> > > > > > > > >> > > > > > > > > > > > > > This is so we know what we
> >>> > assigned
> >>> > > > when
> >>> > > > > > we
> >>> > > > > > > > >> reload
> >>> > > > > > > > >> > > the
> >>> > > > > > > > >> > > > > > > > > transaction
> >>> > > > > > > > >> > > > > > > > > > > log.
> >>> > > > > > > > >> > > > > > > > > > > > > > Once we complete, we
> >>> transition
> >>> > this
> >>> > > > ID
> >>> > > > > to
> >>> > > > > > > the
> >>> > > > > > > > >> main
> >>> > > > > > > > >> > > > > > > (non-tagged
> >>> > > > > > > > >> > > > > > > > > > > field)
> >>> > > > > > > > >> > > > > > > > > > > > > and
> >>> > > > > > > > >> > > > > > > > > > > > > > have the previous producer
> ID
> >>> > field
> >>> > > > > filled
> >>> > > > > > > in.
> >>> > > > > > > > >> This
> >>> > > > > > > > >> > > is
> >>> > > > > > > > >> > > > so
> >>> > > > > > > > >> > > > > > we
> >>> > > > > > > > >> > > > > > > > can
> >>> > > > > > > > >> > > > > > > > > > > > identify
> >>> > > > > > > > >> > > > > > > > > > > > > > in a retry case the
> operation
> >>> > > > completed
> >>> > > > > > > > >> > successfully
> >>> > > > > > > > >> > > > and
> >>> > > > > > > > >> > > > > we
> >>> > > > > > > > >> > > > > > > > don't
> >>> > > > > > > > >> > > > > > > > > > > fence
> >>> > > > > > > > >> > > > > > > > > > > > > our
> >>> > > > > > > > >> > > > > > > > > > > > > > producer. The downgrade
> >>> > > compatibility
> >>> > > > I
> >>> > > > > > > > mention
> >>> > > > > > > > >> is
> >>> > > > > > > > >> > > that
> >>> > > > > > > > >> > > > > we
> >>> > > > > > > > >> > > > > > > keep
> >>> > > > > > > > >> > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > same
> >>> > > > > > > > >> > > > > > > > > > > > > > producer ID and epoch in the
> >>> main
> >>> > > > > > > (non-tagged)
> >>> > > > > > > > >> > fields
> >>> > > > > > > > >> > > > as
> >>> > > > > > > > >> > > > > we
> >>> > > > > > > > >> > > > > > > did
> >>> > > > > > > > >> > > > > > > > > > > before
> >>> > > > > > > > >> > > > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > code on the server side. If
> >>> the
> >>> > > server
> >>> > > > > > > > >> downgrades,
> >>> > > > > > > > >> > we
> >>> > > > > > > > >> > > > are
> >>> > > > > > > > >> > > > > > > still
> >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> >>> > > > > > > > >> > > > > > > > > > > > > > This addresses both the
> >>> prepare
> >>> > and
> >>> > > > > > complete
> >>> > > > > > > > >> state
> >>> > > > > > > > >> > > > > > > downgrades.
> >>> > > > > > > > >> > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> >>> > > > > > > > >> > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > On Fri, Jan 12, 2024 at
> >>> 10:21 AM
> >>> > Jun
> >>> > > > Rao
> >>> > > > > > > > >> > > > > > > > > <jun@confluent.io.invalid
> >>> > > > > > > > >> > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> >>> > > > > > > > >> > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > Hi, Justine,
> >>> > > > > > > > >> > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks for the reply.
> Sorry
> >>> for
> >>> > > the
> >>> > > > > > > delay. I
> >>> > > > > > > > >> > have a
> >>> > > > > > > > >> > > > few
> >>> > > > > > > > >> > > > > > > more
> >>> > > > > > > > >> > > > > > > > > > > > comments.
> >>> > > > > > > > >> > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I think the
> motivation
> >>> > > section
> >>> > > > > > could
> >>> > > > > > > be
> >>> > > > > > > > >> > > > improved.
> >>> > > > > > > > >> > > > > > One
> >>> > > > > > > > >> > > > > > > of
> >>> > > > > > > > >> > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > > motivations listed by the
> >>> KIP is
> >>> > > > "This
> >>> > > > > > can
> >>> > > > > > > > >> happen
> >>> > > > > > > > >> > > > when
> >>> > > > > > > > >> > > > > a
> >>> > > > > > > > >> > > > > > > > > message
> >>> > > > > > > > >> > > > > > > > > > > gets
> >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> >>> > > > > > > > >> > > > > > > > > > > > > > > or delayed due to
> networking
> >>> > > issues
> >>> > > > > or a
> >>> > > > > > > > >> network
> >>> > > > > > > > >> > > > > > partition,
> >>> > > > > > > > >> > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > transaction
> >>> > > > > > > > >> > > > > > > > > > > > > > > aborts, and then the
> delayed
> >>> > > message
> >>> > > > > > > finally
> >>> > > > > > > > >> > comes
> >>> > > > > > > > >> > > > > in.".
> >>> > > > > > > > >> > > > > > > This
> >>> > > > > > > > >> > > > > > > > > > seems
> >>> > > > > > > > >> > > > > > > > > > > > not
> >>> > > > > > > > >> > > > > > > > > > > > > > > very accurate. Without
> >>> KIP-890,
> >>> > > > > > currently,
> >>> > > > > > > > if
> >>> > > > > > > > >> the
> >>> > > > > > > > >> > > > > > > coordinator
> >>> > > > > > > > >> > > > > > > > > > times
> >>> > > > > > > > >> > > > > > > > > > > > out
> >>> > > > > > > > >> > > > > > > > > > > > > > and
> >>> > > > > > > > >> > > > > > > > > > > > > > > aborts an ongoing
> >>> transaction,
> >>> > it
> >>> > > > > > already
> >>> > > > > > > > >> bumps
> >>> > > > > > > > >> > up
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > > > epoch
> >>> > > > > > > > >> > > > > > > > in
> >>> > > > > > > > >> > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> >>> > > > > > > > >> > > > > > > > > > > > > > > which prevents the delayed
> >>> > produce
> >>> > > > > > message
> >>> > > > > > > > >> from
> >>> > > > > > > > >> > > being
> >>> > > > > > > > >> > > > > > added
> >>> > > > > > > > >> > > > > > > > to
> >>> > > > > > > > >> > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > user
> >>> > > > > > > > >> > > > > > > > > > > > > > > partition. What can cause
> a
> >>> > > hanging
> >>> > > > > > > > >> transaction
> >>> > > > > > > > >> > is
> >>> > > > > > > > >> > > > that
> >>> > > > > > > > >> > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > producer
> >>> > > > > > > > >> > > > > > > > > > > > > > > completes (either aborts
> or
> >>> > > > commits) a
> >>> > > > > > > > >> > transaction
> >>> > > > > > > > >> > > > > before
> >>> > > > > > > > >> > > > > > > > > > > receiving a
> >>> > > > > > > > >> > > > > > > > > > > > > > > successful ack on messages
> >>> > > published
> >>> > > > > in
> >>> > > > > > > the
> >>> > > > > > > > >> same
> >>> > > > > > > > >> > > txn.
> >>> > > > > > > > >> > > > > In
> >>> > > > > > > > >> > > > > > > this
> >>> > > > > > > > >> > > > > > > > > > case,
> >>> > > > > > > > >> > > > > > > > > > > > > it's
> >>> > > > > > > > >> > > > > > > > > > > > > > > possible for the delayed
> >>> message
> >>> > > to
> >>> > > > be
> >>> > > > > > > > >> appended
> >>> > > > > > > > >> > to
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > > > > > partition
> >>> > > > > > > > >> > > > > > > > > > > > after
> >>> > > > > > > > >> > > > > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > > marker, causing a
> >>> transaction to
> >>> > > > hang.
> >>> > > > > > > > >> > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > A similar issue (not
> >>> mentioned
> >>> > in
> >>> > > > the
> >>> > > > > > > > >> motivation)
> >>> > > > > > > > >> > > > could
> >>> > > > > > > > >> > > > > > > > happen
> >>> > > > > > > > >> > > > > > > > > on
> >>> > > > > > > > >> > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > > marker in the
> coordinator's
> >>> log.
> >>> > > For
> >>> > > > > > > > example,
> >>> > > > > > > > >> > it's
> >>> > > > > > > > >> > > > > > possible
> >>> > > > > > > > >> > > > > > > > for
> >>> > > > > > > > >> > > > > > > > > > an
> >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest to be
> delayed
> >>> on
> >>> > the
> >>> > > > > > > > >> coordinator.
> >>> > > > > > > > >> > By
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > > time
> >>> > > > > > > > >> > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > delayed
> >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest is
> processed,
> >>> it's
> >>> > > > > > possible
> >>> > > > > > > > that
> >>> > > > > > > > >> > the
> >>> > > > > > > > >> > > > > > previous
> >>> > > > > > > > >> > > > > > > > txn
> >>> > > > > > > > >> > > > > > > > > > has
> >>> > > > > > > > >> > > > > > > > > > > > > > already
> >>> > > > > > > > >> > > > > > > > > > > > > > > completed and a new txn
> has
> >>> > > started.
> >>> > > > > > > > >> Currently,
> >>> > > > > > > > >> > > since
> >>> > > > > > > > >> > > > > the
> >>> > > > > > > > >> > > > > > > > epoch
> >>> > > > > > > > >> > > > > > > > > > is
> >>> > > > > > > > >> > > > > > > > > > > > not
> >>> > > > > > > > >> > > > > > > > > > > > > > > bumped on every txn, the
> >>> delayed
> >>> > > > > > > > EndTxnRequest
> >>> > > > > > > > >> > will
> >>> > > > > > > > >> > > > add
> >>> > > > > > > > >> > > > > > an
> >>> > > > > > > > >> > > > > > > > > > > unexpected
> >>> > > > > > > > >> > > > > > > > > > > > > > > prepare marker (and
> >>> eventually a
> >>> > > > > > complete
> >>> > > > > > > > >> marker)
> >>> > > > > > > > >> > > to
> >>> > > > > > > > >> > > > > the
> >>> > > > > > > > >> > > > > > > > > ongoing
> >>> > > > > > > > >> > > > > > > > > > > txn.
> >>> > > > > > > > >> > > > > > > > > > > > > > This
> >>> > > > > > > > >> > > > > > > > > > > > > > > won't cause the
> transaction
> >>> to
> >>> > > hang,
> >>> > > > > but
> >>> > > > > > > it
> >>> > > > > > > > >> will
> >>> > > > > > > > >> > > > break
> >>> > > > > > > > >> > > > > > the
> >>> > > > > > > > >> > > > > > > > EoS
> >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> >>> > > > > > > > >> > > > > > > > > > > > > > > The proposal in this KIP
> >>> will
> >>> > > > address
> >>> > > > > > this
> >>> > > > > > > > >> issue
> >>> > > > > > > > >> > > too.
> >>> > > > > > > > >> > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > 101. "However, I was
> >>> writing it
> >>> > so
> >>> > > > > that
> >>> > > > > > we
> >>> > > > > > > > can
> >>> > > > > > > > >> > > > > > distinguish
> >>> > > > > > > > >> > > > > > > > > > between
> >>> > > > > > > > >> > > > > > > > > > > > > > > old clients where we don't
> >>> have
> >>> > > the
> >>> > > > > > > ability
> >>> > > > > > > > do
> >>> > > > > > > > >> > this
> >>> > > > > > > > >> > > > > > > operation
> >>> > > > > > > > >> > > > > > > > > and
> >>> > > > > > > > >> > > > > > > > > > > new
> >>> > > > > > > > >> > > > > > > > > > > > > > > clients that can. (Old
> >>> clients
> >>> > > don't
> >>> > > > > > bump
> >>> > > > > > > > the
> >>> > > > > > > > >> > epoch
> >>> > > > > > > > >> > > > on
> >>> > > > > > > > >> > > > > > > > commit,
> >>> > > > > > > > >> > > > > > > > > so
> >>> > > > > > > > >> > > > > > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > > > > > can't
> >>> > > > > > > > >> > > > > > > > > > > > > > > say for sure the write
> >>> belongs
> >>> > to
> >>> > > > the
> >>> > > > > > > given
> >>> > > > > > > > >> > > > > > transaction)."
> >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I am wondering why
> we
> >>> need
> >>> > > to
> >>> > > > > > > > >> distinguish
> >>> > > > > > > > >> > > > whether
> >>> > > > > > > > >> > > > > > the
> >>> > > > > > > > >> > > > > > > > > > marker
> >>> > > > > > > > >> > > > > > > > > > > is
> >>> > > > > > > > >> > > > > > > > > > > > > > > written by the old and the
> >>> new
> >>> > > > client.
> >>> > > > > > > Could
> >>> > > > > > > > >> you
> >>> > > > > > > > >> > > > > describe
> >>> > > > > > > > >> > > > > > > > what
> >>> > > > > > > > >> > > > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > > do
> >>> > > > > > > > >> > > > > > > > > > > > > > > differently if we know the
> >>> > marker
> >>> > > is
> >>> > > > > > > written
> >>> > > > > > > > >> by
> >>> > > > > > > > >> > the
> >>> > > > > > > > >> > > > new
> >>> > > > > > > > >> > > > > > > > client?
> >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If we do need a way
> to
> >>> > > > > distinguish
> >>> > > > > > > > >> whether
> >>> > > > > > > > >> > > the
> >>> > > > > > > > >> > > > > > marker
> >>> > > > > > > > >> > > > > > > > is
> >>> > > > > > > > >> > > > > > > > > > > > written
> >>> > > > > > > > >> > > > > > > > > > > > > by
> >>> > > > > > > > >> > > > > > > > > > > > > > > the old and the new
> client.
> >>> > Would
> >>> > > it
> >>> > > > > be
> >>> > > > > > > > >> simpler
> >>> > > > > > > > >> > to
> >>> > > > > > > > >> > > > just
> >>> > > > > > > > >> > > > > > > > > > introduce a
> >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> >>> > > > > > > > >> > > > > > > > > > > > > > > field instead of
> indirectly
> >>> > > through
> >>> > > > > the
> >>> > > > > > > > >> previous
> >>> > > > > > > > >> > > > > produce
> >>> > > > > > > > >> > > > > > ID
> >>> > > > > > > > >> > > > > > > > > > field?
> >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3 It's not clear to me
> >>> why
> >>> > we
> >>> > > > only
> >>> > > > > > add
> >>> > > > > > > > the
> >>> > > > > > > > >> > > > previous
> >>> > > > > > > > >> > > > > > > > produce
> >>> > > > > > > > >> > > > > > > > > > ID
> >>> > > > > > > > >> > > > > > > > > > > > > field
> >>> > > > > > > > >> > > > > > > > > > > > > > in
> >>> > > > > > > > >> > > > > > > > > > > > > > > the complete marker, but
> >>> not in
> >>> > > the
> >>> > > > > > > prepare
> >>> > > > > > > > >> > marker.
> >>> > > > > > > > >> > > > If
> >>> > > > > > > > >> > > > > we
> >>> > > > > > > > >> > > > > > > > want
> >>> > > > > > > > >> > > > > > > > > to
> >>> > > > > > > > >> > > > > > > > > > > > know
> >>> > > > > > > > >> > > > > > > > > > > > > > > whether a marker is
> written
> >>> by
> >>> > the
> >>> > > > new
> >>> > > > > > > > client
> >>> > > > > > > > >> or
> >>> > > > > > > > >> > > not,
> >>> > > > > > > > >> > > > > it
> >>> > > > > > > > >> > > > > > > > seems
> >>> > > > > > > > >> > > > > > > > > > that
> >>> > > > > > > > >> > > > > > > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > > > > > want
> >>> > > > > > > > >> > > > > > > > > > > > > > > to do this consistently
> for
> >>> all
> >>> > > > > markers.
> >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4 What about the
> >>> > > > > TransactionLogValue
> >>> > > > > > > > >> record
> >>> > > > > > > > >> > > > > > > representing
> >>> > > > > > > > >> > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> >>> > > > > > > > >> > > > > > > > > > > > > > > state? Should we also
> >>> > distinguish
> >>> > > > > > whether
> >>> > > > > > > > it's
> >>> > > > > > > > >> > > > written
> >>> > > > > > > > >> > > > > by
> >>> > > > > > > > >> > > > > > > the
> >>> > > > > > > > >> > > > > > > > > old
> >>> > > > > > > > >> > > > > > > > > > > or
> >>> > > > > > > > >> > > > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > > new client?
> >>> > > > > > > > >> > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In the overflow case,
> >>> it's
> >>> > > > still
> >>> > > > > > not
> >>> > > > > > > > >> clear
> >>> > > > > > > > >> > to
> >>> > > > > > > > >> > > me
> >>> > > > > > > > >> > > > > why
> >>> > > > > > > > >> > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > write
> >>> > > > > > > > >> > > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > > previous produce Id in the
> >>> > prepare
> >>> > > > > > marker
> >>> > > > > > > > >> while
> >>> > > > > > > > >> > > > writing
> >>> > > > > > > > >> > > > > > the
> >>> > > > > > > > >> > > > > > > > > next
> >>> > > > > > > > >> > > > > > > > > > > > > produce
> >>> > > > > > > > >> > > > > > > > > > > > > > Id
> >>> > > > > > > > >> > > > > > > > > > > > > > > in the complete marker.
> You
> >>> > > > mentioned
> >>> > > > > > that
> >>> > > > > > > > >> it's
> >>> > > > > > > > >> > for
> >>> > > > > > > > >> > > > > > > > > downgrading.
> >>> > > > > > > > >> > > > > > > > > > > > > However,
> >>> > > > > > > > >> > > > > > > > > > > > > > > we could downgrade with
> >>> either
> >>> > the
> >>> > > > > > prepare
> >>> > > > > > > > >> marker
> >>> > > > > > > > >> > > or
> >>> > > > > > > > >> > > > > the
> >>> > > > > > > > >> > > > > > > > > complete
> >>> > > > > > > > >> > > > > > > > > > > > > marker.
> >>> > > > > > > > >> > > > > > > > > > > > > > > In either case, the
> >>> downgraded
> >>> > > > > > coordinator
> >>> > > > > > > > >> should
> >>> > > > > > > > >> > > see
> >>> > > > > > > > >> > > > > the
> >>> > > > > > > > >> > > > > > > > same
> >>> > > > > > > > >> > > > > > > > > > > > produce
> >>> > > > > > > > >> > > > > > > > > > > > > id
> >>> > > > > > > > >> > > > > > > > > > > > > > > (probably the previous
> >>> produce
> >>> > > Id),
> >>> > > > > > right?
> >>> > > > > > > > >> > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> >>> > > > > > > > >> > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed, Dec 20, 2023 at
> >>> 6:00 PM
> >>> > > > > Justine
> >>> > > > > > > > Olshan
> >>> > > > > > > > >> > > > > > > > > > > > > > >
> >>> <jo...@confluent.io.invalid>
> >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> >>> > > > > > > > >> > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey Jun,
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks for taking a look
> >>> at
> >>> > the
> >>> > > > KIP
> >>> > > > > > > again.
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > 100. For the epoch
> >>> overflow
> >>> > > case,
> >>> > > > > only
> >>> > > > > > > the
> >>> > > > > > > > >> > marker
> >>> > > > > > > > >> > > > > will
> >>> > > > > > > > >> > > > > > > have
> >>> > > > > > > > >> > > > > > > > > max
> >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> >>> > > > > > > > >> > > > > > > > > > > > > > > This
> >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps the behavior of
> the
> >>> rest
> >>> > > of
> >>> > > > > the
> >>> > > > > > > > >> markers
> >>> > > > > > > > >> > > where
> >>> > > > > > > > >> > > > > the
> >>> > > > > > > > >> > > > > > > > last
> >>> > > > > > > > >> > > > > > > > > > > marker
> >>> > > > > > > > >> > > > > > > > > > > > > is
> >>> > > > > > > > >> > > > > > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch of the transaction
> >>> > > records +
> >>> > > > > 1.
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > 101. You are correct
> that
> >>> we
> >>> > > don't
> >>> > > > > > need
> >>> > > > > > > to
> >>> > > > > > > > >> > write
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > > > > producer
> >>> > > > > > > > >> > > > > > > > > > ID
> >>> > > > > > > > >> > > > > > > > > > > > > since
> >>> > > > > > > > >> > > > > > > > > > > > > > it
> >>> > > > > > > > >> > > > > > > > > > > > > > > > is the same. However, I
> >>> was
> >>> > > > writing
> >>> > > > > it
> >>> > > > > > > so
> >>> > > > > > > > >> that
> >>> > > > > > > > >> > we
> >>> > > > > > > > >> > > > can
> >>> > > > > > > > >> > > > > > > > > > distinguish
> >>> > > > > > > > >> > > > > > > > > > > > > > between
> >>> > > > > > > > >> > > > > > > > > > > > > > > > old clients where we
> don't
> >>> > have
> >>> > > > the
> >>> > > > > > > > ability
> >>> > > > > > > > >> do
> >>> > > > > > > > >> > > this
> >>> > > > > > > > >> > > > > > > > operation
> >>> > > > > > > > >> > > > > > > > > > and
> >>> > > > > > > > >> > > > > > > > > > > > new
> >>> > > > > > > > >> > > > > > > > > > > > > > > > clients that can. (Old
> >>> clients
> >>> > > > don't
> >>> > > > > > > bump
> >>> > > > > > > > >> the
> >>> > > > > > > > >> > > epoch
> >>> > > > > > > > >> > > > > on
> >>> > > > > > > > >> > > > > > > > > commit,
> >>> > > > > > > > >> > > > > > > > > > so
> >>> > > > > > > > >> > > > > > > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> >>> > > > > > > > >> > > > > > > > > > > > > > > > say for sure the write
> >>> belongs
> >>> > > to
> >>> > > > > the
> >>> > > > > > > > given
> >>> > > > > > > > >> > > > > > transaction).
> >>> > > > > > > > >> > > > > > > > If
> >>> > > > > > > > >> > > > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > > > > receive
> >>> > > > > > > > >> > > > > > > > > > > > > > > an
> >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn request from a
> new
> >>> > > client,
> >>> > > > we
> >>> > > > > > > will
> >>> > > > > > > > >> fill
> >>> > > > > > > > >> > > this
> >>> > > > > > > > >> > > > > > > field.
> >>> > > > > > > > >> > > > > > > > We
> >>> > > > > > > > >> > > > > > > > > > can
> >>> > > > > > > > >> > > > > > > > > > > > > > > guarantee
> >>> > > > > > > > >> > > > > > > > > > > > > > > > that any EndTxn requests
> >>> with
> >>> > > the
> >>> > > > > same
> >>> > > > > > > > epoch
> >>> > > > > > > > >> > are
> >>> > > > > > > > >> > > > from
> >>> > > > > > > > >> > > > > > the
> >>> > > > > > > > >> > > > > > > > > same
> >>> > > > > > > > >> > > > > > > > > > > > > producer
> >>> > > > > > > > >> > > > > > > > > > > > > > > and
> >>> > > > > > > > >> > > > > > > > > > > > > > > > the same transaction.
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > 102. In prepare phase,
> we
> >>> have
> >>> > > the
> >>> > > > > > same
> >>> > > > > > > > >> > producer
> >>> > > > > > > > >> > > ID
> >>> > > > > > > > >> > > > > and
> >>> > > > > > > > >> > > > > > > > epoch
> >>> > > > > > > > >> > > > > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > > > > always
> >>> > > > > > > > >> > > > > > > > > > > > > > > > had. It is the producer
> >>> ID and
> >>> > > > epoch
> >>> > > > > > > that
> >>> > > > > > > > >> are
> >>> > > > > > > > >> > on
> >>> > > > > > > > >> > > > the
> >>> > > > > > > > >> > > > > > > > marker.
> >>> > > > > > > > >> > > > > > > > > In
> >>> > > > > > > > >> > > > > > > > > > > > > commit
> >>> > > > > > > > >> > > > > > > > > > > > > > > > phase, we stay the same
> >>> unless
> >>> > > it
> >>> > > > is
> >>> > > > > > the
> >>> > > > > > > > >> > overflow
> >>> > > > > > > > >> > > > > case.
> >>> > > > > > > > >> > > > > > > In
> >>> > > > > > > > >> > > > > > > > > that
> >>> > > > > > > > >> > > > > > > > > > > > case,
> >>> > > > > > > > >> > > > > > > > > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > > > > > > > set the producer ID to
> >>> the new
> >>> > > one
> >>> > > > > we
> >>> > > > > > > > >> generated
> >>> > > > > > > > >> > > and
> >>> > > > > > > > >> > > > > > epoch
> >>> > > > > > > > >> > > > > > > > to
> >>> > > > > > > > >> > > > > > > > > 0
> >>> > > > > > > > >> > > > > > > > > > > > after
> >>> > > > > > > > >> > > > > > > > > > > > > > > > complete. This is for
> >>> > downgrade
> >>> > > > > > > > >> compatibility.
> >>> > > > > > > > >> > > The
> >>> > > > > > > > >> > > > > > tagged
> >>> > > > > > > > >> > > > > > > > > > fields
> >>> > > > > > > > >> > > > > > > > > > > > are
> >>> > > > > > > > >> > > > > > > > > > > > > > just
> >>> > > > > > > > >> > > > > > > > > > > > > > > > safety guards for
> retries
> >>> and
> >>> > > > > > failovers.
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > In prepare phase for
> epoch
> >>> > > > overflow
> >>> > > > > > case
> >>> > > > > > > > >> only
> >>> > > > > > > > >> > we
> >>> > > > > > > > >> > > > > store
> >>> > > > > > > > >> > > > > > > the
> >>> > > > > > > > >> > > > > > > > > next
> >>> > > > > > > > >> > > > > > > > > > > > > > producer
> >>> > > > > > > > >> > > > > > > > > > > > > > > > ID. This is for the case
> >>> where
> >>> > > we
> >>> > > > > > reload
> >>> > > > > > > > the
> >>> > > > > > > > >> > > > > > transaction
> >>> > > > > > > > >> > > > > > > > > > > > coordinator
> >>> > > > > > > > >> > > > > > > > > > > > > in
> >>> > > > > > > > >> > > > > > > > > > > > > > > > prepare state. Once the
> >>> > > > transaction
> >>> > > > > is
> >>> > > > > > > > >> > committed,
> >>> > > > > > > > >> > > > we
> >>> > > > > > > > >> > > > > > can
> >>> > > > > > > > >> > > > > > > > use
> >>> > > > > > > > >> > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > producer
> >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the client already is
> >>> > using.
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > In commit phase, we
> store
> >>> the
> >>> > > > > previous
> >>> > > > > > > > >> producer
> >>> > > > > > > > >> > > ID
> >>> > > > > > > > >> > > > in
> >>> > > > > > > > >> > > > > > > case
> >>> > > > > > > > >> > > > > > > > of
> >>> > > > > > > > >> > > > > > > > > > > > > retries.
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > I think it is easier to
> >>> think
> >>> > of
> >>> > > > it
> >>> > > > > as
> >>> > > > > > > > just
> >>> > > > > > > > >> how
> >>> > > > > > > > >> > > we
> >>> > > > > > > > >> > > > > were
> >>> > > > > > > > >> > > > > > > > > storing
> >>> > > > > > > > >> > > > > > > > > > > > > > producer
> >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> >>> > > > > > > > >> > > > > > > > > > > > > > > > and epoch before, with
> >>> some
> >>> > > extra
> >>> > > > > > > > bookeeping
> >>> > > > > > > > >> > and
> >>> > > > > > > > >> > > > edge
> >>> > > > > > > > >> > > > > > > case
> >>> > > > > > > > >> > > > > > > > > > > handling
> >>> > > > > > > > >> > > > > > > > > > > > > in
> >>> > > > > > > > >> > > > > > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged fields. We have
> to
> >>> do
> >>> > it
> >>> > > > this
> >>> > > > > > way
> >>> > > > > > > > for
> >>> > > > > > > > >> > > > > > > compatibility
> >>> > > > > > > > >> > > > > > > > > with
> >>> > > > > > > > >> > > > > > > > > > > > > > > downgrades.
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > 103. Next producer ID is
> >>> for
> >>> > > > prepare
> >>> > > > > > > > status
> >>> > > > > > > > >> and
> >>> > > > > > > > >> > > > > > previous
> >>> > > > > > > > >> > > > > > > > > > producer
> >>> > > > > > > > >> > > > > > > > > > > > ID
> >>> > > > > > > > >> > > > > > > > > > > > > is
> >>> > > > > > > > >> > > > > > > > > > > > > > > for
> >>> > > > > > > > >> > > > > > > > > > > > > > > > after complete. The
> >>> reason why
> >>> > > we
> >>> > > > > need
> >>> > > > > > > two
> >>> > > > > > > > >> > > separate
> >>> > > > > > > > >> > > > > > > > (tagged)
> >>> > > > > > > > >> > > > > > > > > > > fields
> >>> > > > > > > > >> > > > > > > > > > > > > is
> >>> > > > > > > > >> > > > > > > > > > > > > > > for
> >>> > > > > > > > >> > > > > > > > > > > > > > > > backwards compatibility.
> >>> We
> >>> > need
> >>> > > > to
> >>> > > > > > keep
> >>> > > > > > > > the
> >>> > > > > > > > >> > same
> >>> > > > > > > > >> > > > > > > semantics
> >>> > > > > > > > >> > > > > > > > > for
> >>> > > > > > > > >> > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged field in case
> >>> we
> >>> > > > > downgrade.
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > 104. We set the fields
> as
> >>> we
> >>> > do
> >>> > > in
> >>> > > > > the
> >>> > > > > > > > >> > > > transactional
> >>> > > > > > > > >> > > > > > > state
> >>> > > > > > > > >> > > > > > > > > (as
> >>> > > > > > > > >> > > > > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > > > > need
> >>> > > > > > > > >> > > > > > > > > > > > > > to
> >>> > > > > > > > >> > > > > > > > > > > > > > > > do this for
> compatibility
> >>> --
> >>> > if
> >>> > > we
> >>> > > > > > > > >> downgrade,
> >>> > > > > > > > >> > we
> >>> > > > > > > > >> > > > will
> >>> > > > > > > > >> > > > > > > only
> >>> > > > > > > > >> > > > > > > > > have
> >>> > > > > > > > >> > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged fields) It
> >>> will be
> >>> > > the
> >>> > > > > old
> >>> > > > > > > > >> producer
> >>> > > > > > > > >> > ID
> >>> > > > > > > > >> > > > and
> >>> > > > > > > > >> > > > > > max
> >>> > > > > > > > >> > > > > > > > > > epoch.
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope this helps. Let me
> >>> know
> >>> > if
> >>> > > > you
> >>> > > > > > have
> >>> > > > > > > > >> > further
> >>> > > > > > > > >> > > > > > > questions.
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > Justine
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > On Wed, Dec 20, 2023 at
> >>> > 3:33 PM
> >>> > > > Jun
> >>> > > > > > Rao
> >>> > > > > > > > >> > > > > > > > > > <jun@confluent.io.invalid
> >>> > > > > > > > >> > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi, Justine,
> >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > It seems that you have
> >>> made
> >>> > > some
> >>> > > > > > > changes
> >>> > > > > > > > >> to
> >>> > > > > > > > >> > > > KIP-890
> >>> > > > > > > > >> > > > > > > since
> >>> > > > > > > > >> > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > vote.
> >>> > > > > > > > >> > > > > > > > > > > > > > In
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > particular, we are
> >>> changing
> >>> > > the
> >>> > > > > > format
> >>> > > > > > > > of
> >>> > > > > > > > >> > > > > > > > > > TransactionLogValue.
> >>> > > > > > > > >> > > > > > > > > > > A
> >>> > > > > > > > >> > > > > > > > > > > > > few
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > comments related to
> >>> that.
> >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100. Just to be clear.
> >>> The
> >>> > > > > overflow
> >>> > > > > > > case
> >>> > > > > > > > >> > (i.e.
> >>> > > > > > > > >> > > > > when a
> >>> > > > > > > > >> > > > > > > new
> >>> > > > > > > > >> > > > > > > > > > > > > producerId
> >>> > > > > > > > >> > > > > > > > > > > > > > is
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > generated) is when the
> >>> > current
> >>> > > > > epoch
> >>> > > > > > > > >> equals
> >>> > > > > > > > >> > to
> >>> > > > > > > > >> > > > max
> >>> > > > > > > > >> > > > > -
> >>> > > > > > > > >> > > > > > 1
> >>> > > > > > > > >> > > > > > > > and
> >>> > > > > > > > >> > > > > > > > > > not
> >>> > > > > > > > >> > > > > > > > > > > > max?
> >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101. For the "not
> epoch
> >>> > > > overflow"
> >>> > > > > > > case,
> >>> > > > > > > > we
> >>> > > > > > > > >> > > write
> >>> > > > > > > > >> > > > > the
> >>> > > > > > > > >> > > > > > > > > previous
> >>> > > > > > > > >> > > > > > > > > > > ID
> >>> > > > > > > > >> > > > > > > > > > > > in
> >>> > > > > > > > >> > > > > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > tagged field in the
> >>> complete
> >>> > > > > phase.
> >>> > > > > > Do
> >>> > > > > > > > we
> >>> > > > > > > > >> > need
> >>> > > > > > > > >> > > to
> >>> > > > > > > > >> > > > > do
> >>> > > > > > > > >> > > > > > > that
> >>> > > > > > > > >> > > > > > > > > > since
> >>> > > > > > > > >> > > > > > > > > > > > > > produce
> >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > doesn't change in this
> >>> case?
> >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102. It seems that the
> >>> > meaning
> >>> > > > for
> >>> > > > > > the
> >>> > > > > > > > >> > > > > > > > > > ProducerId/ProducerEpoch
> >>> > > > > > > > >> > > > > > > > > > > > > > fields
> >>> > > > > > > > >> > > > > > > > > > > > > > > in
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > TransactionLogValue
> >>> changes
> >>> > > > > > depending
> >>> > > > > > > on
> >>> > > > > > > > >> the
> >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> >>> > > > > > > > >> > > > > > > > > > > > > When
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > the TransactionStatus
> is
> >>> > > > ongoing,
> >>> > > > > > they
> >>> > > > > > > > >> > > represent
> >>> > > > > > > > >> > > > > the
> >>> > > > > > > > >> > > > > > > > > current
> >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > the current
> >>> ProducerEpoch.
> >>> > > When
> >>> > > > > the
> >>> > > > > > > > >> > > > > TransactionStatus
> >>> > > > > > > > >> > > > > > > is
> >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> >>> PrepareCommit/PrepareAbort,
> >>> > > they
> >>> > > > > > > > represent
> >>> > > > > > > > >> > the
> >>> > > > > > > > >> > > > > > current
> >>> > > > > > > > >> > > > > > > > > > > ProducerId
> >>> > > > > > > > >> > > > > > > > > > > > > and
> >>> > > > > > > > >> > > > > > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > next ProducerEpoch.
> >>> When the
> >>> > > > > > > > >> > TransactionStatus
> >>> > > > > > > > >> > > is
> >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> >>> > > > > > > > >> > > > > > > > > > > > > they
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > further depend on
> >>> whether
> >>> > the
> >>> > > > > epoch
> >>> > > > > > > > >> overflows
> >>> > > > > > > > >> > > or
> >>> > > > > > > > >> > > > > not.
> >>> > > > > > > > >> > > > > > > If
> >>> > > > > > > > >> > > > > > > > > > there
> >>> > > > > > > > >> > > > > > > > > > > is
> >>> > > > > > > > >> > > > > > > > > > > > > no
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > overflow, they
> represent
> >>> > the
> >>> > > > > > current
> >>> > > > > > > > >> > > ProducerId
> >>> > > > > > > > >> > > > > and
> >>> > > > > > > > >> > > > > > > the
> >>> > > > > > > > >> > > > > > > > > next
> >>> > > > > > > > >> > > > > > > > > > > > > > > > ProducerEpoch
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > (max). Otherwise, they
> >>> > > represent
> >>> > > > > the
> >>> > > > > > > > newly
> >>> > > > > > > > >> > > > > generated
> >>> > > > > > > > >> > > > > > > > > > ProducerId
> >>> > > > > > > > >> > > > > > > > > > > > > and a
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > ProducerEpoch of 0. Is
> >>> that
> >>> > > > right?
> >>> > > > > > > This
> >>> > > > > > > > >> seems
> >>> > > > > > > > >> > > not
> >>> > > > > > > > >> > > > > > easy
> >>> > > > > > > > >> > > > > > > to
> >>> > > > > > > > >> > > > > > > > > > > > > understand.
> >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > we provide some
> examples
> >>> > like
> >>> > > > what
> >>> > > > > > > Artem
> >>> > > > > > > > >> has
> >>> > > > > > > > >> > > done
> >>> > > > > > > > >> > > > > in
> >>> > > > > > > > >> > > > > > > > > KIP-939?
> >>> > > > > > > > >> > > > > > > > > > > > Have
> >>> > > > > > > > >> > > > > > > > > > > > > we
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > considered a simpler
> >>> design
> >>> > > > where
> >>> > > > > > > > >> > > > > > > > ProducerId/ProducerEpoch
> >>> > > > > > > > >> > > > > > > > > > > always
> >>> > > > > > > > >> > > > > > > > > > > > > > > > represent
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > the same value (e.g.
> >>> for the
> >>> > > > > current
> >>> > > > > > > > >> > > transaction)
> >>> > > > > > > > >> > > > > > > > > independent
> >>> > > > > > > > >> > > > > > > > > > > of
> >>> > > > > > > > >> > > > > > > > > > > > > the
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > TransactionStatus and
> >>> epoch
> >>> > > > > > overflow?
> >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103. It's not clear to
> >>> me
> >>> > why
> >>> > > we
> >>> > > > > > need
> >>> > > > > > > 3
> >>> > > > > > > > >> > fields:
> >>> > > > > > > > >> > > > > > > > ProducerId,
> >>> > > > > > > > >> > > > > > > > > > > > > > > > PrevProducerId,
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > NextProducerId. Could
> we
> >>> > just
> >>> > > > have
> >>> > > > > > > > >> ProducerId
> >>> > > > > > > > >> > > and
> >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104. For
> >>> > > WriteTxnMarkerRequests,
> >>> > > > > if
> >>> > > > > > > the
> >>> > > > > > > > >> > > producer
> >>> > > > > > > > >> > > > > > epoch
> >>> > > > > > > > >> > > > > > > > > > > overflows,
> >>> > > > > > > > >> > > > > > > > > > > > > > what
> >>> > > > > > > > >> > > > > > > > > > > > > > > do
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > we set the producerId
> >>> and
> >>> > the
> >>> > > > > > > > >> producerEpoch?
> >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > Thanks,
> >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > > >
> >>> > > > > > > > >> > > > > > > > >
> >>> > > > > > > > >> > > > > > > >
> >>> > > > > > > > >> > > > > > >
> >>> > > > > > > > >> > > > > >
> >>> > > > > > > > >> > > > >
> >>> > > > > > > > >> > > >
> >>> > > > > > > > >> > >
> >>> > > > > > > > >> >
> >>> > > > > > > > >>
> >>> > > > > > > > >
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Justine Olshan <jo...@confluent.io.INVALID>]

Thanks Calvin,

The changes make sense to me. Can we add the defaults to the KIP page as
well? Thanks!

Justine

On Tue, Feb 11, 2025 at 3:27 PM Calvin Liu <ca...@confluent.io.invalid>
wrote:

> Hi Jun,
> Thanks for the feedback, will update the name as suggested.
> The default values are
> add.partitions.to.txn.retry.backoff.ms=20
> -- This is because the client side AddPartitionsToTxn retry backoff is
> 20ms.
> add.partitions.to.txn.retry.backoff.max.ms=100
> -- We want this value to be small to avoid client side timeout and retry,
> but also long enough for the markers to propagate.
>
>
> On Tue, Feb 11, 2025 at 2:36 PM Jun Rao <ju...@confluent.io.invalid> wrote:
>
> > Hi, Calvin,
> >
> > Thanks for the update. The change looks good to me. Some minor comments.
> >
> > For retry timeout, our convention is to use retry.backoff.max.ms and
> > retry.backoff.ms. So, to be consistent, the new configurations should be
> > add.partitions.to.txn.retry.backoff.max.ms and
> > add.partitions.to.txn.retry.backoff.ms.
> >
> > Also, could you specify the defaults for these two values?
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> >
> > On Mon, Feb 10, 2025 at 2:45 PM Calvin Liu <ca...@confluent.io.invalid>
> > wrote:
> >
> > > Hey team,
> > > In order to address the performance issue in the
> > > https://issues.apache.org/jira/browse/KAFKA-18654. We are making a
> > change
> > > to retry the AddPartitionToTxn requests on the server side when hitting
> > the
> > > CONCURRENT_TRANSACTION error. This error is common after the
> transaction
> > is
> > > committed but before the markers are fully propagated to all the
> involved
> > > partitions. With the server side retry, the client can avoid the
> frequent
> > > and expensive retry for the produce request. In order to better control
> > the
> > > retry behavior, we proposed to add 2 following configs.
> > >
> > >    - *add.partitions.to.txn.max.timeout.ms
> > >    <http://add.partitions.to.txn.max.timeout.ms>* defines the maximum
> > > retry
> > >    timeout when the server attempts to add the partition to the
> > > transaction.
> > >    - *add.partitions.to.txn.retry.backoff.ms
> > >    <http://add.partitions.to.txn.retry.backoff.ms>* defines how
> > frequently
> > >    the server will retry the AddPartitionToTxn.
> > >
> > > Thanks!
> > >
> > > On Thu, Nov 14, 2024 at 1:52 PM Justine Olshan
> > > <jo...@confluent.io.invalid>
> > > wrote:
> > >
> > > > Hey folks, just wanted to share another quick update with KIP-890.
> > > >
> > > > Making the change to bump epoch after every transaction means that we
> > no
> > > > longer need to call InitProducerId during the life of the producer to
> > > fence
> > > > requests from previous transactions. With KIP-890 part 2, the new
> > client
> > > > will no longer call InitProducerId except on startup to fence a
> > previous
> > > > instance.
> > > >
> > > > With this change, some other calls to InitProducerId were inspected
> > > > including the call after receiving an InvalidPidMappingException.
> This
> > > > exception was changed to abortable as part of KIP-360: Improve
> > > reliability
> > > > of idempotent/transactional producer
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820
> > > > >.
> > > > However, this change means that we can violate EOS guarantees. As an
> > > > example:
> > > >
> > > > Consider an application that is copying data from one partition to
> > > another
> > > >
> > > >    - Application instance A processes to offset 4
> > > >    - Application instance B comes up and fences application instance
> A
> > > >    - Application instance B processes to offset 5
> > > >    - Application instances A and B are idle for
> > > > transaction.id.expiration.ms,
> > > >    transaction id expires on server
> > > >    - Application instance A attempts to process offset 5 (since in
> its
> > > >    view, that is next) -- if we recover from invalid pid mapping, we
> > can
> > > >    duplicate this processing
> > > >
> > > > Thus, INVALID_PID_MAPPING should be fatal to the producer and as part
> > of
> > > > KIP-890 the new client in 4.0 will treat the exception as fatal.
> > > >
> > > > This is consistent with KIP-1050: Consistent error handling for
> > > > Transactions
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions
> > > > >
> > > > where
> > > > errors that are fatal to the producer are in the "application
> > > recoverable"
> > > > category. This is a grouping that indicates to the client that the
> > > producer
> > > > needs to restart and recovery on the application side is necessary.
> > > > KIP-1050 is approved so we are consistent with that decision.
> > > >
> > > > Let me know if there are any questions. I've updated the KIP to
> reflect
> > > > this change.
> > > >
> > > > Justine
> > > >
> > > > On Wed, Jul 10, 2024 at 3:34 PM Jun Rao <ju...@confluent.io.invalid>
> > > wrote:
> > > >
> > > > > Sounds good, Justine. It would be useful to document that in the
> KIP.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Jul 10, 2024 at 2:59 PM Justine Olshan
> > > > > <jo...@confluent.io.invalid>
> > > > > wrote:
> > > > >
> > > > > > The client will send the newest EndTxn request version if and
> only
> > if
> > > > > both
> > > > > > the client and the server support kip-890 part 2.
> > > > > > We set the value in the record based on the EndTxn version.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <jun@confluent.io.invalid
> >
> > > > > wrote:
> > > > > >
> > > > > > > Hi, Justine,
> > > > > > >
> > > > > > > Thanks for the reply.
> > > > > > >
> > > > > > > 120. If the broker sends TV Y for the finalized version in
> > > > > > > ApiVersionResponse, but the client doesn't support Y, how does
> > the
> > > > > broker
> > > > > > > know the TV that the client supports?
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Jun,
> > > > > > > >
> > > > > > > > No worries. Work on this KIP has been blocked for a bit
> anyways
> > > --
> > > > > > > catching
> > > > > > > > up and rereading what I wrote :)
> > > > > > > >
> > > > > > > > 120. ClientTransactionProtocolVersion is the transaction
> > version
> > > as
> > > > > > > defined
> > > > > > > > by the highest transaction version (feature version value)
> > > > supported
> > > > > by
> > > > > > > the
> > > > > > > > client and the server. This works by the broker sending an
> > > > > > > > ApiVersionsRequest to the client with the finalized version.
> > > > Assuming
> > > > > > > > kip-890 part 2 is enabled by transaction version Y, if this
> > > request
> > > > > > > > contains finalized version Y and the client has the logic to
> > set
> > > > this
> > > > > > > > field, it will set Y. If the server has Y - 1 (kip 890 part 2
> > not
> > > > > > enable)
> > > > > > > > the client will send Y - 1, even though the client has the
> > > ability
> > > > to
> > > > > > > > support kip-890 part 2.
> > > > > > > >
> > > > > > > > 121. You are correct that this is not needed. However,
> > currently
> > > > that
> > > > > > > field
> > > > > > > > is already being set in memory -- just not written to disk. I
> > > think
> > > > > it
> > > > > > is
> > > > > > > > ok to write it to disk though. Let me know if you think
> > > otherwise.
> > > > > > > >
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao
> > <jun@confluent.io.invalid
> > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Justine,
> > > > > > > > >
> > > > > > > > > Thanks for the update and sorry for the late reply.
> > > > > > > > >
> > > > > > > > > 120. I am wondering what value is used for
> > > > > > > > > ClientTransactionProtocolVersion. Is it the version of the
> > > > > > > EndTxnRequest?
> > > > > > > > >
> > > > > > > > > 121. Earlier, you made the change to set lastProducerId in
> > > > PREPARE
> > > > > to
> > > > > > > > > indicate that the marker is written for the new client.
> With
> > > the
> > > > > new
> > > > > > > > > ClientTransactionProtocolVersion field, it seems this is no
> > > > longer
> > > > > > > > > necessary.
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi there -- another update!
> > > > > > > > > >
> > > > > > > > > > When looking into the implementation for the safe epoch
> > > bumps I
> > > > > > > > realized
> > > > > > > > > > that we are already populating previousProducerID in
> memory
> > > as
> > > > > part
> > > > > > > of
> > > > > > > > > > KIP-360.
> > > > > > > > > > If we are to start using flexible fields, it is better to
> > > > always
> > > > > > use
> > > > > > > > this
> > > > > > > > > > information and have an explicit (tagged) field to
> indicate
> > > > > whether
> > > > > > > the
> > > > > > > > > > client supports KIP-890 part 2.
> > > > > > > > > >
> > > > > > > > > > I've included the extra field and how it is set in the
> KIP.
> > > > I've
> > > > > > also
> > > > > > > > > > updated the KIP to explain that we will be setting the
> > tagged
> > > > > > fields
> > > > > > > > when
> > > > > > > > > > they are available for all transitions.
> > > > > > > > > >
> > > > > > > > > > Finally, I added clearer text about the transaction
> > protocol
> > > > > > versions
> > > > > > > > > > included with this KIP. 1 for flexible transaction state
> > > > records
> > > > > > and
> > > > > > > 2
> > > > > > > > > for
> > > > > > > > > > KIP-890 part 2 enablement.
> > > > > > > > > >
> > > > > > > > > > Justine
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <
> > > > > > jolshan@confluent.io
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey there -- small update to the KIP,
> > > > > > > > > > >
> > > > > > > > > > > The KIP mentioned introducing ABORTABLE_ERROR and
> bumping
> > > > > > > > > TxnOffsetCommit
> > > > > > > > > > > and Produce requests. I've changed the name in the KIP
> to
> > > > > > > > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > > > > > > > AbortableTransactionException to match the pattern we
> had
> > > for
> > > > > > other
> > > > > > > > > > errors.
> > > > > > > > > > > I also mentioned bumping all 6 transactional APIs so we
> > can
> > > > > > future
> > > > > > > > > > > proof/support the error on the client going forward.
> If a
> > > > > future
> > > > > > > > change
> > > > > > > > > > > wants to have an error scenario that requires us to
> abort
> > > the
> > > > > > > > > > transaction,
> > > > > > > > > > > we can rely on the 3.8+ clients to support it. We ran
> > into
> > > > > issues
> > > > > > > > > finding
> > > > > > > > > > > good/generic error codes that older clients could
> support
> > > > while
> > > > > > > > working
> > > > > > > > > > on
> > > > > > > > > > > this KIP, so this should help in the future.
> > > > > > > > > > >
> > > > > > > > > > > The features discussion is still ongoing in KIP-1022.
> > Will
> > > > > update
> > > > > > > > again
> > > > > > > > > > > here when that concludes.
> > > > > > > > > > >
> > > > > > > > > > > Justine
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> > > > > > > jolshan@confluent.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> I don't think AddPartitions is a good example since we
> > > > > currenly
> > > > > > > > don't
> > > > > > > > > > >> gate the version on TV or MV. (We only set a different
> > > flag
> > > > > > > > depending
> > > > > > > > > on
> > > > > > > > > > >> the TV)
> > > > > > > > > > >>
> > > > > > > > > > >> Even if we did want to gate it on TV, I think the idea
> > is
> > > to
> > > > > > move
> > > > > > > > away
> > > > > > > > > > >> from MV gating inter broker protocols. Ideally we can
> > get
> > > > to a
> > > > > > > state
> > > > > > > > > > where
> > > > > > > > > > >> MV is just used for metadata changes.
> > > > > > > > > > >>
> > > > > > > > > > >> I think some of this discussion might fit more with
> the
> > > > > feature
> > > > > > > > > version
> > > > > > > > > > >> KIP, so I can try to open that up soon. Until we
> settle
> > > > that,
> > > > > > some
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >> work in KIP-890 is blocked.
> > > > > > > > > > >>
> > > > > > > > > > >> Justine
> > > > > > > > > > >>
> > > > > > > > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao
> > > > > <jun@confluent.io.invalid
> > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> Hi, Justine,
> > > > > > > > > > >>>
> > > > > > > > > > >>> Thanks for the reply.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Since AddPartitions is an inter broker request, will
> > its
> > > > > > version
> > > > > > > be
> > > > > > > > > > gated
> > > > > > > > > > >>> only by TV or other features like MV too? For
> example,
> > if
> > > > we
> > > > > > need
> > > > > > > > to
> > > > > > > > > > >>> change
> > > > > > > > > > >>> the protocol for AddPartitions for reasons other than
> > txn
> > > > > > > > > verification
> > > > > > > > > > in
> > > > > > > > > > >>> the future, will the new version be gated by a new
> MV?
> > If
> > > > so,
> > > > > > > does
> > > > > > > > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > > > > > > > >>>
> > > > > > > > > > >>> Jun
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>> > One TV gates the flexible feature version (no rpcs
> > > > > involved,
> > > > > > > only
> > > > > > > > > the
> > > > > > > > > > >>> > transactional records that should only be gated by
> > TV)
> > > > > > > > > > >>> > Another TV gates the ability to turn on kip-890
> part
> > 2.
> > > > > This
> > > > > > > > would
> > > > > > > > > > >>> gate the
> > > > > > > > > > >>> > version of Produce and EndTxn (likely only used by
> > > > > > > transactions),
> > > > > > > > > and
> > > > > > > > > > >>> > specifies a flag in AddPartitionsToTxn though the
> > > version
> > > > > is
> > > > > > > > > already
> > > > > > > > > > >>> used
> > > > > > > > > > >>> > without TV.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > I think the only concern is the Produce request and
> > we
> > > > > could
> > > > > > > > > consider
> > > > > > > > > > >>> work
> > > > > > > > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Justine
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> > > > > > > <jun@confluent.io.invalid
> > > > > > > > >
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > > Hi, Justine,
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Which PRC/record protocols will TV guard? Going
> > > > forward,
> > > > > > will
> > > > > > > > > those
> > > > > > > > > > >>> > > PRC/record protocols only be guarded by TV and
> not
> > by
> > > > > other
> > > > > > > > > > features
> > > > > > > > > > >>> like
> > > > > > > > > > >>> > > MV?
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Jun
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > > > > > > > >>> > <jolshan@confluent.io.invalid
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > wrote:
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > > Hi Jun,
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > Sorry I think I misunderstood your question or
> > > > answered
> > > > > > > > > > >>> incorrectly.
> > > > > > > > > > >>> > The
> > > > > > > > > > >>> > > TV
> > > > > > > > > > >>> > > > version should ideally be fully independent
> from
> > > MV.
> > > > > > > > > > >>> > > > At least for the changes I proposed, TV should
> > not
> > > > > affect
> > > > > > > MV
> > > > > > > > > and
> > > > > > > > > > MV
> > > > > > > > > > >>> > > should
> > > > > > > > > > >>> > > > not affect TV/
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > I think if we downgrade TV, only that feature
> > > should
> > > > > > > > downgrade.
> > > > > > > > > > >>> > Likewise
> > > > > > > > > > >>> > > > the same with MV. The finalizedFeatures should
> > just
> > > > > > reflect
> > > > > > > > the
> > > > > > > > > > >>> feature
> > > > > > > > > > >>> > > > downgrade we made.
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > I also plan to write a new KIP for managing the
> > > disk
> > > > > > format
> > > > > > > > and
> > > > > > > > > > >>> upgrade
> > > > > > > > > > >>> > > > tool as we will need new flags to support these
> > > > > features.
> > > > > > > > That
> > > > > > > > > > >>> should
> > > > > > > > > > >>> > > help
> > > > > > > > > > >>> > > > clarify some things.
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > Justine
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > > > > > > > <jun@confluent.io.invalid
> > > > > > > > > > >
> > > > > > > > > > >>> > > wrote:
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > > Hi, Justine,
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > Thanks for the reply.
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > So, if we downgrade TV, we could implicitly
> > > > downgrade
> > > > > > > > another
> > > > > > > > > > >>> feature
> > > > > > > > > > >>> > > > (say
> > > > > > > > > > >>> > > > > MV) that has dependency (e.g. RPC). What
> would
> > we
> > > > > > return
> > > > > > > > for
> > > > > > > > > > >>> > > > > FinalizedFeatures for MV in
> ApiVersionsResponse
> > > in
> > > > > that
> > > > > > > > case?
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > Thanks,
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > Jun
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > > > > > > > >>> > > > <jolshan@confluent.io.invalid
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > wrote:
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > Yes, the idea is that if we downgrade TV
> > > > > (transaction
> > > > > > > > > > version)
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > will
> > > > > > > > > > >>> > > > > stop
> > > > > > > > > > >>> > > > > > using the add partitions to txn
> optimization
> > > and
> > > > > stop
> > > > > > > > > writing
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > flexible
> > > > > > > > > > >>> > > > > > feature version of the log.
> > > > > > > > > > >>> > > > > > In the compatibility section I included
> some
> > > > > > > explanations
> > > > > > > > > on
> > > > > > > > > > >>> how
> > > > > > > > > > >>> > this
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > done.
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > Thanks,
> > > > > > > > > > >>> > > > > > Justine
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > > > > > > > >>> <ju...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > wrote:
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > Thanks for the update.
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > If we ever downgrade the transaction
> > feature,
> > > > any
> > > > > > > > feature
> > > > > > > > > > >>> > depending
> > > > > > > > > > >>> > > > on
> > > > > > > > > > >>> > > > > > > changes on top of those RPC/record
> > > > > > > > > > >>> > > > > > >
> > > (AddPartitionsToTxnRequest/TransactionLogValue)
> > > > > > > changes
> > > > > > > > > > made
> > > > > > > > > > >>> in
> > > > > > > > > > >>> > > > KIP-890
> > > > > > > > > > >>> > > > > > > will be automatically downgraded too?
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > Jun
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine
> > > Olshan
> > > > > > > > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > wrote:
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > I wanted to get back to you about your
> > > > > questions
> > > > > > > > about
> > > > > > > > > > >>> MV/IBP.
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > Looking at the options, I think it
> makes
> > > the
> > > > > most
> > > > > > > > sense
> > > > > > > > > > to
> > > > > > > > > > >>> > > create a
> > > > > > > > > > >>> > > > > > > > separate feature for transactions and
> use
> > > > that
> > > > > to
> > > > > > > > > version
> > > > > > > > > > >>> gate
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > features
> > > > > > > > > > >>> > > > > > > > we need to version gate (flexible
> > > > transactional
> > > > > > > state
> > > > > > > > > > >>> records
> > > > > > > > > > >>> > and
> > > > > > > > > > >>> > > > > using
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > new protocol)
> > > > > > > > > > >>> > > > > > > > I've updated the KIP to include this
> > > change.
> > > > > > > > Hopefully
> > > > > > > > > > >>> that's
> > > > > > > > > > >>> > > > > > everything
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > need for this KIP :)
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine
> > > > Olshan
> > > > > <
> > > > > > > > > > >>> > > > jolshan@confluent.io
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > > Thanks Jun,
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > I will update the KIP with the prev
> > field
> > > > for
> > > > > > > > prepare
> > > > > > > > > > as
> > > > > > > > > > >>> > well.
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > PREPARE
> > > > > > > > > > >>> > > > > > > > > producerId: x
> > > > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged
> > field): x
> > > > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > or z
> > > > if
> > > > > y
> > > > > > > will
> > > > > > > > > > >>> overflow
> > > > > > > > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > COMPLETE
> > > > > > > > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged
> > field): x
> > > > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > > > > > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we
> > > overflowed
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > Thanks again,
> > > > > > > > > > >>> > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun
> Rao
> > > > > > > > > > >>> > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > > > > > > > >>> > > > > > > > >> (1) My point was that the
> coordinator
> > > > could
> > > > > > fail
> > > > > > > > > right
> > > > > > > > > > >>> after
> > > > > > > > > > >>> > > > > writing
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> prepare marker. When the new txn
> > > > coordinator
> > > > > > > > > generates
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > complete
> > > > > > > > > > >>> > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> after the failover, it needs some
> > field
> > > > from
> > > > > > the
> > > > > > > > > > prepare
> > > > > > > > > > >>> > > marker
> > > > > > > > > > >>> > > > to
> > > > > > > > > > >>> > > > > > > > >> determine whether it's written by
> the
> > > new
> > > > > > > client.
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> (2) The changing of the behavior
> > sounds
> > > > good
> > > > > > to
> > > > > > > > me.
> > > > > > > > > We
> > > > > > > > > > >>> only
> > > > > > > > > > >>> > > want
> > > > > > > > > > >>> > > > > to
> > > > > > > > > > >>> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> success if the prepare state is
> > written
> > > by
> > > > > the
> > > > > > > new
> > > > > > > > > > >>> client.
> > > > > > > > > > >>> > So,
> > > > > > > > > > >>> > > > in
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> non-overflow case, it seems that we
> > also
> > > > > need
> > > > > > > sth
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >>> > > prepare
> > > > > > > > > > >>> > > > > > > marker
> > > > > > > > > > >>> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> tell us whether it's written by the
> > new
> > > > > > client.
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> 112. Thanks for the explanation.
> That
> > > > sounds
> > > > > > > good
> > > > > > > > to
> > > > > > > > > > me.
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> Jun
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM
> > Justine
> > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> <jo...@confluent.io.invalid>
> wrote:
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> > 101.3 I realized that I actually
> > have
> > > > two
> > > > > > > > > questions.
> > > > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> > > need
> > > > to
> > > > > > > write
> > > > > > > > > the
> > > > > > > > > > >>> > > previous
> > > > > > > > > > >>> > > > > > > produce
> > > > > > > > > > >>> > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > tagged field in the end maker so
> > that
> > > we
> > > > > > know
> > > > > > > if
> > > > > > > > > the
> > > > > > > > > > >>> > marker
> > > > > > > > > > >>> > > is
> > > > > > > > > > >>> > > > > > from
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> new
> > > > > > > > > > >>> > > > > > > > >> > client. Since the end maker is
> > derived
> > > > > from
> > > > > > > the
> > > > > > > > > > >>> prepare
> > > > > > > > > > >>> > > > marker,
> > > > > > > > > > >>> > > > > > > should
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > write the previous produce Id in
> the
> > > > > prepare
> > > > > > > > > marker
> > > > > > > > > > >>> field
> > > > > > > > > > >>> > > too?
> > > > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > > > >>> > > > > > > > >> > we will lose this information when
> > > > > deriving
> > > > > > > the
> > > > > > > > > end
> > > > > > > > > > >>> > marker.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > The "previous" producer ID is in
> the
> > > > > normal
> > > > > > > > > producer
> > > > > > > > > > >>> ID
> > > > > > > > > > >>> > > field.
> > > > > > > > > > >>> > > > > So
> > > > > > > > > > >>> > > > > > > yes,
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > need it in prepare and that was
> > always
> > > > the
> > > > > > > plan.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I
> will
> > > > > > enumerate
> > > > > > > > the
> > > > > > > > > > >>> fields
> > > > > > > > > > >>> > and
> > > > > > > > > > >>> > > > add
> > > > > > > > > > >>> > > > > > > them
> > > > > > > > > > >>> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > > > > > > > >>> > > > > > > > >> > Say we have producer ID x and
> epoch
> > y.
> > > > > When
> > > > > > we
> > > > > > > > > > >>> overflow
> > > > > > > > > > >>> > > epoch
> > > > > > > > > > >>> > > > y
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > get
> > > > > > > > > > >>> > > > > > > > >> > producer ID Z.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > PREPARE
> > > > > > > > > > >>> > > > > > > > >> > producerId: x
> > > > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> > > field):
> > > > > > empty
> > > > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field):
> empty
> > > or
> > > > z
> > > > > > if y
> > > > > > > > > will
> > > > > > > > > > >>> > overflow
> > > > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > COMPLETE
> > > > > > > > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> > > field):
> > > > x
> > > > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field):
> empty
> > > > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we
> > > > overflowed
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > (2) In the prepare phase, if we
> > retry
> > > > and
> > > > > > see
> > > > > > > > > epoch
> > > > > > > > > > -
> > > > > > > > > > >>> 1 +
> > > > > > > > > > >>> > ID
> > > > > > > > > > >>> > > > in
> > > > > > > > > > >>> > > > > > last
> > > > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > > > >>> > > > > > > > >> > fields and are issuing the same
> > > command
> > > > > (ie
> > > > > > > > commit
> > > > > > > > > > not
> > > > > > > > > > >>> > > abort),
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> > success. The logic before KIP-890
> > > seems
> > > > to
> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > > > >> > in this case. Are we intentionally
> > > > making
> > > > > > this
> > > > > > > > > > change?
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > Hmm -- we would fence the producer
> > if
> > > > the
> > > > > > > epoch
> > > > > > > > is
> > > > > > > > > > >>> bumped
> > > > > > > > > > >>> > > and
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > get a
> > > > > > > > > > >>> > > > > > > > >> > lower epoch. Yes -- we are
> > > intentionally
> > > > > > > adding
> > > > > > > > > this
> > > > > > > > > > >>> to
> > > > > > > > > > >>> > > > prevent
> > > > > > > > > > >>> > > > > > > > fencing.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > 112. We already merged the code
> that
> > > > adds
> > > > > > the
> > > > > > > > > > >>> VerifyOnly
> > > > > > > > > > >>> > > field
> > > > > > > > > > >>> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which
> is
> > an
> > > > > inter
> > > > > > > > > broker
> > > > > > > > > > >>> > request.
> > > > > > > > > > >>> > > > It
> > > > > > > > > > >>> > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> that
> > > > > > > > > > >>> > > > > > > > >> > we didn't bump up the IBP for
> that.
> > Do
> > > > you
> > > > > > > know
> > > > > > > > > why?
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > We no longer need IBP for all
> > > > interbroker
> > > > > > > > requests
> > > > > > > > > > as
> > > > > > > > > > >>> > > > > ApiVersions
> > > > > > > > > > >>> > > > > > > > should
> > > > > > > > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > > > > > > > >>> > > > > > > > >> > We also handle unsupported version
> > > > errors
> > > > > > > > > correctly
> > > > > > > > > > >>> if we
> > > > > > > > > > >>> > > > > receive
> > > > > > > > > > >>> > > > > > > them
> > > > > > > > > > >>> > > > > > > > >> in
> > > > > > > > > > >>> > > > > > > > >> > edge cases like
> upgrades/downgrades.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > Justine
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM
> Jun
> > > Rao
> > > > > > > > > > >>> > > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> wrote:
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > 101.3 I realized that I actually
> > > have
> > > > > two
> > > > > > > > > > questions.
> > > > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> > > need
> > > > to
> > > > > > > write
> > > > > > > > > the
> > > > > > > > > > >>> > > previous
> > > > > > > > > > >>> > > > > > > produce
> > > > > > > > > > >>> > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > > tagged field in the end maker so
> > > that
> > > > we
> > > > > > > know
> > > > > > > > if
> > > > > > > > > > the
> > > > > > > > > > >>> > > marker
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > > from
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > new
> > > > > > > > > > >>> > > > > > > > >> > > client. Since the end maker is
> > > derived
> > > > > > from
> > > > > > > > the
> > > > > > > > > > >>> prepare
> > > > > > > > > > >>> > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> should we
> > > > > > > > > > >>> > > > > > > > >> > > write the previous produce Id in
> > the
> > > > > > prepare
> > > > > > > > > > marker
> > > > > > > > > > >>> > field
> > > > > > > > > > >>> > > > too?
> > > > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > > > >>> > > > > > > > >> > > we will lose this information
> when
> > > > > > deriving
> > > > > > > > the
> > > > > > > > > > end
> > > > > > > > > > >>> > > marker.
> > > > > > > > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we
> > > retry
> > > > > and
> > > > > > > see
> > > > > > > > > > epoch
> > > > > > > > > > >>> - 1
> > > > > > > > > > >>> > +
> > > > > > > > > > >>> > > ID
> > > > > > > > > > >>> > > > > in
> > > > > > > > > > >>> > > > > > > last
> > > > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > > > >>> > > > > > > > >> > > fields and are issuing the same
> > > > command
> > > > > > (ie
> > > > > > > > > commit
> > > > > > > > > > >>> not
> > > > > > > > > > >>> > > > abort),
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> return
> > > > > > > > > > >>> > > > > > > > >> > > success. The logic before
> KIP-890
> > > > seems
> > > > > to
> > > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > > > >> > > in this case. Are we
> intentionally
> > > > > making
> > > > > > > this
> > > > > > > > > > >>> change?
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > 112. We already merged the code
> > that
> > > > > adds
> > > > > > > the
> > > > > > > > > > >>> VerifyOnly
> > > > > > > > > > >>> > > > field
> > > > > > > > > > >>> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which
> > is
> > > an
> > > > > > inter
> > > > > > > > > > broker
> > > > > > > > > > >>> > > request.
> > > > > > > > > > >>> > > > > It
> > > > > > > > > > >>> > > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > > >>> > > > > > > > >> > > we didn't bump up the IBP for
> > that.
> > > Do
> > > > > you
> > > > > > > > know
> > > > > > > > > > why?
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > Jun
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM
> > > > Justine
> > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen"
> > to
> > > > > > "current
> > > > > > > > > > >>> producer id
> > > > > > > > > > >>> > > and
> > > > > > > > > > >>> > > > > > > epoch"
> > > > > > > > > > >>> > > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > > >>> > > > > > > > >> > > > was the part that was
> confusing
> > > > > > > > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > > > > > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > > > > > > > >>> > > > > > > > >> > > > 112 We still need it. But I am
> > > still
> > > > > > > > > finalizing
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > design.
> > > > > > > > > > >>> > > > > I
> > > > > > > > > > >>> > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > update
> > > > > > > > > > >>> > > > > > > > >> > > > the KIP once I get the
> > information
> > > > > > > > finalized.
> > > > > > > > > > >>> Sorry
> > > > > > > > > > >>> > for
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > delays.
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at
> 10:50 AM
> > > Jun
> > > > > Rao
> > > > > > > > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow
> > case,
> > > > the
> > > > > > > > previous
> > > > > > > > > > ID
> > > > > > > > > > >>> is
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > same
> > > > > > > > > > >>> > > > > > as
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > ID for the complete marker
> > too,
> > > > but
> > > > > we
> > > > > > > set
> > > > > > > > > the
> > > > > > > > > > >>> > > previous
> > > > > > > > > > >>> > > > ID
> > > > > > > > > > >>> > > > > > in
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > marker. Earlier you
> mentioned
> > > that
> > > > > > this
> > > > > > > is
> > > > > > > > > to
> > > > > > > > > > >>> know
> > > > > > > > > > >>> > > that
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > > > >>> > > > > > > > >> > > > > written by the new client so
> > > that
> > > > we
> > > > > > > could
> > > > > > > > > > >>> return
> > > > > > > > > > >>> > > > success
> > > > > > > > > > >>> > > > > on
> > > > > > > > > > >>> > > > > > > > >> retried
> > > > > > > > > > >>> > > > > > > > >> > > > > endMarker requests. I was
> > trying
> > > > to
> > > > > > > > > understand
> > > > > > > > > > >>> why
> > > > > > > > > > >>> > > this
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > not
> > > > > > > > > > >>> > > > > > > > >> needed
> > > > > > > > > > >>> > > > > > > > >> > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > the prepare marker since
> retry
> > > can
> > > > > > > happen
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >>> > > prepare
> > > > > > > > > > >>> > > > > > state
> > > > > > > > > > >>> > > > > > > > >> too.
> > > > > > > > > > >>> > > > > > > > >> > Is
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > reason that in the prepare
> > > state,
> > > > we
> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > > > >> > > > instead
> > > > > > > > > > >>> > > > > > > > >> > > > > of success on retried
> endMaker
> > > > > > requests?
> > > > > > > > If
> > > > > > > > > > so,
> > > > > > > > > > >>> > should
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > change
> > > > > > > > > > >>> > > > > > > > >> "If
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID
> > in
> > > > last
> > > > > > > seen
> > > > > > > > > > >>> fields and
> > > > > > > > > > >>> > > are
> > > > > > > > > > >>> > > > > > > issuing
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > command (ie commit not
> abort)
> > we
> > > > can
> > > > > > > > return
> > > > > > > > > > >>> (with
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > epoch)"
> > > > > > > > > > >>> > > > > > > > >> > > > > accordingly?
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP,
> a
> > > > > delayed
> > > > > > > > > endMaker
> > > > > > > > > > >>> > request
> > > > > > > > > > >>> > > > > > carries
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > > > >>> > > > > > > > >> > > > > epoch and won't be fenced.
> > This
> > > > can
> > > > > > > > > > >>> commit/abort a
> > > > > > > > > > >>> > > > future
> > > > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure
> if
> > > we
> > > > > have
> > > > > > > > seen
> > > > > > > > > > >>> this in
> > > > > > > > > > >>> > > > > practice
> > > > > > > > > > >>> > > > > > > > >> though.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would
> be
> > > > useful
> > > > > > to
> > > > > > > > make
> > > > > > > > > > it
> > > > > > > > > > >>> > clear
> > > > > > > > > > >>> > > > that
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> now
> > > > > > > > > > >>> > > > > > > > >> > > > > populate the lastSeen field
> > from
> > > > the
> > > > > > log
> > > > > > > > > > >>> reliably.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > > > >>> > > > > since
> > > > > > > > > > >>> > > > > > > > it's
> > > > > > > > > > >>> > > > > > > > >> > > called
> > > > > > > > > > >>> > > > > > > > >> > > > > across brokers and we are
> > > changing
> > > > > its
> > > > > > > > > schema.
> > > > > > > > > > >>> Are
> > > > > > > > > > >>> > you
> > > > > > > > > > >>> > > > > > saying
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > don't
> > > > > > > > > > >>> > > > > > > > >> > > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > it any more? I thought that
> we
> > > > > already
> > > > > > > > > > >>> implemented
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > server
> > > > > > > > > > >>> > > > > > > > side
> > > > > > > > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > > > >>> > > > > across
> > > > > > > > > > >>> > > > > > > > >> brokers.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at
> > 5:05 PM
> > > > > > Justine
> > > > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > >
> <jolshan@confluent.io.invalid
> > >
> > > > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the
> > > previous
> > > > ID
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>> Prepare
> > > > > > > > > > >>> > > > field
> > > > > > > > > > >>> > > > > > > since
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > > it. It is the same
> producer
> > ID
> > > > as
> > > > > > the
> > > > > > > > main
> > > > > > > > > > >>> > producer
> > > > > > > > > > >>> > > ID
> > > > > > > > > > >>> > > > > > > field.
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to
> > > > reread
> > > > > > your
> > > > > > > > > > message
> > > > > > > > > > >>> > about
> > > > > > > > > > >>> > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > markers.
> > > > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn
> > > marker
> > > > > > after
> > > > > > > > the
> > > > > > > > > > >>> > > transaction
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > > > already
> > > > > > > > > > >>> > > > > > > > >> > > > > complete?
> > > > > > > > > > >>> > > > > > > > >> > > > > > So we will commit the next
> > > > > > transaction
> > > > > > > > > early
> > > > > > > > > > >>> > without
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > fixes
> > > > > > > > > > >>> > > > > > > > >> in
> > > > > > > > > > >>> > > > > > > > >> > > part
> > > > > > > > > > >>> > > > > > > > >> > > > 2?
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 111 Yes -- this
> terminology
> > > was
> > > > > used
> > > > > > > in
> > > > > > > > a
> > > > > > > > > > >>> previous
> > > > > > > > > > >>> > > KIP
> > > > > > > > > > >>> > > > > and
> > > > > > > > > > >>> > > > > > > > never
> > > > > > > > > > >>> > > > > > > > >> > > > > > implemented it in the log
> --
> > > > only
> > > > > in
> > > > > > > > > memory
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which
> interbroker
> > > > > > protocol
> > > > > > > > are
> > > > > > > > > > you
> > > > > > > > > > >>> > > > referring
> > > > > > > > > > >>> > > > > > to?
> > > > > > > > > > >>> > > > > > > I
> > > > > > > > > > >>> > > > > > > > am
> > > > > > > > > > >>> > > > > > > > >> > > > working
> > > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > the design for the work to
> > > > remove
> > > > > > the
> > > > > > > > > extra
> > > > > > > > > > >>> add
> > > > > > > > > > >>> > > > > partitions
> > > > > > > > > > >>> > > > > > > > call
> > > > > > > > > > >>> > > > > > > > >> > and I
> > > > > > > > > > >>> > > > > > > > >> > > > > right
> > > > > > > > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I
> > > have
> > > > > yet
> > > > > > to
> > > > > > > > > > update
> > > > > > > > > > >>> that
> > > > > > > > > > >>> > > > > section
> > > > > > > > > > >>> > > > > > > as
> > > > > > > > > > >>> > > > > > > > I
> > > > > > > > > > >>> > > > > > > > >> > > > finalize
> > > > > > > > > > >>> > > > > > > > >> > > > > > the design so please stay
> > > tuned.
> > > > > Was
> > > > > > > > there
> > > > > > > > > > >>> > anything
> > > > > > > > > > >>> > > > else
> > > > > > > > > > >>> > > > > > you
> > > > > > > > > > >>> > > > > > > > >> > thought
> > > > > > > > > > >>> > > > > > > > >> > > > > needed
> > > > > > > > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at
> > > 3:07 PM
> > > > > Jun
> > > > > > > Rao
> > > > > > > > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > I don't see this create
> > any
> > > > > issue.
> > > > > > > It
> > > > > > > > > just
> > > > > > > > > > >>> makes
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > a
> > > > > > > > > > >>> > > > > > bit
> > > > > > > > > > >>> > > > > > > > >> hard to
> > > > > > > > > > >>> > > > > > > > >> > > > > explain
> > > > > > > > > > >>> > > > > > > > >> > > > > > > what this non-tagged
> > produce
> > > > id
> > > > > > > field
> > > > > > > > > > >>> means. We
> > > > > > > > > > >>> > > are
> > > > > > > > > > >>> > > > > > > > >> essentially
> > > > > > > > > > >>> > > > > > > > >> > > > trying
> > > > > > > > > > >>> > > > > > > > >> > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > combine two actions
> > > > (completing
> > > > > a
> > > > > > > txn
> > > > > > > > > and
> > > > > > > > > > >>> init a
> > > > > > > > > > >>> > > new
> > > > > > > > > > >>> > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> Id)
> > > > > > > > > > >>> > > > > > > > >> > > in a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > single record. But, this
> > may
> > > > be
> > > > > > fine
> > > > > > > > > too.
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > A few other follow up
> > > > comments.
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason
> > > that
> > > > we
> > > > > > > only
> > > > > > > > > set
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > previous
> > > > > > > > > > >>> > > > > > > > >> produce id
> > > > > > > > > > >>> > > > > > > > >> > > > > tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > field in the complete
> > > marker,
> > > > > but
> > > > > > > not
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >>> > > prepare
> > > > > > > > > > >>> > > > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > prepare state, we always
> > > > return
> > > > > > > > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > retried
> > > > > > > > > > >>> > > > > > > > >> > > > > > endMaker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > requests?
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 110. "I believe your
> > second
> > > > > point
> > > > > > is
> > > > > > > > > > >>> mentioned
> > > > > > > > > > >>> > in
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > KIP. I
> > > > > > > > > > >>> > > > > > > > >> can
> > > > > > > > > > >>> > > > > > > > >> > > add
> > > > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > text on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > The delayed message
> case
> > > can
> > > > > > also
> > > > > > > > > > violate
> > > > > > > > > > >>> EOS
> > > > > > > > > > >>> > if
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > message
> > > > > > > > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > > > > > > > addPartitionsToTxn
> > > > > > > > > > >>> > request
> > > > > > > > > > >>> > > > > comes
> > > > > > > > > > >>> > > > > > > in.
> > > > > > > > > > >>> > > > > > > > >> > > > > Effectively
> > > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > may see a message from a
> > > > > previous
> > > > > > > > > > (aborted)
> > > > > > > > > > >>> > > > > transaction
> > > > > > > > > > >>> > > > > > > > become
> > > > > > > > > > >>> > > > > > > > >> > part
> > > > > > > > > > >>> > > > > > > > >> > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > The above is the case
> > when a
> > > > > > delayed
> > > > > > > > > > >>> message is
> > > > > > > > > > >>> > > > > appended
> > > > > > > > > > >>> > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > data
> > > > > > > > > > >>> > > > > > > > >> > > > > > > partition. What I
> > mentioned
> > > > is a
> > > > > > > > > slightly
> > > > > > > > > > >>> > > different
> > > > > > > > > > >>> > > > > case
> > > > > > > > > > >>> > > > > > > > when
> > > > > > > > > > >>> > > > > > > > >> a
> > > > > > > > > > >>> > > > > > > > >> > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > marker is appended to
> the
> > > > > > > transaction
> > > > > > > > > log
> > > > > > > > > > >>> > > partition.
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once
> we
> > > > move
> > > > > > past
> > > > > > > > the
> > > > > > > > > > >>> Prepare
> > > > > > > > > > >>> > > and
> > > > > > > > > > >>> > > > > > > > Complete
> > > > > > > > > > >>> > > > > > > > >> > > states,
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > don’t need to worry
> about
> > > > > lastSeen
> > > > > > > > > fields
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > > clear
> > > > > > > > > > >>> > > > > > them,
> > > > > > > > > > >>> > > > > > > > just
> > > > > > > > > > >>> > > > > > > > >> > > handle
> > > > > > > > > > >>> > > > > > > > >> > > > > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > > > transitions as normal.".
> > Is
> > > > the
> > > > > > > > lastSeen
> > > > > > > > > > >>> field
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > same
> > > > > > > > > > >>> > > > > > as
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field
> in
> > > > > > > > > > >>> TransactionLogValue?
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 112. Since the kip
> changes
> > > the
> > > > > > > > > > inter-broker
> > > > > > > > > > >>> > > > protocol,
> > > > > > > > > > >>> > > > > > > should
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > > bump
> > > > > > > > > > >>> > > > > > > > >> > > > up
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this
> > > > feature
> > > > > > only
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > >>> > KRaft
> > > > > > > > > > >>> > > > > mode?
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at
> > > > 11:13 AM
> > > > > > > > Justine
> > > > > > > > > > >>> Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > <jolshan@confluent.io.invalid
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > I'm glad we are
> getting
> > to
> > > > > > > > convergence
> > > > > > > > > > on
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > design.
> > > > > > > > > > >>> > > > > > :)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > While I understand it
> > > seems
> > > > a
> > > > > > > little
> > > > > > > > > > >>> "weird".
> > > > > > > > > > >>> > > I'm
> > > > > > > > > > >>> > > > > not
> > > > > > > > > > >>> > > > > > > sure
> > > > > > > > > > >>> > > > > > > > >> what
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > benefit
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > of writing an extra
> > record
> > > > to
> > > > > > the
> > > > > > > > log.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool
> to
> > > > > > describe
> > > > > > > > > > >>> transactions
> > > > > > > > > > >>> > > > won't
> > > > > > > > > > >>> > > > > > > work
> > > > > > > > > > >>> > > > > > > > >> (ie,
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > state is needed to
> > > calculate
> > > > > the
> > > > > > > > time
> > > > > > > > > > >>> since
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > completed?)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > If we have a reason
> like
> > > > this,
> > > > > > it
> > > > > > > is
> > > > > > > > > > >>> enough to
> > > > > > > > > > >>> > > > > > convince
> > > > > > > > > > >>> > > > > > > me
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > such
> > > > > > > > > > >>> > > > > > > > >> > > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > extra record. It seems
> > > like
> > > > it
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > >>> > > replacing
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > record
> > > > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is
> this
> > > > > correct?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024
> at
> > > > > 5:14 PM
> > > > > > > Jun
> > > > > > > > > Rao
> > > > > > > > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks for the
> > > > explanation.
> > > > > I
> > > > > > > > > > >>> understand the
> > > > > > > > > > >>> > > > > > intention
> > > > > > > > > > >>> > > > > > > > >> now.
> > > > > > > > > > >>> > > > > > > > >> > In
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > case, we set the
> > > > non-tagged
> > > > > > > field
> > > > > > > > to
> > > > > > > > > > >>> the old
> > > > > > > > > > >>> > > pid
> > > > > > > > > > >>> > > > > > (and
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> max
> > > > > > > > > > >>> > > > > > > > >> > > > > epoch)
> > > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > prepare marker so
> that
> > > we
> > > > > > could
> > > > > > > > > > >>> correctly
> > > > > > > > > > >>> > > write
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> marker to
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > data
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > partition if the
> > broker
> > > > > > > > downgrades.
> > > > > > > > > > When
> > > > > > > > > > >>> > > writing
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> complete
> > > > > > > > > > >>> > > > > > > > >> > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > know the marker has
> > > > already
> > > > > > been
> > > > > > > > > > >>> written to
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > data
> > > > > > > > > > >>> > > > > > > > >> > partition.
> > > > > > > > > > >>> > > > > > > > >> > > > We
> > > > > > > > > > >>> > > > > > > > >> > > > > > set
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to
> > the
> > > > new
> > > > > > pid
> > > > > > > to
> > > > > > > > > > avoid
> > > > > > > > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > client if the broker
> > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > The above seems to
> > work.
> > > > > It's
> > > > > > > > just a
> > > > > > > > > > bit
> > > > > > > > > > >>> > > > > > inconsistent
> > > > > > > > > > >>> > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> a
> > > > > > > > > > >>> > > > > > > > >> > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > and a complete
> marker
> > to
> > > > use
> > > > > > > > > different
> > > > > > > > > > >>> pids
> > > > > > > > > > >>> > in
> > > > > > > > > > >>> > > > > this
> > > > > > > > > > >>> > > > > > > > >> special
> > > > > > > > > > >>> > > > > > > > >> > > case.
> > > > > > > > > > >>> > > > > > > > >> > > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrade with the
> > > > complete
> > > > > > > > marker,
> > > > > > > > > it
> > > > > > > > > > >>> seems
> > > > > > > > > > >>> > > > that
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > never
> > > > > > > > > > >>> > > > > > > > >> > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > able
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > write the complete
> > > marker
> > > > > with
> > > > > > > the
> > > > > > > > > old
> > > > > > > > > > >>> pid.
> > > > > > > > > > >>> > > Not
> > > > > > > > > > >>> > > > > sure
> > > > > > > > > > >>> > > > > > > if
> > > > > > > > > > >>> > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > causes
> > > > > > > > > > >>> > > > > > > > >> > > > > any
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems
> a
> > > bit
> > > > > > weird.
> > > > > > > > > > >>> Instead of
> > > > > > > > > > >>> > > > > writing
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > with the new pid,
> > could
> > > we
> > > > > > write
> > > > > > > > two
> > > > > > > > > > >>> > records:
> > > > > > > > > > >>> > > a
> > > > > > > > > > >>> > > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > with
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > old pid followed by
> a
> > > > > > > > > > >>> TransactionLogValue
> > > > > > > > > > >>> > with
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > new
> > > > > > > > > > >>> > > > > > > > pid
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > empty
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > state? We could make
> > the
> > > > two
> > > > > > > > records
> > > > > > > > > > in
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > same
> > > > > > > > > > >>> > > > > > batch
> > > > > > > > > > >>> > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > > >>> > > > > > > > >> > > > they
> > > > > > > > > > >>> > > > > > > > >> > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > added to the log
> > > > atomically.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024
> > at
> > > > > > 5:40 PM
> > > > > > > > > > Justine
> > > > > > > > > > >>> > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > <jolshan@confluent.io.invalid
> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare
> > marker
> > > > is
> > > > > > > > written,
> > > > > > > > > > >>> but the
> > > > > > > > > > >>> > > > > endTxn
> > > > > > > > > > >>> > > > > > > > >> response
> > > > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > > > >>> > > > > > > > >> > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > received
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > by the client when
> > the
> > > > > > server
> > > > > > > > > > >>> downgrades
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare
> > > marker
> > > > is
> > > > > > > > > written,
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > endTxn
> > > > > > > > > > >>> > > > > > > > response
> > > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > > >>> > > > > > > > >> > > > > received
> > > > > > > > > > >>> > > > > > > > >> > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > client when the
> > server
> > > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > I think I am
> still a
> > > > > little
> > > > > > > > > > confused.
> > > > > > > > > > >>> In
> > > > > > > > > > >>> > > both
> > > > > > > > > > >>> > > > of
> > > > > > > > > > >>> > > > > > > these
> > > > > > > > > > >>> > > > > > > > >> > cases,
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction log
> has
> > > the
> > > > > old
> > > > > > > > > producer
> > > > > > > > > > >>> ID.
> > > > > > > > > > >>> > We
> > > > > > > > > > >>> > > > > don't
> > > > > > > > > > >>> > > > > > > > write
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > in the prepare
> > > marker's
> > > > > non
> > > > > > > > tagged
> > > > > > > > > > >>> fields.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > If the server
> > > downgrades
> > > > > > now,
> > > > > > > it
> > > > > > > > > > would
> > > > > > > > > > >>> > read
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > records
> > > > > > > > > > >>> > > > > > > > >> not
> > > > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields and the
> > > complete
> > > > > > marker
> > > > > > > > > will
> > > > > > > > > > >>> also
> > > > > > > > > > >>> > > have
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > old
> > > > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > ID.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (If we had used
> the
> > > new
> > > > > > > producer
> > > > > > > > > ID,
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > would
> > > > > > > > > > >>> > > > > not
> > > > > > > > > > >>> > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > transactional
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > correctness since
> > the
> > > > > > producer
> > > > > > > > id
> > > > > > > > > > >>> doesn't
> > > > > > > > > > >>> > > > match
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > state would not be
> > > > correct
> > > > > > on
> > > > > > > > the
> > > > > > > > > > data
> > > > > > > > > > >>> > > > > partition.)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > In the overflow
> > case,
> > > > I'd
> > > > > > > expect
> > > > > > > > > the
> > > > > > > > > > >>> > > following
> > > > > > > > > > >>> > > > > to
> > > > > > > > > > >>> > > > > > > > >> happen on
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > side
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we
> retry
> > > > EndTxn
> > > > > > --
> > > > > > > it
> > > > > > > > > is
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > same
> > > > > > > > > > >>> > > > > > > producer
> > > > > > > > > > >>> > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > -
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 1
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > would fence the
> > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't
> > > retry
> > > > > > > EndTxn
> > > > > > > > > and
> > > > > > > > > > >>> use
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > >> producer id
> > > > > > > > > > >>> > > > > > > > >> > > > which
> > > > > > > > > > >>> > > > > > > > >> > > > > > > would
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > > > > > > > InvalidPidMappingException
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have
> > > > special
> > > > > > > > handling
> > > > > > > > > > for
> > > > > > > > > > >>> > when
> > > > > > > > > > >>> > > a
> > > > > > > > > > >>> > > > > > server
> > > > > > > > > > >>> > > > > > > > >> > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > When
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > reconnects we
> could
> > > get
> > > > an
> > > > > > API
> > > > > > > > > > version
> > > > > > > > > > >>> > > request
> > > > > > > > > > >>> > > > > > > showing
> > > > > > > > > > >>> > > > > > > > >> > > KIP-890
> > > > > > > > > > >>> > > > > > > > >> > > > > > part 2
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > not supported. In
> > that
> > > > > case,
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > >>> call
> > > > > > > > > > >>> > > > > > > > initProducerId
> > > > > > > > > > >>> > > > > > > > >> to
> > > > > > > > > > >>> > > > > > > > >> > > > abort
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In
> the
> > > > > > overflow
> > > > > > > > > case,
> > > > > > > > > > >>> this
> > > > > > > > > > >>> > > > > correctly
> > > > > > > > > > >>> > > > > > > > gives
> > > > > > > > > > >>> > > > > > > > >> > us a
> > > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > I guess the
> > > > corresponding
> > > > > > case
> > > > > > > > > would
> > > > > > > > > > >>> be
> > > > > > > > > > >>> > > where
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> *complete
> > > > > > > > > > >>> > > > > > > > >> > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > *is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > written but the
> > endTxn
> > > > is
> > > > > > not
> > > > > > > > > > >>> received by
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > client
> > > > > > > > > > >>> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > server
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This
> > would
> > > > > > result
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>> > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > coordinator
> > > > > > > > > > >>> > > > > > > > >> > > > > having
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old
> > > one.
> > > > > If
> > > > > > > the
> > > > > > > > > > client
> > > > > > > > > > >>> > > > retries,
> > > > > > > > > > >>> > > > > it
> > > > > > > > > > >>> > > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > > receive
> > > > > > > > > > >>> > > > > > > > >> > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > InvalidPidMappingException.
> > > > > > > The
> > > > > > > > > > >>> > > InitProducerId
> > > > > > > > > > >>> > > > > > > > scenario
> > > > > > > > > > >>> > > > > > > > >> > above
> > > > > > > > > > >>> > > > > > > > >> > > > > would
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > help
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my
> > > > > > compatibility
> > > > > > > > > story
> > > > > > > > > > is
> > > > > > > > > > >>> > meant
> > > > > > > > > > >>> > > > to
> > > > > > > > > > >>> > > > > > > > support
> > > > > > > > > > >>> > > > > > > > >> > > > > downgrades
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > server
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > side in keeping
> the
> > > > > > > > transactional
> > > > > > > > > > >>> > > correctness.
> > > > > > > > > > >>> > > > > > > Keeping
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > from
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is
> > not
> > > > the
> > > > > > > > > priority.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I
> > can
> > > > > also
> > > > > > > add
> > > > > > > > > text
> > > > > > > > > > >>> in
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > KIP
> > > > > > > > > > >>> > > > > > > about
> > > > > > > > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > think that fixes
> > some
> > > > edge
> > > > > > > > cases.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12,
> 2024
> > > at
> > > > > > > 4:10 PM
> > > > > > > > > Jun
> > > > > > > > > > >>> Rao
> > > > > > > > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the
> > > reply.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we
> > > don't
> > > > > need
> > > > > > > to
> > > > > > > > > > >>> optimize
> > > > > > > > > > >>> > for
> > > > > > > > > > >>> > > > > > fencing
> > > > > > > > > > >>> > > > > > > > >> during
> > > > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Regarding
> > > consistency,
> > > > > > there
> > > > > > > > are
> > > > > > > > > > two
> > > > > > > > > > >>> > > > possible
> > > > > > > > > > >>> > > > > > > cases:
> > > > > > > > > > >>> > > > > > > > >> (1)
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is written, but
> > the
> > > > > endTxn
> > > > > > > > > > response
> > > > > > > > > > >>> is
> > > > > > > > > > >>> > not
> > > > > > > > > > >>> > > > > > > received
> > > > > > > > > > >>> > > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > when
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > server
> downgrades;
> > > (2)
> > > > > > the
> > > > > > > > > > prepare
> > > > > > > > > > >>> > marker
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > > > written,
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > endTxn
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is received by
> the
> > > > > client
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > >>> > server
> > > > > > > > > > >>> > > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> In
> > > > > > > > > > >>> > > > > > > > >> > > (1),
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > will have the
> old
> > > > > produce
> > > > > > Id
> > > > > > > > and
> > > > > > > > > > in
> > > > > > > > > > >>> (2),
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> will
> > > > > > > > > > >>> > > > > > > > >> > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If
> we
> > > > > > downgrade
> > > > > > > > > right
> > > > > > > > > > >>> after
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > can't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > consistent to
> both
> > > (1)
> > > > > and
> > > > > > > (2)
> > > > > > > > > > >>> since we
> > > > > > > > > > >>> > > can
> > > > > > > > > > >>> > > > > only
> > > > > > > > > > >>> > > > > > > put
> > > > > > > > > > >>> > > > > > > > >> one
> > > > > > > > > > >>> > > > > > > > >> > > > value
> > > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > existing produce
> > Id
> > > > > field.
> > > > > > > > It's
> > > > > > > > > > >>> also not
> > > > > > > > > > >>> > > > clear
> > > > > > > > > > >>> > > > > > > which
> > > > > > > > > > >>> > > > > > > > >> case
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > So we could
> > probably
> > > > be
> > > > > > > > > consistent
> > > > > > > > > > >>> with
> > > > > > > > > > >>> > > > either
> > > > > > > > > > >>> > > > > > > case.
> > > > > > > > > > >>> > > > > > > > >> By
> > > > > > > > > > >>> > > > > > > > >> > > > putting
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in
> the
> > > > > prepare
> > > > > > > > > marker,
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > are
> > > > > > > > > > >>> > > > > > > consistent
> > > > > > > > > > >>> > > > > > > > >> with
> > > > > > > > > > >>> > > > > > > > >> > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > (2)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > also has the
> > slight
> > > > > > benefit
> > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > >>> > > produce
> > > > > > > > > > >>> > > > > > field
> > > > > > > > > > >>> > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > complete marker
> > are
> > > > > > > consistent
> > > > > > > > > in
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > overflow
> > > > > > > > > > >>> > > > > > > case.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12,
> > 2024
> > > > at
> > > > > > > > 3:11 PM
> > > > > > > > > > >>> Justine
> > > > > > > > > > >>> > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > <jolshan@confluent.io.invalid
> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the case
> you
> > > > > > describe,
> > > > > > > we
> > > > > > > > > > would
> > > > > > > > > > >>> > need
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > > have
> > > > > > > > > > >>> > > > > > > a
> > > > > > > > > > >>> > > > > > > > >> > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > request,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > successful
> > EndTxn,
> > > > > and a
> > > > > > > > > > >>> successful
> > > > > > > > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > then
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn
> > > > request
> > > > > > go
> > > > > > > > > > through
> > > > > > > > > > >>> for
> > > > > > > > > > >>> > a
> > > > > > > > > > >>> > > > > given
> > > > > > > > > > >>> > > > > > > > >> producer.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to
> > > figure
> > > > > out
> > > > > > > if
> > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > >>> > > possible
> > > > > > > > > > >>> > > > > for
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > client
> > > > > > > > > > >>> > > > > > > > >> > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > a previous
> > request
> > > > is
> > > > > > > > delayed
> > > > > > > > > > >>> > somewhere.
> > > > > > > > > > >>> > > > But
> > > > > > > > > > >>> > > > > > > yes,
> > > > > > > > > > >>> > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > > > >>> > > > > > > > >> > > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > think
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > would fence
> the
> > > > > client.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the
> > > overflow
> > > > > > case.
> > > > > > > > In
> > > > > > > > > > the
> > > > > > > > > > >>> > > overflow
> > > > > > > > > > >>> > > > > > case,
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are
> > > different
> > > > on
> > > > > > the
> > > > > > > > > > marker
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > on
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> > > > > > So
> > > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to
> > use
> > > > the
> > > > > > max
> > > > > > > > > epoch
> > > > > > > > > > >>> but
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > start
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and
> > > > epoch 0
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>> > > > transactional
> > > > > > > > > > >>> > > > > > > state.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the server
> > > > > downgrade
> > > > > > > > case,
> > > > > > > > > we
> > > > > > > > > > >>> want
> > > > > > > > > > >>> > to
> > > > > > > > > > >>> > > > see
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > > >>> > > > > > > > >> > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > what the
> client
> > > will
> > > > > > have.
> > > > > > > > If
> > > > > > > > > we
> > > > > > > > > > >>> > > complete
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> commit,
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > state is
> > reloaded,
> > > > we
> > > > > > need
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > >>> > > > producer
> > > > > > > > > > >>> > > > > ID
> > > > > > > > > > >>> > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > there
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid
> > > producer
> > > > ID
> > > > > > > > > mapping.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > The server
> > > downgrade
> > > > > > cases
> > > > > > > > are
> > > > > > > > > > >>> > > considering
> > > > > > > > > > >>> > > > > > > > >> > transactional
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing
> > > from
> > > > > > > previous
> > > > > > > > > > >>> behavior
> > > > > > > > > > >>> > --
> > > > > > > > > > >>> > > > and
> > > > > > > > > > >>> > > > > > are
> > > > > > > > > > >>> > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > concerned
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > about
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the
> > > > safety
> > > > > > from
> > > > > > > > > > fencing
> > > > > > > > > > >>> > > retries
> > > > > > > > > > >>> > > > > (as
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> have
> > > > > > > > > > >>> > > > > > > > >> > > > > > downgraded
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to
> > > > > support).
> > > > > > > > > Perhaps
> > > > > > > > > > >>> this
> > > > > > > > > > >>> > is
> > > > > > > > > > >>> > > a
> > > > > > > > > > >>> > > > > > trade
> > > > > > > > > > >>> > > > > > > > off,
> > > > > > > > > > >>> > > > > > > > >> > but
> > > > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > > think
> > > > > > > > > > >>> > > > > > > > >> > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client
> > > > > > downgrades,
> > > > > > > > it
> > > > > > > > > > will
> > > > > > > > > > >>> > have
> > > > > > > > > > >>> > > > > > > restarted
> > > > > > > > > > >>> > > > > > > > >> and
> > > > > > > > > > >>> > > > > > > > >> > it
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > ok
> > > > > > > > > > >>> > > > > > > > >> > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > have a new
> > > producer
> > > > ID
> > > > > > > too).
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan
> 12,
> > > 2024
> > > > > at
> > > > > > > > > 11:42 AM
> > > > > > > > > > >>> Jun
> > > > > > > > > > >>> > Rao
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for
> the
> > > > > reply.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If
> the
> > > > marker
> > > > > > is
> > > > > > > > > > written
> > > > > > > > > > >>> by
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > >> client, we
> > > > > > > > > > >>> > > > > > > > >> > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the last
> email
> > > > > > guarantee
> > > > > > > > > that
> > > > > > > > > > >>> any
> > > > > > > > > > >>> > > EndTxn
> > > > > > > > > > >>> > > > > > > > requests
> > > > > > > > > > >>> > > > > > > > >> > with
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from the
> same
> > > > > producer
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > >>> same
> > > > > > > > > > >>> > > > > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> Then
> > > > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > return a
> > fenced
> > > > > error
> > > > > > > but
> > > > > > > > > can
> > > > > > > > > > >>> handle
> > > > > > > > > > >>> > > > > > > gracefully
> > > > > > > > > > >>> > > > > > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > described
> > > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When a
> delayed
> > > > > EndTnx
> > > > > > > > > request
> > > > > > > > > > is
> > > > > > > > > > >>> > > > > processed,
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> txn
> > > > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > > could
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next
> > > txn.
> > > > I
> > > > > > > guess
> > > > > > > > in
> > > > > > > > > > >>> this
> > > > > > > > > > >>> > case
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > still
> > > > > > > > > > >>> > > > > > > > >> > return
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > fenced
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed
> > > > request?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry,
> my
> > > > > > question
> > > > > > > > was
> > > > > > > > > > >>> > > inaccurate.
> > > > > > > > > > >>> > > > > What
> > > > > > > > > > >>> > > > > > > you
> > > > > > > > > > >>> > > > > > > > >> > > > described
> > > > > > > > > > >>> > > > > > > > >> > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > "The
> downgrade
> > > > > > > > > compatibility I
> > > > > > > > > > >>> > mention
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > that
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > keep
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in
> > the
> > > > > main
> > > > > > > > > > >>> (non-tagged)
> > > > > > > > > > >>> > > > fields
> > > > > > > > > > >>> > > > > as
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> did
> > > > > > > > > > >>> > > > > > > > >> > > > before
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > code
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > server
> side."
> > If
> > > > we
> > > > > > want
> > > > > > > > to
> > > > > > > > > do
> > > > > > > > > > >>> this,
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> that we
> > > > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > > > >>> > > > > > > > >> > > > > > > use
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > current
> > produce
> > > Id
> > > > > and
> > > > > > > max
> > > > > > > > > > >>> epoch in
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > existing
> > > > > > > > > > >>> > > > > > > > >> > > > producerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> producerEpoch
> > > > fields
> > > > > > for
> > > > > > > > > both
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > prepare
> > > > > > > > > > >>> > > > > > and
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > The
> downgrade
> > > can
> > > > > > happen
> > > > > > > > > after
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > complete
> > > > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > > > >>> > > > > > > > >> > > > > > written.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > With
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > you
> described,
> > > the
> > > > > > > > > downgraded
> > > > > > > > > > >>> > > > coordinator
> > > > > > > > > > >>> > > > > > will
> > > > > > > > > > >>> > > > > > > > see
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of
> the
> > > old
> > > > > > one.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan
> > 12,
> > > > 2024
> > > > > > at
> > > > > > > > > > 10:44 AM
> > > > > > > > > > >>> > > Justine
> > > > > > > > > > >>> > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > <jolshan@confluent.io.invalid
> > > > > > > > > > >
> > > > > > > > > > >>> > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can
> update
> > > the
> > > > > > > > > > description.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe
> > your
> > > > > > second
> > > > > > > > > point
> > > > > > > > > > is
> > > > > > > > > > >>> > > > mentioned
> > > > > > > > > > >>> > > > > > in
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > KIP.
> > > > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > add
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it
> > is
> > > > > > helpful.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The
> > delayed
> > > > > > message
> > > > > > > > case
> > > > > > > > > > can
> > > > > > > > > > >>> > also
> > > > > > > > > > >>> > > > > > violate
> > > > > > > > > > >>> > > > > > > > EOS
> > > > > > > > > > >>> > > > > > > > >> if
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in
> > after
> > > > the
> > > > > > > next
> > > > > > > > > > >>> > > > > addPartitionsToTxn
> > > > > > > > > > >>> > > > > > > > >> request
> > > > > > > > > > >>> > > > > > > > >> > > > comes
> > > > > > > > > > >>> > > > > > > > >> > > > > > in.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a
> > > > message
> > > > > > > from a
> > > > > > > > > > >>> previous
> > > > > > > > > > >>> > > > > > (aborted)
> > > > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > become
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > part
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > next
> > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the
> > marker
> > > is
> > > > > > > written
> > > > > > > > > by
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > new
> > > > > > > > > > >>> > > > > > client,
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> can
> > > > > > > > > > >>> > > > > > > > >> > > as I
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email
> > > > > guarantee
> > > > > > > > that
> > > > > > > > > > any
> > > > > > > > > > >>> > EndTxn
> > > > > > > > > > >>> > > > > > > requests
> > > > > > > > > > >>> > > > > > > > >> with
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same
> > > > producer
> > > > > > and
> > > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > >>> > > > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> Then we
> > > > > > > > > > >>> > > > > > > > >> > > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced
> error
> > > but
> > > > > can
> > > > > > > > > handle
> > > > > > > > > > >>> > > gracefully
> > > > > > > > > > >>> > > > > as
> > > > > > > > > > >>> > > > > > > > >> described
> > > > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't
> > think
> > > a
> > > > > > > boolean
> > > > > > > > is
> > > > > > > > > > >>> useful
> > > > > > > > > > >>> > > > since
> > > > > > > > > > >>> > > > > it
> > > > > > > > > > >>> > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > directly
> > > > > > > > > > >>> > > > > > > > >> > > > > > > encoded
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence
> or
> > > > lack
> > > > > of
> > > > > > > the
> > > > > > > > > > >>> tagged
> > > > > > > > > > >>> > > field
> > > > > > > > > > >>> > > > > > being
> > > > > > > > > > >>> > > > > > > > >> > written.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> > prepare
> > > > > > marker
> > > > > > > we
> > > > > > > > > > will
> > > > > > > > > > >>> have
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In
> > the
> > > > > > Complete
> > > > > > > > > state
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > may
> > > > > > > > > > >>> > > > not.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not
> sure
> > > why
> > > > > the
> > > > > > > > > ongoing
> > > > > > > > > > >>> state
> > > > > > > > > > >>> > > > > matters
> > > > > > > > > > >>> > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> this
> > > > > > > > > > >>> > > > > > > > >> > > > KIP.
> > > > > > > > > > >>> > > > > > > > >> > > > > It
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > does
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not
> sure
> > > > what
> > > > > > you
> > > > > > > > are
> > > > > > > > > > >>> > referring
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > > about
> > > > > > > > > > >>> > > > > > > > >> > writing
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the
> > > > prepare
> > > > > > > > marker.
> > > > > > > > > > >>> This is
> > > > > > > > > > >>> > > not
> > > > > > > > > > >>> > > > in
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> KIP.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> > > overflow
> > > > > > case,
> > > > > > > we
> > > > > > > > > > >>> write the
> > > > > > > > > > >>> > > > > > > > >> nextProducerId
> > > > > > > > > > >>> > > > > > > > >> > in
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so
> > we
> > > > know
> > > > > > > what
> > > > > > > > we
> > > > > > > > > > >>> > assigned
> > > > > > > > > > >>> > > > when
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> reload
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we
> > > > complete,
> > > > > we
> > > > > > > > > > >>> transition
> > > > > > > > > > >>> > this
> > > > > > > > > > >>> > > > ID
> > > > > > > > > > >>> > > > > to
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> main
> > > > > > > > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the
> > > > previous
> > > > > > > > producer
> > > > > > > > > > ID
> > > > > > > > > > >>> > field
> > > > > > > > > > >>> > > > > filled
> > > > > > > > > > >>> > > > > > > in.
> > > > > > > > > > >>> > > > > > > > >> This
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry
> > > case
> > > > > the
> > > > > > > > > > operation
> > > > > > > > > > >>> > > > completed
> > > > > > > > > > >>> > > > > > > > >> > successfully
> > > > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer.
> > The
> > > > > > > downgrade
> > > > > > > > > > >>> > > compatibility
> > > > > > > > > > >>> > > > I
> > > > > > > > > > >>> > > > > > > > mention
> > > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > keep
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> ID
> > > and
> > > > > > epoch
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>> main
> > > > > > > > > > >>> > > > > > > (non-tagged)
> > > > > > > > > > >>> > > > > > > > >> > fields
> > > > > > > > > > >>> > > > > > > > >> > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > did
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on
> the
> > > > server
> > > > > > > side.
> > > > > > > > > If
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > server
> > > > > > > > > > >>> > > > > > > > >> downgrades,
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > still
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > addresses
> > > > > both
> > > > > > > the
> > > > > > > > > > >>> prepare
> > > > > > > > > > >>> > and
> > > > > > > > > > >>> > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> state
> > > > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri,
> Jan
> > > 12,
> > > > > 2024
> > > > > > > at
> > > > > > > > > > >>> 10:21 AM
> > > > > > > > > > >>> > Jun
> > > > > > > > > > >>> > > > Rao
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi,
> > Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks
> for
> > > the
> > > > > > > reply.
> > > > > > > > > > Sorry
> > > > > > > > > > >>> for
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > delay. I
> > > > > > > > > > >>> > > > > > > > >> > have a
> > > > > > > > > > >>> > > > > > > > >> > > > few
> > > > > > > > > > >>> > > > > > > > >> > > > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I
> > think
> > > > the
> > > > > > > > > > motivation
> > > > > > > > > > >>> > > section
> > > > > > > > > > >>> > > > > > could
> > > > > > > > > > >>> > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > improved.
> > > > > > > > > > >>> > > > > > > > >> > > > > > One
> > > > > > > > > > >>> > > > > > > > >> > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > motivations
> > > > > listed
> > > > > > > by
> > > > > > > > > the
> > > > > > > > > > >>> KIP is
> > > > > > > > > > >>> > > > "This
> > > > > > > > > > >>> > > > > > can
> > > > > > > > > > >>> > > > > > > > >> happen
> > > > > > > > > > >>> > > > > > > > >> > > > when
> > > > > > > > > > >>> > > > > > > > >> > > > > a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > message
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or
> delayed
> > > due
> > > > > to
> > > > > > > > > > networking
> > > > > > > > > > >>> > > issues
> > > > > > > > > > >>> > > > > or a
> > > > > > > > > > >>> > > > > > > > >> network
> > > > > > > > > > >>> > > > > > > > >> > > > > > partition,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts,
> > and
> > > > then
> > > > > > the
> > > > > > > > > > delayed
> > > > > > > > > > >>> > > message
> > > > > > > > > > >>> > > > > > > finally
> > > > > > > > > > >>> > > > > > > > >> > comes
> > > > > > > > > > >>> > > > > > > > >> > > > > in.".
> > > > > > > > > > >>> > > > > > > > >> > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very
> > > accurate.
> > > > > > > Without
> > > > > > > > > > >>> KIP-890,
> > > > > > > > > > >>> > > > > > currently,
> > > > > > > > > > >>> > > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts
> an
> > > > > ongoing
> > > > > > > > > > >>> transaction,
> > > > > > > > > > >>> > it
> > > > > > > > > > >>> > > > > > already
> > > > > > > > > > >>> > > > > > > > >> bumps
> > > > > > > > > > >>> > > > > > > > >> > up
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which
> > > prevents
> > > > > the
> > > > > > > > > delayed
> > > > > > > > > > >>> > produce
> > > > > > > > > > >>> > > > > > message
> > > > > > > > > > >>> > > > > > > > >> from
> > > > > > > > > > >>> > > > > > > > >> > > being
> > > > > > > > > > >>> > > > > > > > >> > > > > > added
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> partition.
> > > > What
> > > > > > can
> > > > > > > > > cause
> > > > > > > > > > a
> > > > > > > > > > >>> > > hanging
> > > > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > > > >>> > > > > > > > >> > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> completes
> > > > > (either
> > > > > > > > aborts
> > > > > > > > > > or
> > > > > > > > > > >>> > > > commits) a
> > > > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > before
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> successful
> > > ack
> > > > > on
> > > > > > > > > messages
> > > > > > > > > > >>> > > published
> > > > > > > > > > >>> > > > > in
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> same
> > > > > > > > > > >>> > > > > > > > >> > > txn.
> > > > > > > > > > >>> > > > > > > > >> > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible
> > for
> > > > the
> > > > > > > > delayed
> > > > > > > > > > >>> message
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > be
> > > > > > > > > > >>> > > > > > > > >> appended
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker,
> > > > causing
> > > > > a
> > > > > > > > > > >>> transaction to
> > > > > > > > > > >>> > > > hang.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A
> similar
> > > > issue
> > > > > > (not
> > > > > > > > > > >>> mentioned
> > > > > > > > > > >>> > in
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > >> motivation)
> > > > > > > > > > >>> > > > > > > > >> > > > could
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > happen
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker
> in
> > > the
> > > > > > > > > > coordinator's
> > > > > > > > > > >>> log.
> > > > > > > > > > >>> > > For
> > > > > > > > > > >>> > > > > > > > example,
> > > > > > > > > > >>> > > > > > > > >> > it's
> > > > > > > > > > >>> > > > > > > > >> > > > > > possible
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > EndTxnRequest
> > > > to
> > > > > > be
> > > > > > > > > > delayed
> > > > > > > > > > >>> on
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > > > >> coordinator.
> > > > > > > > > > >>> > > > > > > > >> > By
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > time
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > EndTxnRequest
> > > > is
> > > > > > > > > > processed,
> > > > > > > > > > >>> it's
> > > > > > > > > > >>> > > > > > possible
> > > > > > > > > > >>> > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > txn
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> completed
> > > and
> > > > a
> > > > > > new
> > > > > > > > txn
> > > > > > > > > > has
> > > > > > > > > > >>> > > started.
> > > > > > > > > > >>> > > > > > > > >> Currently,
> > > > > > > > > > >>> > > > > > > > >> > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped
> on
> > > > every
> > > > > > txn,
> > > > > > > > the
> > > > > > > > > > >>> delayed
> > > > > > > > > > >>> > > > > > > > EndTxnRequest
> > > > > > > > > > >>> > > > > > > > >> > will
> > > > > > > > > > >>> > > > > > > > >> > > > add
> > > > > > > > > > >>> > > > > > > > >> > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare
> > > marker
> > > > > > (and
> > > > > > > > > > >>> eventually a
> > > > > > > > > > >>> > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> marker)
> > > > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't
> > cause
> > > > the
> > > > > > > > > > transaction
> > > > > > > > > > >>> to
> > > > > > > > > > >>> > > hang,
> > > > > > > > > > >>> > > > > but
> > > > > > > > > > >>> > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> will
> > > > > > > > > > >>> > > > > > > > >> > > > break
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The
> > proposal
> > > > in
> > > > > > this
> > > > > > > > KIP
> > > > > > > > > > >>> will
> > > > > > > > > > >>> > > > address
> > > > > > > > > > >>> > > > > > this
> > > > > > > > > > >>> > > > > > > > >> issue
> > > > > > > > > > >>> > > > > > > > >> > > too.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.
> > > > "However, I
> > > > > > was
> > > > > > > > > > >>> writing it
> > > > > > > > > > >>> > so
> > > > > > > > > > >>> > > > > that
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > distinguish
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old
> > clients
> > > > > where
> > > > > > we
> > > > > > > > > don't
> > > > > > > > > > >>> have
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > ability
> > > > > > > > > > >>> > > > > > > > do
> > > > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > operation
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients
> > that
> > > > > can.
> > > > > > > (Old
> > > > > > > > > > >>> clients
> > > > > > > > > > >>> > > don't
> > > > > > > > > > >>> > > > > > bump
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for
> > sure
> > > > the
> > > > > > > write
> > > > > > > > > > >>> belongs
> > > > > > > > > > >>> > to
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > given
> > > > > > > > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I
> am
> > > > > > wondering
> > > > > > > > why
> > > > > > > > > > we
> > > > > > > > > > >>> need
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > > > > >> distinguish
> > > > > > > > > > >>> > > > > > > > >> > > > whether
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written
> by
> > > the
> > > > > old
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > >>> new
> > > > > > > > > > >>> > > > client.
> > > > > > > > > > >>> > > > > > > Could
> > > > > > > > > > >>> > > > > > > > >> you
> > > > > > > > > > >>> > > > > > > > >> > > > > describe
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > what
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > differently
> > > if
> > > > > we
> > > > > > > know
> > > > > > > > > the
> > > > > > > > > > >>> > marker
> > > > > > > > > > >>> > > is
> > > > > > > > > > >>> > > > > > > written
> > > > > > > > > > >>> > > > > > > > >> by
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > client?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If
> > we
> > > do
> > > > > > need
> > > > > > > a
> > > > > > > > > way
> > > > > > > > > > to
> > > > > > > > > > >>> > > > > distinguish
> > > > > > > > > > >>> > > > > > > > >> whether
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old
> > and
> > > > the
> > > > > > new
> > > > > > > > > > client.
> > > > > > > > > > >>> > Would
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > > be
> > > > > > > > > > >>> > > > > > > > >> simpler
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > > just
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field
> > > instead
> > > > of
> > > > > > > > > > indirectly
> > > > > > > > > > >>> > > through
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> previous
> > > > > > > > > > >>> > > > > > > > >> > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3
> It's
> > > not
> > > > > > clear
> > > > > > > > to
> > > > > > > > > me
> > > > > > > > > > >>> why
> > > > > > > > > > >>> > we
> > > > > > > > > > >>> > > > only
> > > > > > > > > > >>> > > > > > add
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > complete
> > > > > > marker,
> > > > > > > > but
> > > > > > > > > > >>> not in
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > marker.
> > > > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > want
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether
> a
> > > > marker
> > > > > > is
> > > > > > > > > > written
> > > > > > > > > > >>> by
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> or
> > > > > > > > > > >>> > > > > > > > >> > > not,
> > > > > > > > > > >>> > > > > > > > >> > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do
> this
> > > > > > > > consistently
> > > > > > > > > > for
> > > > > > > > > > >>> all
> > > > > > > > > > >>> > > > > markers.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4
> What
> > > > about
> > > > > > the
> > > > > > > > > > >>> > > > > TransactionLogValue
> > > > > > > > > > >>> > > > > > > > >> record
> > > > > > > > > > >>> > > > > > > > >> > > > > > > representing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state?
> > > Should
> > > > we
> > > > > > > also
> > > > > > > > > > >>> > distinguish
> > > > > > > > > > >>> > > > > > whether
> > > > > > > > > > >>> > > > > > > > it's
> > > > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > > > >>> > > > > > > > >> > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > old
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new
> > client?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In
> > the
> > > > > > overflow
> > > > > > > > > case,
> > > > > > > > > > >>> it's
> > > > > > > > > > >>> > > > still
> > > > > > > > > > >>> > > > > > not
> > > > > > > > > > >>> > > > > > > > >> clear
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > me
> > > > > > > > > > >>> > > > > > > > >> > > > > why
> > > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous
> > > > produce
> > > > > > Id
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>> > prepare
> > > > > > > > > > >>> > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> while
> > > > > > > > > > >>> > > > > > > > >> > > > writing
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the
> > > > complete
> > > > > > > > marker.
> > > > > > > > > > You
> > > > > > > > > > >>> > > > mentioned
> > > > > > > > > > >>> > > > > > that
> > > > > > > > > > >>> > > > > > > > >> it's
> > > > > > > > > > >>> > > > > > > > >> > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could
> > > > > downgrade
> > > > > > > > with
> > > > > > > > > > >>> either
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> marker
> > > > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In
> either
> > > > case,
> > > > > > the
> > > > > > > > > > >>> downgraded
> > > > > > > > > > >>> > > > > > coordinator
> > > > > > > > > > >>> > > > > > > > >> should
> > > > > > > > > > >>> > > > > > > > >> > > see
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> (probably
> > > the
> > > > > > > previous
> > > > > > > > > > >>> produce
> > > > > > > > > > >>> > > Id),
> > > > > > > > > > >>> > > > > > right?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed,
> > Dec
> > > > 20,
> > > > > > 2023
> > > > > > > > at
> > > > > > > > > > >>> 6:00 PM
> > > > > > > > > > >>> > > > > Justine
> > > > > > > > > > >>> > > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey
> Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks
> > for
> > > > > > taking
> > > > > > > a
> > > > > > > > > look
> > > > > > > > > > >>> at
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > KIP
> > > > > > > > > > >>> > > > > > > again.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100.
> For
> > > the
> > > > > > epoch
> > > > > > > > > > >>> overflow
> > > > > > > > > > >>> > > case,
> > > > > > > > > > >>> > > > > only
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > will
> > > > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > max
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps
> > the
> > > > > > behavior
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >>> rest
> > > > > > > > > > >>> > > of
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> markers
> > > > > > > > > > >>> > > > > > > > >> > > where
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > last
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch
> of
> > > the
> > > > > > > > > transaction
> > > > > > > > > > >>> > > records +
> > > > > > > > > > >>> > > > > 1.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101.
> You
> > > are
> > > > > > > correct
> > > > > > > > > > that
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > don't
> > > > > > > > > > >>> > > > > > need
> > > > > > > > > > >>> > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > write
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the
> > > same.
> > > > > > > > However,
> > > > > > > > > I
> > > > > > > > > > >>> was
> > > > > > > > > > >>> > > > writing
> > > > > > > > > > >>> > > > > it
> > > > > > > > > > >>> > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> that
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old
> > > clients
> > > > > > where
> > > > > > > we
> > > > > > > > > > don't
> > > > > > > > > > >>> > have
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > ability
> > > > > > > > > > >>> > > > > > > > >> do
> > > > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > operation
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> clients
> > > that
> > > > > > can.
> > > > > > > > (Old
> > > > > > > > > > >>> clients
> > > > > > > > > > >>> > > > don't
> > > > > > > > > > >>> > > > > > > bump
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say
> for
> > > sure
> > > > > the
> > > > > > > > write
> > > > > > > > > > >>> belongs
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > given
> > > > > > > > > > >>> > > > > > > > >> > > > > > transaction).
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn
> > > > request
> > > > > > > from
> > > > > > > > a
> > > > > > > > > > new
> > > > > > > > > > >>> > > client,
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> fill
> > > > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > field.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > We
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> guarantee
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that
> any
> > > > > EndTxn
> > > > > > > > > requests
> > > > > > > > > > >>> with
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > same
> > > > > > > > > > >>> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > are
> > > > > > > > > > >>> > > > > > > > >> > > > from
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the
> same
> > > > > > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102.
> In
> > > > > prepare
> > > > > > > > phase,
> > > > > > > > > > we
> > > > > > > > > > >>> have
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had.
> It
> > is
> > > > the
> > > > > > > > > producer
> > > > > > > > > > >>> ID and
> > > > > > > > > > >>> > > > epoch
> > > > > > > > > > >>> > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> are
> > > > > > > > > > >>> > > > > > > > >> > on
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase,
> > we
> > > > stay
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > >>> unless
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > overflow
> > > > > > > > > > >>> > > > > > > > >> > > > > case.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set
> the
> > > > > producer
> > > > > > > ID
> > > > > > > > to
> > > > > > > > > > >>> the new
> > > > > > > > > > >>> > > one
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > > >> generated
> > > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > complete.
> > > > This
> > > > > > is
> > > > > > > > for
> > > > > > > > > > >>> > downgrade
> > > > > > > > > > >>> > > > > > > > >> compatibility.
> > > > > > > > > > >>> > > > > > > > >> > > The
> > > > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety
> > > > guards
> > > > > > for
> > > > > > > > > > retries
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > > > > > failovers.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In
> > prepare
> > > > > phase
> > > > > > > for
> > > > > > > > > > epoch
> > > > > > > > > > >>> > > > overflow
> > > > > > > > > > >>> > > > > > case
> > > > > > > > > > >>> > > > > > > > >> only
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > > store
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID.
> This
> > > is
> > > > > for
> > > > > > > the
> > > > > > > > > case
> > > > > > > > > > >>> where
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > > reload
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> prepare
> > > > state.
> > > > > > > Once
> > > > > > > > > the
> > > > > > > > > > >>> > > > transaction
> > > > > > > > > > >>> > > > > is
> > > > > > > > > > >>> > > > > > > > >> > committed,
> > > > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > use
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the
> > > > client
> > > > > > > > already
> > > > > > > > > is
> > > > > > > > > > >>> > using.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In
> > commit
> > > > > phase,
> > > > > > > we
> > > > > > > > > > store
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > previous
> > > > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I
> think
> > it
> > > > is
> > > > > > > easier
> > > > > > > > > to
> > > > > > > > > > >>> think
> > > > > > > > > > >>> > of
> > > > > > > > > > >>> > > > it
> > > > > > > > > > >>> > > > > as
> > > > > > > > > > >>> > > > > > > > just
> > > > > > > > > > >>> > > > > > > > >> how
> > > > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > were
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > epoch
> > > > > > before,
> > > > > > > > with
> > > > > > > > > > >>> some
> > > > > > > > > > >>> > > extra
> > > > > > > > > > >>> > > > > > > > bookeeping
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > > edge
> > > > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged
> > > > fields.
> > > > > > We
> > > > > > > > have
> > > > > > > > > > to
> > > > > > > > > > >>> do
> > > > > > > > > > >>> > it
> > > > > > > > > > >>> > > > this
> > > > > > > > > > >>> > > > > > way
> > > > > > > > > > >>> > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > with
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103.
> > Next
> > > > > > producer
> > > > > > > > ID
> > > > > > > > > is
> > > > > > > > > > >>> for
> > > > > > > > > > >>> > > > prepare
> > > > > > > > > > >>> > > > > > > > status
> > > > > > > > > > >>> > > > > > > > >> and
> > > > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after
> > > > > complete.
> > > > > > > The
> > > > > > > > > > >>> reason why
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > need
> > > > > > > > > > >>> > > > > > > two
> > > > > > > > > > >>> > > > > > > > >> > > separate
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > backwards
> > > > > > > > > compatibility.
> > > > > > > > > > >>> We
> > > > > > > > > > >>> > need
> > > > > > > > > > >>> > > > to
> > > > > > > > > > >>> > > > > > keep
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > semantics
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > non-tagged
> > > > > field
> > > > > > > in
> > > > > > > > > case
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > > > downgrade.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104.
> We
> > > set
> > > > > the
> > > > > > > > fields
> > > > > > > > > > as
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > do
> > > > > > > > > > >>> > > in
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > transactional
> > > > > > > > > > >>> > > > > > > > >> > > > > > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do
> this
> > > for
> > > > > > > > > > compatibility
> > > > > > > > > > >>> --
> > > > > > > > > > >>> > if
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > > > > >> downgrade,
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > will
> > > > > > > > > > >>> > > > > > > > >> > > > > > > only
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > non-tagged
> > > > > > fields)
> > > > > > > > It
> > > > > > > > > > >>> will be
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > old
> > > > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > > > >>> > > > > > > > >> > ID
> > > > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > max
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope
> > this
> > > > > helps.
> > > > > > > Let
> > > > > > > > > me
> > > > > > > > > > >>> know
> > > > > > > > > > >>> > if
> > > > > > > > > > >>> > > > you
> > > > > > > > > > >>> > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > further
> > > > > > > > > > >>> > > > > > > > >> > > > > > > questions.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On
> Wed,
> > > Dec
> > > > > 20,
> > > > > > > 2023
> > > > > > > > > at
> > > > > > > > > > >>> > 3:33 PM
> > > > > > > > > > >>> > > > Jun
> > > > > > > > > > >>> > > > > > Rao
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi,
> > > > Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It
> > seems
> > > > > that
> > > > > > > you
> > > > > > > > > have
> > > > > > > > > > >>> made
> > > > > > > > > > >>> > > some
> > > > > > > > > > >>> > > > > > > changes
> > > > > > > > > > >>> > > > > > > > >> to
> > > > > > > > > > >>> > > > > > > > >> > > > KIP-890
> > > > > > > > > > >>> > > > > > > > >> > > > > > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > particular,
> > > > > we
> > > > > > > are
> > > > > > > > > > >>> changing
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > format
> > > > > > > > > > >>> > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> TransactionLogValue.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > comments
> > > > > > related
> > > > > > > > to
> > > > > > > > > > >>> that.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100.
> > > Just
> > > > to
> > > > > > be
> > > > > > > > > clear.
> > > > > > > > > > >>> The
> > > > > > > > > > >>> > > > > overflow
> > > > > > > > > > >>> > > > > > > case
> > > > > > > > > > >>> > > > > > > > >> > (i.e.
> > > > > > > > > > >>> > > > > > > > >> > > > > when a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > generated)
> > > > > is
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > >>> > current
> > > > > > > > > > >>> > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> equals
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > > max
> > > > > > > > > > >>> > > > > > > > >> > > > > -
> > > > > > > > > > >>> > > > > > > > >> > > > > > 1
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101.
> > For
> > > > the
> > > > > > > "not
> > > > > > > > > > epoch
> > > > > > > > > > >>> > > > overflow"
> > > > > > > > > > >>> > > > > > > case,
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > write
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> tagged
> > > > field
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>> complete
> > > > > > > > > > >>> > > > > phase.
> > > > > > > > > > >>> > > > > > Do
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > need
> > > > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > do
> > > > > > > > > > >>> > > > > > > > >> > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > doesn't
> > > > > change
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > > >>> case?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102.
> > It
> > > > > seems
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > >>> > meaning
> > > > > > > > > > >>> > > > for
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > ProducerId/ProducerEpoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > TransactionLogValue
> > > > > > > > > > >>> changes
> > > > > > > > > > >>> > > > > > depending
> > > > > > > > > > >>> > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > > > TransactionStatus
> > > > > > > > > > is
> > > > > > > > > > >>> > > > ongoing,
> > > > > > > > > > >>> > > > > > they
> > > > > > > > > > >>> > > > > > > > >> > > represent
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > current
> > > > > > > > > > >>> ProducerEpoch.
> > > > > > > > > > >>> > > When
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > > > > > > > >>> > > > > > > > >> > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> PrepareCommit/PrepareAbort,
> > > > > > > > > > >>> > > they
> > > > > > > > > > >>> > > > > > > > represent
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > > > > > > > ProducerEpoch.
> > > > > > > > > > >>> When the
> > > > > > > > > > >>> > > > > > > > >> > TransactionStatus
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > further
> > > > > depend
> > > > > > > on
> > > > > > > > > > >>> whether
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> overflows
> > > > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > > > >>> > > > > > > > >> > > > > not.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > overflow,
> > > > > they
> > > > > > > > > > represent
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > ProducerEpoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> (max).
> > > > > > > Otherwise,
> > > > > > > > > they
> > > > > > > > > > >>> > > represent
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > newly
> > > > > > > > > > >>> > > > > > > > >> > > > > generated
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > ProducerEpoch
> > > > > > of
> > > > > > > > 0.
> > > > > > > > > Is
> > > > > > > > > > >>> that
> > > > > > > > > > >>> > > > right?
> > > > > > > > > > >>> > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> seems
> > > > > > > > > > >>> > > > > > > > >> > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > easy
> > > > > > > > > > >>> > > > > > > > >> > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we
> > > provide
> > > > > > some
> > > > > > > > > > examples
> > > > > > > > > > >>> > like
> > > > > > > > > > >>> > > > what
> > > > > > > > > > >>> > > > > > > Artem
> > > > > > > > > > >>> > > > > > > > >> has
> > > > > > > > > > >>> > > > > > > > >> > > done
> > > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > considered a
> > > > > > > > simpler
> > > > > > > > > > >>> design
> > > > > > > > > > >>> > > > where
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> ProducerId/ProducerEpoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > represent
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > same
> > > > > value
> > > > > > > > (e.g.
> > > > > > > > > > >>> for the
> > > > > > > > > > >>> > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > transaction)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > TransactionStatus
> > > > > > > > > and
> > > > > > > > > > >>> epoch
> > > > > > > > > > >>> > > > > > overflow?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103.
> > > It's
> > > > > not
> > > > > > > > clear
> > > > > > > > > to
> > > > > > > > > > >>> me
> > > > > > > > > > >>> > why
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > > need
> > > > > > > > > > >>> > > > > > > 3
> > > > > > > > > > >>> > > > > > > > >> > fields:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > PrevProducerId,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > NextProducerId.
> > > > > > > > > Could
> > > > > > > > > > we
> > > > > > > > > > >>> > just
> > > > > > > > > > >>> > > > have
> > > > > > > > > > >>> > > > > > > > >> ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104.
> > For
> > > > > > > > > > >>> > > WriteTxnMarkerRequests,
> > > > > > > > > > >>> > > > > if
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we
> set
> > > the
> > > > > > > > > producerId
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > > > >> producerEpoch?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> >
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Calvin,

Thanks for the reply. The changes look good to me.

Jun

On Tue, Feb 11, 2025 at 3:27 PM Calvin Liu <ca...@confluent.io.invalid>
wrote:

> Hi Jun,
> Thanks for the feedback, will update the name as suggested.
> The default values are
> add.partitions.to.txn.retry.backoff.ms=20
> -- This is because the client side AddPartitionsToTxn retry backoff is
> 20ms.
> add.partitions.to.txn.retry.backoff.max.ms=100
> -- We want this value to be small to avoid client side timeout and retry,
> but also long enough for the markers to propagate.
>
>
> On Tue, Feb 11, 2025 at 2:36 PM Jun Rao <ju...@confluent.io.invalid> wrote:
>
> > Hi, Calvin,
> >
> > Thanks for the update. The change looks good to me. Some minor comments.
> >
> > For retry timeout, our convention is to use retry.backoff.max.ms and
> > retry.backoff.ms. So, to be consistent, the new configurations should be
> > add.partitions.to.txn.retry.backoff.max.ms and
> > add.partitions.to.txn.retry.backoff.ms.
> >
> > Also, could you specify the defaults for these two values?
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> >
> > On Mon, Feb 10, 2025 at 2:45 PM Calvin Liu <ca...@confluent.io.invalid>
> > wrote:
> >
> > > Hey team,
> > > In order to address the performance issue in the
> > > https://issues.apache.org/jira/browse/KAFKA-18654. We are making a
> > change
> > > to retry the AddPartitionToTxn requests on the server side when hitting
> > the
> > > CONCURRENT_TRANSACTION error. This error is common after the
> transaction
> > is
> > > committed but before the markers are fully propagated to all the
> involved
> > > partitions. With the server side retry, the client can avoid the
> frequent
> > > and expensive retry for the produce request. In order to better control
> > the
> > > retry behavior, we proposed to add 2 following configs.
> > >
> > >    - *add.partitions.to.txn.max.timeout.ms
> > >    <http://add.partitions.to.txn.max.timeout.ms>* defines the maximum
> > > retry
> > >    timeout when the server attempts to add the partition to the
> > > transaction.
> > >    - *add.partitions.to.txn.retry.backoff.ms
> > >    <http://add.partitions.to.txn.retry.backoff.ms>* defines how
> > frequently
> > >    the server will retry the AddPartitionToTxn.
> > >
> > > Thanks!
> > >
> > > On Thu, Nov 14, 2024 at 1:52 PM Justine Olshan
> > > <jo...@confluent.io.invalid>
> > > wrote:
> > >
> > > > Hey folks, just wanted to share another quick update with KIP-890.
> > > >
> > > > Making the change to bump epoch after every transaction means that we
> > no
> > > > longer need to call InitProducerId during the life of the producer to
> > > fence
> > > > requests from previous transactions. With KIP-890 part 2, the new
> > client
> > > > will no longer call InitProducerId except on startup to fence a
> > previous
> > > > instance.
> > > >
> > > > With this change, some other calls to InitProducerId were inspected
> > > > including the call after receiving an InvalidPidMappingException.
> This
> > > > exception was changed to abortable as part of KIP-360: Improve
> > > reliability
> > > > of idempotent/transactional producer
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820
> > > > >.
> > > > However, this change means that we can violate EOS guarantees. As an
> > > > example:
> > > >
> > > > Consider an application that is copying data from one partition to
> > > another
> > > >
> > > >    - Application instance A processes to offset 4
> > > >    - Application instance B comes up and fences application instance
> A
> > > >    - Application instance B processes to offset 5
> > > >    - Application instances A and B are idle for
> > > > transaction.id.expiration.ms,
> > > >    transaction id expires on server
> > > >    - Application instance A attempts to process offset 5 (since in
> its
> > > >    view, that is next) -- if we recover from invalid pid mapping, we
> > can
> > > >    duplicate this processing
> > > >
> > > > Thus, INVALID_PID_MAPPING should be fatal to the producer and as part
> > of
> > > > KIP-890 the new client in 4.0 will treat the exception as fatal.
> > > >
> > > > This is consistent with KIP-1050: Consistent error handling for
> > > > Transactions
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions
> > > > >
> > > > where
> > > > errors that are fatal to the producer are in the "application
> > > recoverable"
> > > > category. This is a grouping that indicates to the client that the
> > > producer
> > > > needs to restart and recovery on the application side is necessary.
> > > > KIP-1050 is approved so we are consistent with that decision.
> > > >
> > > > Let me know if there are any questions. I've updated the KIP to
> reflect
> > > > this change.
> > > >
> > > > Justine
> > > >
> > > > On Wed, Jul 10, 2024 at 3:34 PM Jun Rao <ju...@confluent.io.invalid>
> > > wrote:
> > > >
> > > > > Sounds good, Justine. It would be useful to document that in the
> KIP.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Jul 10, 2024 at 2:59 PM Justine Olshan
> > > > > <jo...@confluent.io.invalid>
> > > > > wrote:
> > > > >
> > > > > > The client will send the newest EndTxn request version if and
> only
> > if
> > > > > both
> > > > > > the client and the server support kip-890 part 2.
> > > > > > We set the value in the record based on the EndTxn version.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <jun@confluent.io.invalid
> >
> > > > > wrote:
> > > > > >
> > > > > > > Hi, Justine,
> > > > > > >
> > > > > > > Thanks for the reply.
> > > > > > >
> > > > > > > 120. If the broker sends TV Y for the finalized version in
> > > > > > > ApiVersionResponse, but the client doesn't support Y, how does
> > the
> > > > > broker
> > > > > > > know the TV that the client supports?
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Jun,
> > > > > > > >
> > > > > > > > No worries. Work on this KIP has been blocked for a bit
> anyways
> > > --
> > > > > > > catching
> > > > > > > > up and rereading what I wrote :)
> > > > > > > >
> > > > > > > > 120. ClientTransactionProtocolVersion is the transaction
> > version
> > > as
> > > > > > > defined
> > > > > > > > by the highest transaction version (feature version value)
> > > > supported
> > > > > by
> > > > > > > the
> > > > > > > > client and the server. This works by the broker sending an
> > > > > > > > ApiVersionsRequest to the client with the finalized version.
> > > > Assuming
> > > > > > > > kip-890 part 2 is enabled by transaction version Y, if this
> > > request
> > > > > > > > contains finalized version Y and the client has the logic to
> > set
> > > > this
> > > > > > > > field, it will set Y. If the server has Y - 1 (kip 890 part 2
> > not
> > > > > > enable)
> > > > > > > > the client will send Y - 1, even though the client has the
> > > ability
> > > > to
> > > > > > > > support kip-890 part 2.
> > > > > > > >
> > > > > > > > 121. You are correct that this is not needed. However,
> > currently
> > > > that
> > > > > > > field
> > > > > > > > is already being set in memory -- just not written to disk. I
> > > think
> > > > > it
> > > > > > is
> > > > > > > > ok to write it to disk though. Let me know if you think
> > > otherwise.
> > > > > > > >
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao
> > <jun@confluent.io.invalid
> > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Justine,
> > > > > > > > >
> > > > > > > > > Thanks for the update and sorry for the late reply.
> > > > > > > > >
> > > > > > > > > 120. I am wondering what value is used for
> > > > > > > > > ClientTransactionProtocolVersion. Is it the version of the
> > > > > > > EndTxnRequest?
> > > > > > > > >
> > > > > > > > > 121. Earlier, you made the change to set lastProducerId in
> > > > PREPARE
> > > > > to
> > > > > > > > > indicate that the marker is written for the new client.
> With
> > > the
> > > > > new
> > > > > > > > > ClientTransactionProtocolVersion field, it seems this is no
> > > > longer
> > > > > > > > > necessary.
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi there -- another update!
> > > > > > > > > >
> > > > > > > > > > When looking into the implementation for the safe epoch
> > > bumps I
> > > > > > > > realized
> > > > > > > > > > that we are already populating previousProducerID in
> memory
> > > as
> > > > > part
> > > > > > > of
> > > > > > > > > > KIP-360.
> > > > > > > > > > If we are to start using flexible fields, it is better to
> > > > always
> > > > > > use
> > > > > > > > this
> > > > > > > > > > information and have an explicit (tagged) field to
> indicate
> > > > > whether
> > > > > > > the
> > > > > > > > > > client supports KIP-890 part 2.
> > > > > > > > > >
> > > > > > > > > > I've included the extra field and how it is set in the
> KIP.
> > > > I've
> > > > > > also
> > > > > > > > > > updated the KIP to explain that we will be setting the
> > tagged
> > > > > > fields
> > > > > > > > when
> > > > > > > > > > they are available for all transitions.
> > > > > > > > > >
> > > > > > > > > > Finally, I added clearer text about the transaction
> > protocol
> > > > > > versions
> > > > > > > > > > included with this KIP. 1 for flexible transaction state
> > > > records
> > > > > > and
> > > > > > > 2
> > > > > > > > > for
> > > > > > > > > > KIP-890 part 2 enablement.
> > > > > > > > > >
> > > > > > > > > > Justine
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <
> > > > > > jolshan@confluent.io
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey there -- small update to the KIP,
> > > > > > > > > > >
> > > > > > > > > > > The KIP mentioned introducing ABORTABLE_ERROR and
> bumping
> > > > > > > > > TxnOffsetCommit
> > > > > > > > > > > and Produce requests. I've changed the name in the KIP
> to
> > > > > > > > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > > > > > > > AbortableTransactionException to match the pattern we
> had
> > > for
> > > > > > other
> > > > > > > > > > errors.
> > > > > > > > > > > I also mentioned bumping all 6 transactional APIs so we
> > can
> > > > > > future
> > > > > > > > > > > proof/support the error on the client going forward.
> If a
> > > > > future
> > > > > > > > change
> > > > > > > > > > > wants to have an error scenario that requires us to
> abort
> > > the
> > > > > > > > > > transaction,
> > > > > > > > > > > we can rely on the 3.8+ clients to support it. We ran
> > into
> > > > > issues
> > > > > > > > > finding
> > > > > > > > > > > good/generic error codes that older clients could
> support
> > > > while
> > > > > > > > working
> > > > > > > > > > on
> > > > > > > > > > > this KIP, so this should help in the future.
> > > > > > > > > > >
> > > > > > > > > > > The features discussion is still ongoing in KIP-1022.
> > Will
> > > > > update
> > > > > > > > again
> > > > > > > > > > > here when that concludes.
> > > > > > > > > > >
> > > > > > > > > > > Justine
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> > > > > > > jolshan@confluent.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> I don't think AddPartitions is a good example since we
> > > > > currenly
> > > > > > > > don't
> > > > > > > > > > >> gate the version on TV or MV. (We only set a different
> > > flag
> > > > > > > > depending
> > > > > > > > > on
> > > > > > > > > > >> the TV)
> > > > > > > > > > >>
> > > > > > > > > > >> Even if we did want to gate it on TV, I think the idea
> > is
> > > to
> > > > > > move
> > > > > > > > away
> > > > > > > > > > >> from MV gating inter broker protocols. Ideally we can
> > get
> > > > to a
> > > > > > > state
> > > > > > > > > > where
> > > > > > > > > > >> MV is just used for metadata changes.
> > > > > > > > > > >>
> > > > > > > > > > >> I think some of this discussion might fit more with
> the
> > > > > feature
> > > > > > > > > version
> > > > > > > > > > >> KIP, so I can try to open that up soon. Until we
> settle
> > > > that,
> > > > > > some
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >> work in KIP-890 is blocked.
> > > > > > > > > > >>
> > > > > > > > > > >> Justine
> > > > > > > > > > >>
> > > > > > > > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao
> > > > > <jun@confluent.io.invalid
> > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> Hi, Justine,
> > > > > > > > > > >>>
> > > > > > > > > > >>> Thanks for the reply.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Since AddPartitions is an inter broker request, will
> > its
> > > > > > version
> > > > > > > be
> > > > > > > > > > gated
> > > > > > > > > > >>> only by TV or other features like MV too? For
> example,
> > if
> > > > we
> > > > > > need
> > > > > > > > to
> > > > > > > > > > >>> change
> > > > > > > > > > >>> the protocol for AddPartitions for reasons other than
> > txn
> > > > > > > > > verification
> > > > > > > > > > in
> > > > > > > > > > >>> the future, will the new version be gated by a new
> MV?
> > If
> > > > so,
> > > > > > > does
> > > > > > > > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > > > > > > > >>>
> > > > > > > > > > >>> Jun
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>> > One TV gates the flexible feature version (no rpcs
> > > > > involved,
> > > > > > > only
> > > > > > > > > the
> > > > > > > > > > >>> > transactional records that should only be gated by
> > TV)
> > > > > > > > > > >>> > Another TV gates the ability to turn on kip-890
> part
> > 2.
> > > > > This
> > > > > > > > would
> > > > > > > > > > >>> gate the
> > > > > > > > > > >>> > version of Produce and EndTxn (likely only used by
> > > > > > > transactions),
> > > > > > > > > and
> > > > > > > > > > >>> > specifies a flag in AddPartitionsToTxn though the
> > > version
> > > > > is
> > > > > > > > > already
> > > > > > > > > > >>> used
> > > > > > > > > > >>> > without TV.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > I think the only concern is the Produce request and
> > we
> > > > > could
> > > > > > > > > consider
> > > > > > > > > > >>> work
> > > > > > > > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Justine
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> > > > > > > <jun@confluent.io.invalid
> > > > > > > > >
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > > Hi, Justine,
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Which PRC/record protocols will TV guard? Going
> > > > forward,
> > > > > > will
> > > > > > > > > those
> > > > > > > > > > >>> > > PRC/record protocols only be guarded by TV and
> not
> > by
> > > > > other
> > > > > > > > > > features
> > > > > > > > > > >>> like
> > > > > > > > > > >>> > > MV?
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Thanks,
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Jun
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > > > > > > > >>> > <jolshan@confluent.io.invalid
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > wrote:
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > > Hi Jun,
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > Sorry I think I misunderstood your question or
> > > > answered
> > > > > > > > > > >>> incorrectly.
> > > > > > > > > > >>> > The
> > > > > > > > > > >>> > > TV
> > > > > > > > > > >>> > > > version should ideally be fully independent
> from
> > > MV.
> > > > > > > > > > >>> > > > At least for the changes I proposed, TV should
> > not
> > > > > affect
> > > > > > > MV
> > > > > > > > > and
> > > > > > > > > > MV
> > > > > > > > > > >>> > > should
> > > > > > > > > > >>> > > > not affect TV/
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > I think if we downgrade TV, only that feature
> > > should
> > > > > > > > downgrade.
> > > > > > > > > > >>> > Likewise
> > > > > > > > > > >>> > > > the same with MV. The finalizedFeatures should
> > just
> > > > > > reflect
> > > > > > > > the
> > > > > > > > > > >>> feature
> > > > > > > > > > >>> > > > downgrade we made.
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > I also plan to write a new KIP for managing the
> > > disk
> > > > > > format
> > > > > > > > and
> > > > > > > > > > >>> upgrade
> > > > > > > > > > >>> > > > tool as we will need new flags to support these
> > > > > features.
> > > > > > > > That
> > > > > > > > > > >>> should
> > > > > > > > > > >>> > > help
> > > > > > > > > > >>> > > > clarify some things.
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > Justine
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > > > > > > > <jun@confluent.io.invalid
> > > > > > > > > > >
> > > > > > > > > > >>> > > wrote:
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > > Hi, Justine,
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > Thanks for the reply.
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > So, if we downgrade TV, we could implicitly
> > > > downgrade
> > > > > > > > another
> > > > > > > > > > >>> feature
> > > > > > > > > > >>> > > > (say
> > > > > > > > > > >>> > > > > MV) that has dependency (e.g. RPC). What
> would
> > we
> > > > > > return
> > > > > > > > for
> > > > > > > > > > >>> > > > > FinalizedFeatures for MV in
> ApiVersionsResponse
> > > in
> > > > > that
> > > > > > > > case?
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > Thanks,
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > Jun
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > > > > > > > >>> > > > <jolshan@confluent.io.invalid
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > wrote:
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > Yes, the idea is that if we downgrade TV
> > > > > (transaction
> > > > > > > > > > version)
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > will
> > > > > > > > > > >>> > > > > stop
> > > > > > > > > > >>> > > > > > using the add partitions to txn
> optimization
> > > and
> > > > > stop
> > > > > > > > > writing
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > flexible
> > > > > > > > > > >>> > > > > > feature version of the log.
> > > > > > > > > > >>> > > > > > In the compatibility section I included
> some
> > > > > > > explanations
> > > > > > > > > on
> > > > > > > > > > >>> how
> > > > > > > > > > >>> > this
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > done.
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > Thanks,
> > > > > > > > > > >>> > > > > > Justine
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > > > > > > > >>> <ju...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > wrote:
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > Thanks for the update.
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > If we ever downgrade the transaction
> > feature,
> > > > any
> > > > > > > > feature
> > > > > > > > > > >>> > depending
> > > > > > > > > > >>> > > > on
> > > > > > > > > > >>> > > > > > > changes on top of those RPC/record
> > > > > > > > > > >>> > > > > > >
> > > (AddPartitionsToTxnRequest/TransactionLogValue)
> > > > > > > changes
> > > > > > > > > > made
> > > > > > > > > > >>> in
> > > > > > > > > > >>> > > > KIP-890
> > > > > > > > > > >>> > > > > > > will be automatically downgraded too?
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > Jun
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine
> > > Olshan
> > > > > > > > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > wrote:
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > I wanted to get back to you about your
> > > > > questions
> > > > > > > > about
> > > > > > > > > > >>> MV/IBP.
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > Looking at the options, I think it
> makes
> > > the
> > > > > most
> > > > > > > > sense
> > > > > > > > > > to
> > > > > > > > > > >>> > > create a
> > > > > > > > > > >>> > > > > > > > separate feature for transactions and
> use
> > > > that
> > > > > to
> > > > > > > > > version
> > > > > > > > > > >>> gate
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > features
> > > > > > > > > > >>> > > > > > > > we need to version gate (flexible
> > > > transactional
> > > > > > > state
> > > > > > > > > > >>> records
> > > > > > > > > > >>> > and
> > > > > > > > > > >>> > > > > using
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > new protocol)
> > > > > > > > > > >>> > > > > > > > I've updated the KIP to include this
> > > change.
> > > > > > > > Hopefully
> > > > > > > > > > >>> that's
> > > > > > > > > > >>> > > > > > everything
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > need for this KIP :)
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine
> > > > Olshan
> > > > > <
> > > > > > > > > > >>> > > > jolshan@confluent.io
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > > > > Thanks Jun,
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > I will update the KIP with the prev
> > field
> > > > for
> > > > > > > > prepare
> > > > > > > > > > as
> > > > > > > > > > >>> > well.
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > PREPARE
> > > > > > > > > > >>> > > > > > > > > producerId: x
> > > > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged
> > field): x
> > > > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > or z
> > > > if
> > > > > y
> > > > > > > will
> > > > > > > > > > >>> overflow
> > > > > > > > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > COMPLETE
> > > > > > > > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged
> > field): x
> > > > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > > > > > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we
> > > overflowed
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > Thanks again,
> > > > > > > > > > >>> > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun
> Rao
> > > > > > > > > > >>> > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > > > > > > > >>> > > > > > > > >> (1) My point was that the
> coordinator
> > > > could
> > > > > > fail
> > > > > > > > > right
> > > > > > > > > > >>> after
> > > > > > > > > > >>> > > > > writing
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> prepare marker. When the new txn
> > > > coordinator
> > > > > > > > > generates
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > complete
> > > > > > > > > > >>> > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> after the failover, it needs some
> > field
> > > > from
> > > > > > the
> > > > > > > > > > prepare
> > > > > > > > > > >>> > > marker
> > > > > > > > > > >>> > > > to
> > > > > > > > > > >>> > > > > > > > >> determine whether it's written by
> the
> > > new
> > > > > > > client.
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> (2) The changing of the behavior
> > sounds
> > > > good
> > > > > > to
> > > > > > > > me.
> > > > > > > > > We
> > > > > > > > > > >>> only
> > > > > > > > > > >>> > > want
> > > > > > > > > > >>> > > > > to
> > > > > > > > > > >>> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> success if the prepare state is
> > written
> > > by
> > > > > the
> > > > > > > new
> > > > > > > > > > >>> client.
> > > > > > > > > > >>> > So,
> > > > > > > > > > >>> > > > in
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> non-overflow case, it seems that we
> > also
> > > > > need
> > > > > > > sth
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >>> > > prepare
> > > > > > > > > > >>> > > > > > > marker
> > > > > > > > > > >>> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> tell us whether it's written by the
> > new
> > > > > > client.
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> 112. Thanks for the explanation.
> That
> > > > sounds
> > > > > > > good
> > > > > > > > to
> > > > > > > > > > me.
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> Jun
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM
> > Justine
> > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> <jo...@confluent.io.invalid>
> wrote:
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >> > 101.3 I realized that I actually
> > have
> > > > two
> > > > > > > > > questions.
> > > > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> > > need
> > > > to
> > > > > > > write
> > > > > > > > > the
> > > > > > > > > > >>> > > previous
> > > > > > > > > > >>> > > > > > > produce
> > > > > > > > > > >>> > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > tagged field in the end maker so
> > that
> > > we
> > > > > > know
> > > > > > > if
> > > > > > > > > the
> > > > > > > > > > >>> > marker
> > > > > > > > > > >>> > > is
> > > > > > > > > > >>> > > > > > from
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> new
> > > > > > > > > > >>> > > > > > > > >> > client. Since the end maker is
> > derived
> > > > > from
> > > > > > > the
> > > > > > > > > > >>> prepare
> > > > > > > > > > >>> > > > marker,
> > > > > > > > > > >>> > > > > > > should
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > write the previous produce Id in
> the
> > > > > prepare
> > > > > > > > > marker
> > > > > > > > > > >>> field
> > > > > > > > > > >>> > > too?
> > > > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > > > >>> > > > > > > > >> > we will lose this information when
> > > > > deriving
> > > > > > > the
> > > > > > > > > end
> > > > > > > > > > >>> > marker.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > The "previous" producer ID is in
> the
> > > > > normal
> > > > > > > > > producer
> > > > > > > > > > >>> ID
> > > > > > > > > > >>> > > field.
> > > > > > > > > > >>> > > > > So
> > > > > > > > > > >>> > > > > > > yes,
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > need it in prepare and that was
> > always
> > > > the
> > > > > > > plan.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I
> will
> > > > > > enumerate
> > > > > > > > the
> > > > > > > > > > >>> fields
> > > > > > > > > > >>> > and
> > > > > > > > > > >>> > > > add
> > > > > > > > > > >>> > > > > > > them
> > > > > > > > > > >>> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > > > > > > > >>> > > > > > > > >> > Say we have producer ID x and
> epoch
> > y.
> > > > > When
> > > > > > we
> > > > > > > > > > >>> overflow
> > > > > > > > > > >>> > > epoch
> > > > > > > > > > >>> > > > y
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > get
> > > > > > > > > > >>> > > > > > > > >> > producer ID Z.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > PREPARE
> > > > > > > > > > >>> > > > > > > > >> > producerId: x
> > > > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> > > field):
> > > > > > empty
> > > > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field):
> empty
> > > or
> > > > z
> > > > > > if y
> > > > > > > > > will
> > > > > > > > > > >>> > overflow
> > > > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > COMPLETE
> > > > > > > > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> > > field):
> > > > x
> > > > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field):
> empty
> > > > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we
> > > > overflowed
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > (2) In the prepare phase, if we
> > retry
> > > > and
> > > > > > see
> > > > > > > > > epoch
> > > > > > > > > > -
> > > > > > > > > > >>> 1 +
> > > > > > > > > > >>> > ID
> > > > > > > > > > >>> > > > in
> > > > > > > > > > >>> > > > > > last
> > > > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > > > >>> > > > > > > > >> > fields and are issuing the same
> > > command
> > > > > (ie
> > > > > > > > commit
> > > > > > > > > > not
> > > > > > > > > > >>> > > abort),
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> > success. The logic before KIP-890
> > > seems
> > > > to
> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > > > >> > in this case. Are we intentionally
> > > > making
> > > > > > this
> > > > > > > > > > change?
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > Hmm -- we would fence the producer
> > if
> > > > the
> > > > > > > epoch
> > > > > > > > is
> > > > > > > > > > >>> bumped
> > > > > > > > > > >>> > > and
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > get a
> > > > > > > > > > >>> > > > > > > > >> > lower epoch. Yes -- we are
> > > intentionally
> > > > > > > adding
> > > > > > > > > this
> > > > > > > > > > >>> to
> > > > > > > > > > >>> > > > prevent
> > > > > > > > > > >>> > > > > > > > fencing.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > 112. We already merged the code
> that
> > > > adds
> > > > > > the
> > > > > > > > > > >>> VerifyOnly
> > > > > > > > > > >>> > > field
> > > > > > > > > > >>> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which
> is
> > an
> > > > > inter
> > > > > > > > > broker
> > > > > > > > > > >>> > request.
> > > > > > > > > > >>> > > > It
> > > > > > > > > > >>> > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> that
> > > > > > > > > > >>> > > > > > > > >> > we didn't bump up the IBP for
> that.
> > Do
> > > > you
> > > > > > > know
> > > > > > > > > why?
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > We no longer need IBP for all
> > > > interbroker
> > > > > > > > requests
> > > > > > > > > > as
> > > > > > > > > > >>> > > > > ApiVersions
> > > > > > > > > > >>> > > > > > > > should
> > > > > > > > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > > > > > > > >>> > > > > > > > >> > We also handle unsupported version
> > > > errors
> > > > > > > > > correctly
> > > > > > > > > > >>> if we
> > > > > > > > > > >>> > > > > receive
> > > > > > > > > > >>> > > > > > > them
> > > > > > > > > > >>> > > > > > > > >> in
> > > > > > > > > > >>> > > > > > > > >> > edge cases like
> upgrades/downgrades.
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > Justine
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM
> Jun
> > > Rao
> > > > > > > > > > >>> > > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> wrote:
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > 101.3 I realized that I actually
> > > have
> > > > > two
> > > > > > > > > > questions.
> > > > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> > > need
> > > > to
> > > > > > > write
> > > > > > > > > the
> > > > > > > > > > >>> > > previous
> > > > > > > > > > >>> > > > > > > produce
> > > > > > > > > > >>> > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > > tagged field in the end maker so
> > > that
> > > > we
> > > > > > > know
> > > > > > > > if
> > > > > > > > > > the
> > > > > > > > > > >>> > > marker
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > > from
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > new
> > > > > > > > > > >>> > > > > > > > >> > > client. Since the end maker is
> > > derived
> > > > > > from
> > > > > > > > the
> > > > > > > > > > >>> prepare
> > > > > > > > > > >>> > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> should we
> > > > > > > > > > >>> > > > > > > > >> > > write the previous produce Id in
> > the
> > > > > > prepare
> > > > > > > > > > marker
> > > > > > > > > > >>> > field
> > > > > > > > > > >>> > > > too?
> > > > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > > > >>> > > > > > > > >> > > we will lose this information
> when
> > > > > > deriving
> > > > > > > > the
> > > > > > > > > > end
> > > > > > > > > > >>> > > marker.
> > > > > > > > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we
> > > retry
> > > > > and
> > > > > > > see
> > > > > > > > > > epoch
> > > > > > > > > > >>> - 1
> > > > > > > > > > >>> > +
> > > > > > > > > > >>> > > ID
> > > > > > > > > > >>> > > > > in
> > > > > > > > > > >>> > > > > > > last
> > > > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > > > >>> > > > > > > > >> > > fields and are issuing the same
> > > > command
> > > > > > (ie
> > > > > > > > > commit
> > > > > > > > > > >>> not
> > > > > > > > > > >>> > > > abort),
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> return
> > > > > > > > > > >>> > > > > > > > >> > > success. The logic before
> KIP-890
> > > > seems
> > > > > to
> > > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > > > >> > > in this case. Are we
> intentionally
> > > > > making
> > > > > > > this
> > > > > > > > > > >>> change?
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > 112. We already merged the code
> > that
> > > > > adds
> > > > > > > the
> > > > > > > > > > >>> VerifyOnly
> > > > > > > > > > >>> > > > field
> > > > > > > > > > >>> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which
> > is
> > > an
> > > > > > inter
> > > > > > > > > > broker
> > > > > > > > > > >>> > > request.
> > > > > > > > > > >>> > > > > It
> > > > > > > > > > >>> > > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > > >>> > > > > > > > >> > > we didn't bump up the IBP for
> > that.
> > > Do
> > > > > you
> > > > > > > > know
> > > > > > > > > > why?
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > Jun
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM
> > > > Justine
> > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen"
> > to
> > > > > > "current
> > > > > > > > > > >>> producer id
> > > > > > > > > > >>> > > and
> > > > > > > > > > >>> > > > > > > epoch"
> > > > > > > > > > >>> > > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > > >>> > > > > > > > >> > > > was the part that was
> confusing
> > > > > > > > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > > > > > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > > > > > > > >>> > > > > > > > >> > > > 112 We still need it. But I am
> > > still
> > > > > > > > > finalizing
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > design.
> > > > > > > > > > >>> > > > > I
> > > > > > > > > > >>> > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > update
> > > > > > > > > > >>> > > > > > > > >> > > > the KIP once I get the
> > information
> > > > > > > > finalized.
> > > > > > > > > > >>> Sorry
> > > > > > > > > > >>> > for
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > delays.
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at
> 10:50 AM
> > > Jun
> > > > > Rao
> > > > > > > > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow
> > case,
> > > > the
> > > > > > > > previous
> > > > > > > > > > ID
> > > > > > > > > > >>> is
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > same
> > > > > > > > > > >>> > > > > > as
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > ID for the complete marker
> > too,
> > > > but
> > > > > we
> > > > > > > set
> > > > > > > > > the
> > > > > > > > > > >>> > > previous
> > > > > > > > > > >>> > > > ID
> > > > > > > > > > >>> > > > > > in
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > marker. Earlier you
> mentioned
> > > that
> > > > > > this
> > > > > > > is
> > > > > > > > > to
> > > > > > > > > > >>> know
> > > > > > > > > > >>> > > that
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > > > >>> > > > > > > > >> > > > > written by the new client so
> > > that
> > > > we
> > > > > > > could
> > > > > > > > > > >>> return
> > > > > > > > > > >>> > > > success
> > > > > > > > > > >>> > > > > on
> > > > > > > > > > >>> > > > > > > > >> retried
> > > > > > > > > > >>> > > > > > > > >> > > > > endMarker requests. I was
> > trying
> > > > to
> > > > > > > > > understand
> > > > > > > > > > >>> why
> > > > > > > > > > >>> > > this
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > not
> > > > > > > > > > >>> > > > > > > > >> needed
> > > > > > > > > > >>> > > > > > > > >> > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > the prepare marker since
> retry
> > > can
> > > > > > > happen
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >>> > > prepare
> > > > > > > > > > >>> > > > > > state
> > > > > > > > > > >>> > > > > > > > >> too.
> > > > > > > > > > >>> > > > > > > > >> > Is
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > reason that in the prepare
> > > state,
> > > > we
> > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > > > >> > > > instead
> > > > > > > > > > >>> > > > > > > > >> > > > > of success on retried
> endMaker
> > > > > > requests?
> > > > > > > > If
> > > > > > > > > > so,
> > > > > > > > > > >>> > should
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > change
> > > > > > > > > > >>> > > > > > > > >> "If
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID
> > in
> > > > last
> > > > > > > seen
> > > > > > > > > > >>> fields and
> > > > > > > > > > >>> > > are
> > > > > > > > > > >>> > > > > > > issuing
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > command (ie commit not
> abort)
> > we
> > > > can
> > > > > > > > return
> > > > > > > > > > >>> (with
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > epoch)"
> > > > > > > > > > >>> > > > > > > > >> > > > > accordingly?
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP,
> a
> > > > > delayed
> > > > > > > > > endMaker
> > > > > > > > > > >>> > request
> > > > > > > > > > >>> > > > > > carries
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > > > >>> > > > > > > > >> > > > > epoch and won't be fenced.
> > This
> > > > can
> > > > > > > > > > >>> commit/abort a
> > > > > > > > > > >>> > > > future
> > > > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure
> if
> > > we
> > > > > have
> > > > > > > > seen
> > > > > > > > > > >>> this in
> > > > > > > > > > >>> > > > > practice
> > > > > > > > > > >>> > > > > > > > >> though.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would
> be
> > > > useful
> > > > > > to
> > > > > > > > make
> > > > > > > > > > it
> > > > > > > > > > >>> > clear
> > > > > > > > > > >>> > > > that
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> now
> > > > > > > > > > >>> > > > > > > > >> > > > > populate the lastSeen field
> > from
> > > > the
> > > > > > log
> > > > > > > > > > >>> reliably.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > > > >>> > > > > since
> > > > > > > > > > >>> > > > > > > > it's
> > > > > > > > > > >>> > > > > > > > >> > > called
> > > > > > > > > > >>> > > > > > > > >> > > > > across brokers and we are
> > > changing
> > > > > its
> > > > > > > > > schema.
> > > > > > > > > > >>> Are
> > > > > > > > > > >>> > you
> > > > > > > > > > >>> > > > > > saying
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > don't
> > > > > > > > > > >>> > > > > > > > >> > > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > it any more? I thought that
> we
> > > > > already
> > > > > > > > > > >>> implemented
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > server
> > > > > > > > > > >>> > > > > > > > side
> > > > > > > > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > > > >>> > > > > across
> > > > > > > > > > >>> > > > > > > > >> brokers.
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at
> > 5:05 PM
> > > > > > Justine
> > > > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > >
> <jolshan@confluent.io.invalid
> > >
> > > > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the
> > > previous
> > > > ID
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>> Prepare
> > > > > > > > > > >>> > > > field
> > > > > > > > > > >>> > > > > > > since
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > > it. It is the same
> producer
> > ID
> > > > as
> > > > > > the
> > > > > > > > main
> > > > > > > > > > >>> > producer
> > > > > > > > > > >>> > > ID
> > > > > > > > > > >>> > > > > > > field.
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to
> > > > reread
> > > > > > your
> > > > > > > > > > message
> > > > > > > > > > >>> > about
> > > > > > > > > > >>> > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > markers.
> > > > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn
> > > marker
> > > > > > after
> > > > > > > > the
> > > > > > > > > > >>> > > transaction
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > > > already
> > > > > > > > > > >>> > > > > > > > >> > > > > complete?
> > > > > > > > > > >>> > > > > > > > >> > > > > > So we will commit the next
> > > > > > transaction
> > > > > > > > > early
> > > > > > > > > > >>> > without
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > fixes
> > > > > > > > > > >>> > > > > > > > >> in
> > > > > > > > > > >>> > > > > > > > >> > > part
> > > > > > > > > > >>> > > > > > > > >> > > > 2?
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 111 Yes -- this
> terminology
> > > was
> > > > > used
> > > > > > > in
> > > > > > > > a
> > > > > > > > > > >>> previous
> > > > > > > > > > >>> > > KIP
> > > > > > > > > > >>> > > > > and
> > > > > > > > > > >>> > > > > > > > never
> > > > > > > > > > >>> > > > > > > > >> > > > > > implemented it in the log
> --
> > > > only
> > > > > in
> > > > > > > > > memory
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which
> interbroker
> > > > > > protocol
> > > > > > > > are
> > > > > > > > > > you
> > > > > > > > > > >>> > > > referring
> > > > > > > > > > >>> > > > > > to?
> > > > > > > > > > >>> > > > > > > I
> > > > > > > > > > >>> > > > > > > > am
> > > > > > > > > > >>> > > > > > > > >> > > > working
> > > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > the design for the work to
> > > > remove
> > > > > > the
> > > > > > > > > extra
> > > > > > > > > > >>> add
> > > > > > > > > > >>> > > > > partitions
> > > > > > > > > > >>> > > > > > > > call
> > > > > > > > > > >>> > > > > > > > >> > and I
> > > > > > > > > > >>> > > > > > > > >> > > > > right
> > > > > > > > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I
> > > have
> > > > > yet
> > > > > > to
> > > > > > > > > > update
> > > > > > > > > > >>> that
> > > > > > > > > > >>> > > > > section
> > > > > > > > > > >>> > > > > > > as
> > > > > > > > > > >>> > > > > > > > I
> > > > > > > > > > >>> > > > > > > > >> > > > finalize
> > > > > > > > > > >>> > > > > > > > >> > > > > > the design so please stay
> > > tuned.
> > > > > Was
> > > > > > > > there
> > > > > > > > > > >>> > anything
> > > > > > > > > > >>> > > > else
> > > > > > > > > > >>> > > > > > you
> > > > > > > > > > >>> > > > > > > > >> > thought
> > > > > > > > > > >>> > > > > > > > >> > > > > needed
> > > > > > > > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at
> > > 3:07 PM
> > > > > Jun
> > > > > > > Rao
> > > > > > > > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > I don't see this create
> > any
> > > > > issue.
> > > > > > > It
> > > > > > > > > just
> > > > > > > > > > >>> makes
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > a
> > > > > > > > > > >>> > > > > > bit
> > > > > > > > > > >>> > > > > > > > >> hard to
> > > > > > > > > > >>> > > > > > > > >> > > > > explain
> > > > > > > > > > >>> > > > > > > > >> > > > > > > what this non-tagged
> > produce
> > > > id
> > > > > > > field
> > > > > > > > > > >>> means. We
> > > > > > > > > > >>> > > are
> > > > > > > > > > >>> > > > > > > > >> essentially
> > > > > > > > > > >>> > > > > > > > >> > > > trying
> > > > > > > > > > >>> > > > > > > > >> > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > combine two actions
> > > > (completing
> > > > > a
> > > > > > > txn
> > > > > > > > > and
> > > > > > > > > > >>> init a
> > > > > > > > > > >>> > > new
> > > > > > > > > > >>> > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> Id)
> > > > > > > > > > >>> > > > > > > > >> > > in a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > single record. But, this
> > may
> > > > be
> > > > > > fine
> > > > > > > > > too.
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > A few other follow up
> > > > comments.
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason
> > > that
> > > > we
> > > > > > > only
> > > > > > > > > set
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > previous
> > > > > > > > > > >>> > > > > > > > >> produce id
> > > > > > > > > > >>> > > > > > > > >> > > > > tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > field in the complete
> > > marker,
> > > > > but
> > > > > > > not
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >>> > > prepare
> > > > > > > > > > >>> > > > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > prepare state, we always
> > > > return
> > > > > > > > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > > > > > > > >>> > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > retried
> > > > > > > > > > >>> > > > > > > > >> > > > > > endMaker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > requests?
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 110. "I believe your
> > second
> > > > > point
> > > > > > is
> > > > > > > > > > >>> mentioned
> > > > > > > > > > >>> > in
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > KIP. I
> > > > > > > > > > >>> > > > > > > > >> can
> > > > > > > > > > >>> > > > > > > > >> > > add
> > > > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > text on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > The delayed message
> case
> > > can
> > > > > > also
> > > > > > > > > > violate
> > > > > > > > > > >>> EOS
> > > > > > > > > > >>> > if
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > message
> > > > > > > > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > > > > > > > addPartitionsToTxn
> > > > > > > > > > >>> > request
> > > > > > > > > > >>> > > > > comes
> > > > > > > > > > >>> > > > > > > in.
> > > > > > > > > > >>> > > > > > > > >> > > > > Effectively
> > > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > may see a message from a
> > > > > previous
> > > > > > > > > > (aborted)
> > > > > > > > > > >>> > > > > transaction
> > > > > > > > > > >>> > > > > > > > become
> > > > > > > > > > >>> > > > > > > > >> > part
> > > > > > > > > > >>> > > > > > > > >> > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > The above is the case
> > when a
> > > > > > delayed
> > > > > > > > > > >>> message is
> > > > > > > > > > >>> > > > > appended
> > > > > > > > > > >>> > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > data
> > > > > > > > > > >>> > > > > > > > >> > > > > > > partition. What I
> > mentioned
> > > > is a
> > > > > > > > > slightly
> > > > > > > > > > >>> > > different
> > > > > > > > > > >>> > > > > case
> > > > > > > > > > >>> > > > > > > > when
> > > > > > > > > > >>> > > > > > > > >> a
> > > > > > > > > > >>> > > > > > > > >> > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > marker is appended to
> the
> > > > > > > transaction
> > > > > > > > > log
> > > > > > > > > > >>> > > partition.
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once
> we
> > > > move
> > > > > > past
> > > > > > > > the
> > > > > > > > > > >>> Prepare
> > > > > > > > > > >>> > > and
> > > > > > > > > > >>> > > > > > > > Complete
> > > > > > > > > > >>> > > > > > > > >> > > states,
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > don’t need to worry
> about
> > > > > lastSeen
> > > > > > > > > fields
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > > clear
> > > > > > > > > > >>> > > > > > them,
> > > > > > > > > > >>> > > > > > > > just
> > > > > > > > > > >>> > > > > > > > >> > > handle
> > > > > > > > > > >>> > > > > > > > >> > > > > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > > > transitions as normal.".
> > Is
> > > > the
> > > > > > > > lastSeen
> > > > > > > > > > >>> field
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > same
> > > > > > > > > > >>> > > > > > as
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field
> in
> > > > > > > > > > >>> TransactionLogValue?
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 112. Since the kip
> changes
> > > the
> > > > > > > > > > inter-broker
> > > > > > > > > > >>> > > > protocol,
> > > > > > > > > > >>> > > > > > > should
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > > bump
> > > > > > > > > > >>> > > > > > > > >> > > > up
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this
> > > > feature
> > > > > > only
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > >>> > KRaft
> > > > > > > > > > >>> > > > > mode?
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at
> > > > 11:13 AM
> > > > > > > > Justine
> > > > > > > > > > >>> Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > <jolshan@confluent.io.invalid
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > I'm glad we are
> getting
> > to
> > > > > > > > convergence
> > > > > > > > > > on
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > design.
> > > > > > > > > > >>> > > > > > :)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > While I understand it
> > > seems
> > > > a
> > > > > > > little
> > > > > > > > > > >>> "weird".
> > > > > > > > > > >>> > > I'm
> > > > > > > > > > >>> > > > > not
> > > > > > > > > > >>> > > > > > > sure
> > > > > > > > > > >>> > > > > > > > >> what
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > benefit
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > of writing an extra
> > record
> > > > to
> > > > > > the
> > > > > > > > log.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool
> to
> > > > > > describe
> > > > > > > > > > >>> transactions
> > > > > > > > > > >>> > > > won't
> > > > > > > > > > >>> > > > > > > work
> > > > > > > > > > >>> > > > > > > > >> (ie,
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > state is needed to
> > > calculate
> > > > > the
> > > > > > > > time
> > > > > > > > > > >>> since
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > completed?)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > If we have a reason
> like
> > > > this,
> > > > > > it
> > > > > > > is
> > > > > > > > > > >>> enough to
> > > > > > > > > > >>> > > > > > convince
> > > > > > > > > > >>> > > > > > > me
> > > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > > >>> > > > > > > > >> > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > such
> > > > > > > > > > >>> > > > > > > > >> > > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > extra record. It seems
> > > like
> > > > it
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > >>> > > replacing
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > record
> > > > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is
> this
> > > > > correct?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024
> at
> > > > > 5:14 PM
> > > > > > > Jun
> > > > > > > > > Rao
> > > > > > > > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks for the
> > > > explanation.
> > > > > I
> > > > > > > > > > >>> understand the
> > > > > > > > > > >>> > > > > > intention
> > > > > > > > > > >>> > > > > > > > >> now.
> > > > > > > > > > >>> > > > > > > > >> > In
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > case, we set the
> > > > non-tagged
> > > > > > > field
> > > > > > > > to
> > > > > > > > > > >>> the old
> > > > > > > > > > >>> > > pid
> > > > > > > > > > >>> > > > > > (and
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> max
> > > > > > > > > > >>> > > > > > > > >> > > > > epoch)
> > > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > prepare marker so
> that
> > > we
> > > > > > could
> > > > > > > > > > >>> correctly
> > > > > > > > > > >>> > > write
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> marker to
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > data
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > partition if the
> > broker
> > > > > > > > downgrades.
> > > > > > > > > > When
> > > > > > > > > > >>> > > writing
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> complete
> > > > > > > > > > >>> > > > > > > > >> > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > know the marker has
> > > > already
> > > > > > been
> > > > > > > > > > >>> written to
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > data
> > > > > > > > > > >>> > > > > > > > >> > partition.
> > > > > > > > > > >>> > > > > > > > >> > > > We
> > > > > > > > > > >>> > > > > > > > >> > > > > > set
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to
> > the
> > > > new
> > > > > > pid
> > > > > > > to
> > > > > > > > > > avoid
> > > > > > > > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > client if the broker
> > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > The above seems to
> > work.
> > > > > It's
> > > > > > > > just a
> > > > > > > > > > bit
> > > > > > > > > > >>> > > > > > inconsistent
> > > > > > > > > > >>> > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> a
> > > > > > > > > > >>> > > > > > > > >> > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > and a complete
> marker
> > to
> > > > use
> > > > > > > > > different
> > > > > > > > > > >>> pids
> > > > > > > > > > >>> > in
> > > > > > > > > > >>> > > > > this
> > > > > > > > > > >>> > > > > > > > >> special
> > > > > > > > > > >>> > > > > > > > >> > > case.
> > > > > > > > > > >>> > > > > > > > >> > > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrade with the
> > > > complete
> > > > > > > > marker,
> > > > > > > > > it
> > > > > > > > > > >>> seems
> > > > > > > > > > >>> > > > that
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > never
> > > > > > > > > > >>> > > > > > > > >> > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > able
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > write the complete
> > > marker
> > > > > with
> > > > > > > the
> > > > > > > > > old
> > > > > > > > > > >>> pid.
> > > > > > > > > > >>> > > Not
> > > > > > > > > > >>> > > > > sure
> > > > > > > > > > >>> > > > > > > if
> > > > > > > > > > >>> > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > causes
> > > > > > > > > > >>> > > > > > > > >> > > > > any
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems
> a
> > > bit
> > > > > > weird.
> > > > > > > > > > >>> Instead of
> > > > > > > > > > >>> > > > > writing
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > with the new pid,
> > could
> > > we
> > > > > > write
> > > > > > > > two
> > > > > > > > > > >>> > records:
> > > > > > > > > > >>> > > a
> > > > > > > > > > >>> > > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > with
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > old pid followed by
> a
> > > > > > > > > > >>> TransactionLogValue
> > > > > > > > > > >>> > with
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > new
> > > > > > > > > > >>> > > > > > > > pid
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > empty
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > state? We could make
> > the
> > > > two
> > > > > > > > records
> > > > > > > > > > in
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > same
> > > > > > > > > > >>> > > > > > batch
> > > > > > > > > > >>> > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > > >>> > > > > > > > >> > > > they
> > > > > > > > > > >>> > > > > > > > >> > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > added to the log
> > > > atomically.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024
> > at
> > > > > > 5:40 PM
> > > > > > > > > > Justine
> > > > > > > > > > >>> > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > <jolshan@confluent.io.invalid
> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare
> > marker
> > > > is
> > > > > > > > written,
> > > > > > > > > > >>> but the
> > > > > > > > > > >>> > > > > endTxn
> > > > > > > > > > >>> > > > > > > > >> response
> > > > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > > > >>> > > > > > > > >> > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > received
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > by the client when
> > the
> > > > > > server
> > > > > > > > > > >>> downgrades
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare
> > > marker
> > > > is
> > > > > > > > > written,
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > endTxn
> > > > > > > > > > >>> > > > > > > > response
> > > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > > >>> > > > > > > > >> > > > > received
> > > > > > > > > > >>> > > > > > > > >> > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > client when the
> > server
> > > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > I think I am
> still a
> > > > > little
> > > > > > > > > > confused.
> > > > > > > > > > >>> In
> > > > > > > > > > >>> > > both
> > > > > > > > > > >>> > > > of
> > > > > > > > > > >>> > > > > > > these
> > > > > > > > > > >>> > > > > > > > >> > cases,
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction log
> has
> > > the
> > > > > old
> > > > > > > > > producer
> > > > > > > > > > >>> ID.
> > > > > > > > > > >>> > We
> > > > > > > > > > >>> > > > > don't
> > > > > > > > > > >>> > > > > > > > write
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > in the prepare
> > > marker's
> > > > > non
> > > > > > > > tagged
> > > > > > > > > > >>> fields.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > If the server
> > > downgrades
> > > > > > now,
> > > > > > > it
> > > > > > > > > > would
> > > > > > > > > > >>> > read
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > records
> > > > > > > > > > >>> > > > > > > > >> not
> > > > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields and the
> > > complete
> > > > > > marker
> > > > > > > > > will
> > > > > > > > > > >>> also
> > > > > > > > > > >>> > > have
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > old
> > > > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > ID.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (If we had used
> the
> > > new
> > > > > > > producer
> > > > > > > > > ID,
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > would
> > > > > > > > > > >>> > > > > not
> > > > > > > > > > >>> > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > transactional
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > correctness since
> > the
> > > > > > producer
> > > > > > > > id
> > > > > > > > > > >>> doesn't
> > > > > > > > > > >>> > > > match
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > state would not be
> > > > correct
> > > > > > on
> > > > > > > > the
> > > > > > > > > > data
> > > > > > > > > > >>> > > > > partition.)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > In the overflow
> > case,
> > > > I'd
> > > > > > > expect
> > > > > > > > > the
> > > > > > > > > > >>> > > following
> > > > > > > > > > >>> > > > > to
> > > > > > > > > > >>> > > > > > > > >> happen on
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > side
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we
> retry
> > > > EndTxn
> > > > > > --
> > > > > > > it
> > > > > > > > > is
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > same
> > > > > > > > > > >>> > > > > > > producer
> > > > > > > > > > >>> > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > -
> > > > > > > > > > >>> > > > > > > > >> > > > > > > 1
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > would fence the
> > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't
> > > retry
> > > > > > > EndTxn
> > > > > > > > > and
> > > > > > > > > > >>> use
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > >> producer id
> > > > > > > > > > >>> > > > > > > > >> > > > which
> > > > > > > > > > >>> > > > > > > > >> > > > > > > would
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > > > > > > > InvalidPidMappingException
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have
> > > > special
> > > > > > > > handling
> > > > > > > > > > for
> > > > > > > > > > >>> > when
> > > > > > > > > > >>> > > a
> > > > > > > > > > >>> > > > > > server
> > > > > > > > > > >>> > > > > > > > >> > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > When
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > reconnects we
> could
> > > get
> > > > an
> > > > > > API
> > > > > > > > > > version
> > > > > > > > > > >>> > > request
> > > > > > > > > > >>> > > > > > > showing
> > > > > > > > > > >>> > > > > > > > >> > > KIP-890
> > > > > > > > > > >>> > > > > > > > >> > > > > > part 2
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > not supported. In
> > that
> > > > > case,
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > >>> call
> > > > > > > > > > >>> > > > > > > > initProducerId
> > > > > > > > > > >>> > > > > > > > >> to
> > > > > > > > > > >>> > > > > > > > >> > > > abort
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In
> the
> > > > > > overflow
> > > > > > > > > case,
> > > > > > > > > > >>> this
> > > > > > > > > > >>> > > > > correctly
> > > > > > > > > > >>> > > > > > > > gives
> > > > > > > > > > >>> > > > > > > > >> > us a
> > > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > I guess the
> > > > corresponding
> > > > > > case
> > > > > > > > > would
> > > > > > > > > > >>> be
> > > > > > > > > > >>> > > where
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> *complete
> > > > > > > > > > >>> > > > > > > > >> > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > *is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > written but the
> > endTxn
> > > > is
> > > > > > not
> > > > > > > > > > >>> received by
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > client
> > > > > > > > > > >>> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > server
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This
> > would
> > > > > > result
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>> > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > coordinator
> > > > > > > > > > >>> > > > > > > > >> > > > > having
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old
> > > one.
> > > > > If
> > > > > > > the
> > > > > > > > > > client
> > > > > > > > > > >>> > > > retries,
> > > > > > > > > > >>> > > > > it
> > > > > > > > > > >>> > > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> > > receive
> > > > > > > > > > >>> > > > > > > > >> > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > InvalidPidMappingException.
> > > > > > > The
> > > > > > > > > > >>> > > InitProducerId
> > > > > > > > > > >>> > > > > > > > scenario
> > > > > > > > > > >>> > > > > > > > >> > above
> > > > > > > > > > >>> > > > > > > > >> > > > > would
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > help
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my
> > > > > > compatibility
> > > > > > > > > story
> > > > > > > > > > is
> > > > > > > > > > >>> > meant
> > > > > > > > > > >>> > > > to
> > > > > > > > > > >>> > > > > > > > support
> > > > > > > > > > >>> > > > > > > > >> > > > > downgrades
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > server
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > side in keeping
> the
> > > > > > > > transactional
> > > > > > > > > > >>> > > correctness.
> > > > > > > > > > >>> > > > > > > Keeping
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > from
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is
> > not
> > > > the
> > > > > > > > > priority.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I
> > can
> > > > > also
> > > > > > > add
> > > > > > > > > text
> > > > > > > > > > >>> in
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > KIP
> > > > > > > > > > >>> > > > > > > about
> > > > > > > > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > think that fixes
> > some
> > > > edge
> > > > > > > > cases.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12,
> 2024
> > > at
> > > > > > > 4:10 PM
> > > > > > > > > Jun
> > > > > > > > > > >>> Rao
> > > > > > > > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the
> > > reply.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we
> > > don't
> > > > > need
> > > > > > > to
> > > > > > > > > > >>> optimize
> > > > > > > > > > >>> > for
> > > > > > > > > > >>> > > > > > fencing
> > > > > > > > > > >>> > > > > > > > >> during
> > > > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Regarding
> > > consistency,
> > > > > > there
> > > > > > > > are
> > > > > > > > > > two
> > > > > > > > > > >>> > > > possible
> > > > > > > > > > >>> > > > > > > cases:
> > > > > > > > > > >>> > > > > > > > >> (1)
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is written, but
> > the
> > > > > endTxn
> > > > > > > > > > response
> > > > > > > > > > >>> is
> > > > > > > > > > >>> > not
> > > > > > > > > > >>> > > > > > > received
> > > > > > > > > > >>> > > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > when
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > server
> downgrades;
> > > (2)
> > > > > > the
> > > > > > > > > > prepare
> > > > > > > > > > >>> > marker
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > > > written,
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > endTxn
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is received by
> the
> > > > > client
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > >>> > server
> > > > > > > > > > >>> > > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> In
> > > > > > > > > > >>> > > > > > > > >> > > (1),
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > will have the
> old
> > > > > produce
> > > > > > Id
> > > > > > > > and
> > > > > > > > > > in
> > > > > > > > > > >>> (2),
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> will
> > > > > > > > > > >>> > > > > > > > >> > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If
> we
> > > > > > downgrade
> > > > > > > > > right
> > > > > > > > > > >>> after
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > can't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > consistent to
> both
> > > (1)
> > > > > and
> > > > > > > (2)
> > > > > > > > > > >>> since we
> > > > > > > > > > >>> > > can
> > > > > > > > > > >>> > > > > only
> > > > > > > > > > >>> > > > > > > put
> > > > > > > > > > >>> > > > > > > > >> one
> > > > > > > > > > >>> > > > > > > > >> > > > value
> > > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > existing produce
> > Id
> > > > > field.
> > > > > > > > It's
> > > > > > > > > > >>> also not
> > > > > > > > > > >>> > > > clear
> > > > > > > > > > >>> > > > > > > which
> > > > > > > > > > >>> > > > > > > > >> case
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > So we could
> > probably
> > > > be
> > > > > > > > > consistent
> > > > > > > > > > >>> with
> > > > > > > > > > >>> > > > either
> > > > > > > > > > >>> > > > > > > case.
> > > > > > > > > > >>> > > > > > > > >> By
> > > > > > > > > > >>> > > > > > > > >> > > > putting
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in
> the
> > > > > prepare
> > > > > > > > > marker,
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > are
> > > > > > > > > > >>> > > > > > > consistent
> > > > > > > > > > >>> > > > > > > > >> with
> > > > > > > > > > >>> > > > > > > > >> > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > (2)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > also has the
> > slight
> > > > > > benefit
> > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > >>> > > produce
> > > > > > > > > > >>> > > > > > field
> > > > > > > > > > >>> > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > complete marker
> > are
> > > > > > > consistent
> > > > > > > > > in
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > overflow
> > > > > > > > > > >>> > > > > > > case.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12,
> > 2024
> > > > at
> > > > > > > > 3:11 PM
> > > > > > > > > > >>> Justine
> > > > > > > > > > >>> > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > <jolshan@confluent.io.invalid
> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the case
> you
> > > > > > describe,
> > > > > > > we
> > > > > > > > > > would
> > > > > > > > > > >>> > need
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > > have
> > > > > > > > > > >>> > > > > > > a
> > > > > > > > > > >>> > > > > > > > >> > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > request,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > successful
> > EndTxn,
> > > > > and a
> > > > > > > > > > >>> successful
> > > > > > > > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > then
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn
> > > > request
> > > > > > go
> > > > > > > > > > through
> > > > > > > > > > >>> for
> > > > > > > > > > >>> > a
> > > > > > > > > > >>> > > > > given
> > > > > > > > > > >>> > > > > > > > >> producer.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to
> > > figure
> > > > > out
> > > > > > > if
> > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > >>> > > possible
> > > > > > > > > > >>> > > > > for
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > client
> > > > > > > > > > >>> > > > > > > > >> > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > a previous
> > request
> > > > is
> > > > > > > > delayed
> > > > > > > > > > >>> > somewhere.
> > > > > > > > > > >>> > > > But
> > > > > > > > > > >>> > > > > > > yes,
> > > > > > > > > > >>> > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > > > >>> > > > > > > > >> > > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > think
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > would fence
> the
> > > > > client.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the
> > > overflow
> > > > > > case.
> > > > > > > > In
> > > > > > > > > > the
> > > > > > > > > > >>> > > overflow
> > > > > > > > > > >>> > > > > > case,
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are
> > > different
> > > > on
> > > > > > the
> > > > > > > > > > marker
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > on
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> > > > > > So
> > > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to
> > use
> > > > the
> > > > > > max
> > > > > > > > > epoch
> > > > > > > > > > >>> but
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > start
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and
> > > > epoch 0
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>> > > > transactional
> > > > > > > > > > >>> > > > > > > state.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the server
> > > > > downgrade
> > > > > > > > case,
> > > > > > > > > we
> > > > > > > > > > >>> want
> > > > > > > > > > >>> > to
> > > > > > > > > > >>> > > > see
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > > >>> > > > > > > > >> > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > what the
> client
> > > will
> > > > > > have.
> > > > > > > > If
> > > > > > > > > we
> > > > > > > > > > >>> > > complete
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> commit,
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > state is
> > reloaded,
> > > > we
> > > > > > need
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > >>> > > > producer
> > > > > > > > > > >>> > > > > ID
> > > > > > > > > > >>> > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > there
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid
> > > producer
> > > > ID
> > > > > > > > > mapping.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > The server
> > > downgrade
> > > > > > cases
> > > > > > > > are
> > > > > > > > > > >>> > > considering
> > > > > > > > > > >>> > > > > > > > >> > transactional
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing
> > > from
> > > > > > > previous
> > > > > > > > > > >>> behavior
> > > > > > > > > > >>> > --
> > > > > > > > > > >>> > > > and
> > > > > > > > > > >>> > > > > > are
> > > > > > > > > > >>> > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > concerned
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > about
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the
> > > > safety
> > > > > > from
> > > > > > > > > > fencing
> > > > > > > > > > >>> > > retries
> > > > > > > > > > >>> > > > > (as
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> have
> > > > > > > > > > >>> > > > > > > > >> > > > > > downgraded
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to
> > > > > support).
> > > > > > > > > Perhaps
> > > > > > > > > > >>> this
> > > > > > > > > > >>> > is
> > > > > > > > > > >>> > > a
> > > > > > > > > > >>> > > > > > trade
> > > > > > > > > > >>> > > > > > > > off,
> > > > > > > > > > >>> > > > > > > > >> > but
> > > > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > > think
> > > > > > > > > > >>> > > > > > > > >> > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client
> > > > > > downgrades,
> > > > > > > > it
> > > > > > > > > > will
> > > > > > > > > > >>> > have
> > > > > > > > > > >>> > > > > > > restarted
> > > > > > > > > > >>> > > > > > > > >> and
> > > > > > > > > > >>> > > > > > > > >> > it
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > ok
> > > > > > > > > > >>> > > > > > > > >> > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > have a new
> > > producer
> > > > ID
> > > > > > > too).
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan
> 12,
> > > 2024
> > > > > at
> > > > > > > > > 11:42 AM
> > > > > > > > > > >>> Jun
> > > > > > > > > > >>> > Rao
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for
> the
> > > > > reply.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If
> the
> > > > marker
> > > > > > is
> > > > > > > > > > written
> > > > > > > > > > >>> by
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > >> client, we
> > > > > > > > > > >>> > > > > > > > >> > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the last
> email
> > > > > > guarantee
> > > > > > > > > that
> > > > > > > > > > >>> any
> > > > > > > > > > >>> > > EndTxn
> > > > > > > > > > >>> > > > > > > > requests
> > > > > > > > > > >>> > > > > > > > >> > with
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from the
> same
> > > > > producer
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > >>> same
> > > > > > > > > > >>> > > > > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> Then
> > > > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > return a
> > fenced
> > > > > error
> > > > > > > but
> > > > > > > > > can
> > > > > > > > > > >>> handle
> > > > > > > > > > >>> > > > > > > gracefully
> > > > > > > > > > >>> > > > > > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > described
> > > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When a
> delayed
> > > > > EndTnx
> > > > > > > > > request
> > > > > > > > > > is
> > > > > > > > > > >>> > > > > processed,
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> txn
> > > > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > > could
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next
> > > txn.
> > > > I
> > > > > > > guess
> > > > > > > > in
> > > > > > > > > > >>> this
> > > > > > > > > > >>> > case
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > still
> > > > > > > > > > >>> > > > > > > > >> > return
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > fenced
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed
> > > > request?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry,
> my
> > > > > > question
> > > > > > > > was
> > > > > > > > > > >>> > > inaccurate.
> > > > > > > > > > >>> > > > > What
> > > > > > > > > > >>> > > > > > > you
> > > > > > > > > > >>> > > > > > > > >> > > > described
> > > > > > > > > > >>> > > > > > > > >> > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > "The
> downgrade
> > > > > > > > > compatibility I
> > > > > > > > > > >>> > mention
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > that
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > keep
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in
> > the
> > > > > main
> > > > > > > > > > >>> (non-tagged)
> > > > > > > > > > >>> > > > fields
> > > > > > > > > > >>> > > > > as
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> did
> > > > > > > > > > >>> > > > > > > > >> > > > before
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > code
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > server
> side."
> > If
> > > > we
> > > > > > want
> > > > > > > > to
> > > > > > > > > do
> > > > > > > > > > >>> this,
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> that we
> > > > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > > > >>> > > > > > > > >> > > > > > > use
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > current
> > produce
> > > Id
> > > > > and
> > > > > > > max
> > > > > > > > > > >>> epoch in
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > existing
> > > > > > > > > > >>> > > > > > > > >> > > > producerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> producerEpoch
> > > > fields
> > > > > > for
> > > > > > > > > both
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > prepare
> > > > > > > > > > >>> > > > > > and
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > The
> downgrade
> > > can
> > > > > > happen
> > > > > > > > > after
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > complete
> > > > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > > > >>> > > > > > > > >> > > > > > written.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > With
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > you
> described,
> > > the
> > > > > > > > > downgraded
> > > > > > > > > > >>> > > > coordinator
> > > > > > > > > > >>> > > > > > will
> > > > > > > > > > >>> > > > > > > > see
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of
> the
> > > old
> > > > > > one.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan
> > 12,
> > > > 2024
> > > > > > at
> > > > > > > > > > 10:44 AM
> > > > > > > > > > >>> > > Justine
> > > > > > > > > > >>> > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > <jolshan@confluent.io.invalid
> > > > > > > > > > >
> > > > > > > > > > >>> > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can
> update
> > > the
> > > > > > > > > > description.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe
> > your
> > > > > > second
> > > > > > > > > point
> > > > > > > > > > is
> > > > > > > > > > >>> > > > mentioned
> > > > > > > > > > >>> > > > > > in
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > KIP.
> > > > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > > > >>> > > > > > > > >> > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > add
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it
> > is
> > > > > > helpful.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The
> > delayed
> > > > > > message
> > > > > > > > case
> > > > > > > > > > can
> > > > > > > > > > >>> > also
> > > > > > > > > > >>> > > > > > violate
> > > > > > > > > > >>> > > > > > > > EOS
> > > > > > > > > > >>> > > > > > > > >> if
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in
> > after
> > > > the
> > > > > > > next
> > > > > > > > > > >>> > > > > addPartitionsToTxn
> > > > > > > > > > >>> > > > > > > > >> request
> > > > > > > > > > >>> > > > > > > > >> > > > comes
> > > > > > > > > > >>> > > > > > > > >> > > > > > in.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a
> > > > message
> > > > > > > from a
> > > > > > > > > > >>> previous
> > > > > > > > > > >>> > > > > > (aborted)
> > > > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > become
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > part
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > next
> > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the
> > marker
> > > is
> > > > > > > written
> > > > > > > > > by
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > new
> > > > > > > > > > >>> > > > > > client,
> > > > > > > > > > >>> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> can
> > > > > > > > > > >>> > > > > > > > >> > > as I
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email
> > > > > guarantee
> > > > > > > > that
> > > > > > > > > > any
> > > > > > > > > > >>> > EndTxn
> > > > > > > > > > >>> > > > > > > requests
> > > > > > > > > > >>> > > > > > > > >> with
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same
> > > > producer
> > > > > > and
> > > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > >>> > > > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> Then we
> > > > > > > > > > >>> > > > > > > > >> > > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced
> error
> > > but
> > > > > can
> > > > > > > > > handle
> > > > > > > > > > >>> > > gracefully
> > > > > > > > > > >>> > > > > as
> > > > > > > > > > >>> > > > > > > > >> described
> > > > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't
> > think
> > > a
> > > > > > > boolean
> > > > > > > > is
> > > > > > > > > > >>> useful
> > > > > > > > > > >>> > > > since
> > > > > > > > > > >>> > > > > it
> > > > > > > > > > >>> > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > directly
> > > > > > > > > > >>> > > > > > > > >> > > > > > > encoded
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence
> or
> > > > lack
> > > > > of
> > > > > > > the
> > > > > > > > > > >>> tagged
> > > > > > > > > > >>> > > field
> > > > > > > > > > >>> > > > > > being
> > > > > > > > > > >>> > > > > > > > >> > written.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> > prepare
> > > > > > marker
> > > > > > > we
> > > > > > > > > > will
> > > > > > > > > > >>> have
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In
> > the
> > > > > > Complete
> > > > > > > > > state
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > may
> > > > > > > > > > >>> > > > not.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not
> sure
> > > why
> > > > > the
> > > > > > > > > ongoing
> > > > > > > > > > >>> state
> > > > > > > > > > >>> > > > > matters
> > > > > > > > > > >>> > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> this
> > > > > > > > > > >>> > > > > > > > >> > > > KIP.
> > > > > > > > > > >>> > > > > > > > >> > > > > It
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > does
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not
> sure
> > > > what
> > > > > > you
> > > > > > > > are
> > > > > > > > > > >>> > referring
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > > about
> > > > > > > > > > >>> > > > > > > > >> > writing
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the
> > > > prepare
> > > > > > > > marker.
> > > > > > > > > > >>> This is
> > > > > > > > > > >>> > > not
> > > > > > > > > > >>> > > > in
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> KIP.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> > > overflow
> > > > > > case,
> > > > > > > we
> > > > > > > > > > >>> write the
> > > > > > > > > > >>> > > > > > > > >> nextProducerId
> > > > > > > > > > >>> > > > > > > > >> > in
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so
> > we
> > > > know
> > > > > > > what
> > > > > > > > we
> > > > > > > > > > >>> > assigned
> > > > > > > > > > >>> > > > when
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> reload
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we
> > > > complete,
> > > > > we
> > > > > > > > > > >>> transition
> > > > > > > > > > >>> > this
> > > > > > > > > > >>> > > > ID
> > > > > > > > > > >>> > > > > to
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> main
> > > > > > > > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the
> > > > previous
> > > > > > > > producer
> > > > > > > > > > ID
> > > > > > > > > > >>> > field
> > > > > > > > > > >>> > > > > filled
> > > > > > > > > > >>> > > > > > > in.
> > > > > > > > > > >>> > > > > > > > >> This
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry
> > > case
> > > > > the
> > > > > > > > > > operation
> > > > > > > > > > >>> > > > completed
> > > > > > > > > > >>> > > > > > > > >> > successfully
> > > > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > don't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer.
> > The
> > > > > > > downgrade
> > > > > > > > > > >>> > > compatibility
> > > > > > > > > > >>> > > > I
> > > > > > > > > > >>> > > > > > > > mention
> > > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > keep
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> ID
> > > and
> > > > > > epoch
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>> main
> > > > > > > > > > >>> > > > > > > (non-tagged)
> > > > > > > > > > >>> > > > > > > > >> > fields
> > > > > > > > > > >>> > > > > > > > >> > > > as
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > did
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on
> the
> > > > server
> > > > > > > side.
> > > > > > > > > If
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > server
> > > > > > > > > > >>> > > > > > > > >> downgrades,
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > still
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > addresses
> > > > > both
> > > > > > > the
> > > > > > > > > > >>> prepare
> > > > > > > > > > >>> > and
> > > > > > > > > > >>> > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> state
> > > > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri,
> Jan
> > > 12,
> > > > > 2024
> > > > > > > at
> > > > > > > > > > >>> 10:21 AM
> > > > > > > > > > >>> > Jun
> > > > > > > > > > >>> > > > Rao
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi,
> > Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks
> for
> > > the
> > > > > > > reply.
> > > > > > > > > > Sorry
> > > > > > > > > > >>> for
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > delay. I
> > > > > > > > > > >>> > > > > > > > >> > have a
> > > > > > > > > > >>> > > > > > > > >> > > > few
> > > > > > > > > > >>> > > > > > > > >> > > > > > > more
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I
> > think
> > > > the
> > > > > > > > > > motivation
> > > > > > > > > > >>> > > section
> > > > > > > > > > >>> > > > > > could
> > > > > > > > > > >>> > > > > > > be
> > > > > > > > > > >>> > > > > > > > >> > > > improved.
> > > > > > > > > > >>> > > > > > > > >> > > > > > One
> > > > > > > > > > >>> > > > > > > > >> > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > motivations
> > > > > listed
> > > > > > > by
> > > > > > > > > the
> > > > > > > > > > >>> KIP is
> > > > > > > > > > >>> > > > "This
> > > > > > > > > > >>> > > > > > can
> > > > > > > > > > >>> > > > > > > > >> happen
> > > > > > > > > > >>> > > > > > > > >> > > > when
> > > > > > > > > > >>> > > > > > > > >> > > > > a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > message
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or
> delayed
> > > due
> > > > > to
> > > > > > > > > > networking
> > > > > > > > > > >>> > > issues
> > > > > > > > > > >>> > > > > or a
> > > > > > > > > > >>> > > > > > > > >> network
> > > > > > > > > > >>> > > > > > > > >> > > > > > partition,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts,
> > and
> > > > then
> > > > > > the
> > > > > > > > > > delayed
> > > > > > > > > > >>> > > message
> > > > > > > > > > >>> > > > > > > finally
> > > > > > > > > > >>> > > > > > > > >> > comes
> > > > > > > > > > >>> > > > > > > > >> > > > > in.".
> > > > > > > > > > >>> > > > > > > > >> > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very
> > > accurate.
> > > > > > > Without
> > > > > > > > > > >>> KIP-890,
> > > > > > > > > > >>> > > > > > currently,
> > > > > > > > > > >>> > > > > > > > if
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts
> an
> > > > > ongoing
> > > > > > > > > > >>> transaction,
> > > > > > > > > > >>> > it
> > > > > > > > > > >>> > > > > > already
> > > > > > > > > > >>> > > > > > > > >> bumps
> > > > > > > > > > >>> > > > > > > > >> > up
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which
> > > prevents
> > > > > the
> > > > > > > > > delayed
> > > > > > > > > > >>> > produce
> > > > > > > > > > >>> > > > > > message
> > > > > > > > > > >>> > > > > > > > >> from
> > > > > > > > > > >>> > > > > > > > >> > > being
> > > > > > > > > > >>> > > > > > > > >> > > > > > added
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> partition.
> > > > What
> > > > > > can
> > > > > > > > > cause
> > > > > > > > > > a
> > > > > > > > > > >>> > > hanging
> > > > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > > > >>> > > > > > > > >> > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> completes
> > > > > (either
> > > > > > > > aborts
> > > > > > > > > > or
> > > > > > > > > > >>> > > > commits) a
> > > > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > before
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> successful
> > > ack
> > > > > on
> > > > > > > > > messages
> > > > > > > > > > >>> > > published
> > > > > > > > > > >>> > > > > in
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> same
> > > > > > > > > > >>> > > > > > > > >> > > txn.
> > > > > > > > > > >>> > > > > > > > >> > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible
> > for
> > > > the
> > > > > > > > delayed
> > > > > > > > > > >>> message
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > be
> > > > > > > > > > >>> > > > > > > > >> appended
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker,
> > > > causing
> > > > > a
> > > > > > > > > > >>> transaction to
> > > > > > > > > > >>> > > > hang.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A
> similar
> > > > issue
> > > > > > (not
> > > > > > > > > > >>> mentioned
> > > > > > > > > > >>> > in
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > >> motivation)
> > > > > > > > > > >>> > > > > > > > >> > > > could
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > happen
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker
> in
> > > the
> > > > > > > > > > coordinator's
> > > > > > > > > > >>> log.
> > > > > > > > > > >>> > > For
> > > > > > > > > > >>> > > > > > > > example,
> > > > > > > > > > >>> > > > > > > > >> > it's
> > > > > > > > > > >>> > > > > > > > >> > > > > > possible
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > EndTxnRequest
> > > > to
> > > > > > be
> > > > > > > > > > delayed
> > > > > > > > > > >>> on
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > > > >> coordinator.
> > > > > > > > > > >>> > > > > > > > >> > By
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > time
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > EndTxnRequest
> > > > is
> > > > > > > > > > processed,
> > > > > > > > > > >>> it's
> > > > > > > > > > >>> > > > > > possible
> > > > > > > > > > >>> > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > txn
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> completed
> > > and
> > > > a
> > > > > > new
> > > > > > > > txn
> > > > > > > > > > has
> > > > > > > > > > >>> > > started.
> > > > > > > > > > >>> > > > > > > > >> Currently,
> > > > > > > > > > >>> > > > > > > > >> > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped
> on
> > > > every
> > > > > > txn,
> > > > > > > > the
> > > > > > > > > > >>> delayed
> > > > > > > > > > >>> > > > > > > > EndTxnRequest
> > > > > > > > > > >>> > > > > > > > >> > will
> > > > > > > > > > >>> > > > > > > > >> > > > add
> > > > > > > > > > >>> > > > > > > > >> > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare
> > > marker
> > > > > > (and
> > > > > > > > > > >>> eventually a
> > > > > > > > > > >>> > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> marker)
> > > > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't
> > cause
> > > > the
> > > > > > > > > > transaction
> > > > > > > > > > >>> to
> > > > > > > > > > >>> > > hang,
> > > > > > > > > > >>> > > > > but
> > > > > > > > > > >>> > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> will
> > > > > > > > > > >>> > > > > > > > >> > > > break
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The
> > proposal
> > > > in
> > > > > > this
> > > > > > > > KIP
> > > > > > > > > > >>> will
> > > > > > > > > > >>> > > > address
> > > > > > > > > > >>> > > > > > this
> > > > > > > > > > >>> > > > > > > > >> issue
> > > > > > > > > > >>> > > > > > > > >> > > too.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.
> > > > "However, I
> > > > > > was
> > > > > > > > > > >>> writing it
> > > > > > > > > > >>> > so
> > > > > > > > > > >>> > > > > that
> > > > > > > > > > >>> > > > > > we
> > > > > > > > > > >>> > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > distinguish
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old
> > clients
> > > > > where
> > > > > > we
> > > > > > > > > don't
> > > > > > > > > > >>> have
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > ability
> > > > > > > > > > >>> > > > > > > > do
> > > > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > operation
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients
> > that
> > > > > can.
> > > > > > > (Old
> > > > > > > > > > >>> clients
> > > > > > > > > > >>> > > don't
> > > > > > > > > > >>> > > > > > bump
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for
> > sure
> > > > the
> > > > > > > write
> > > > > > > > > > >>> belongs
> > > > > > > > > > >>> > to
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > given
> > > > > > > > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I
> am
> > > > > > wondering
> > > > > > > > why
> > > > > > > > > > we
> > > > > > > > > > >>> need
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > > > > >> distinguish
> > > > > > > > > > >>> > > > > > > > >> > > > whether
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written
> by
> > > the
> > > > > old
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > >>> new
> > > > > > > > > > >>> > > > client.
> > > > > > > > > > >>> > > > > > > Could
> > > > > > > > > > >>> > > > > > > > >> you
> > > > > > > > > > >>> > > > > > > > >> > > > > describe
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > what
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > differently
> > > if
> > > > > we
> > > > > > > know
> > > > > > > > > the
> > > > > > > > > > >>> > marker
> > > > > > > > > > >>> > > is
> > > > > > > > > > >>> > > > > > > written
> > > > > > > > > > >>> > > > > > > > >> by
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > client?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If
> > we
> > > do
> > > > > > need
> > > > > > > a
> > > > > > > > > way
> > > > > > > > > > to
> > > > > > > > > > >>> > > > > distinguish
> > > > > > > > > > >>> > > > > > > > >> whether
> > > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old
> > and
> > > > the
> > > > > > new
> > > > > > > > > > client.
> > > > > > > > > > >>> > Would
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > > be
> > > > > > > > > > >>> > > > > > > > >> simpler
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > > just
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field
> > > instead
> > > > of
> > > > > > > > > > indirectly
> > > > > > > > > > >>> > > through
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> previous
> > > > > > > > > > >>> > > > > > > > >> > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3
> It's
> > > not
> > > > > > clear
> > > > > > > > to
> > > > > > > > > me
> > > > > > > > > > >>> why
> > > > > > > > > > >>> > we
> > > > > > > > > > >>> > > > only
> > > > > > > > > > >>> > > > > > add
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > complete
> > > > > > marker,
> > > > > > > > but
> > > > > > > > > > >>> not in
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> > marker.
> > > > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > want
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether
> a
> > > > marker
> > > > > > is
> > > > > > > > > > written
> > > > > > > > > > >>> by
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > new
> > > > > > > > > > >>> > > > > > > > client
> > > > > > > > > > >>> > > > > > > > >> or
> > > > > > > > > > >>> > > > > > > > >> > > not,
> > > > > > > > > > >>> > > > > > > > >> > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > seems
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do
> this
> > > > > > > > consistently
> > > > > > > > > > for
> > > > > > > > > > >>> all
> > > > > > > > > > >>> > > > > markers.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4
> What
> > > > about
> > > > > > the
> > > > > > > > > > >>> > > > > TransactionLogValue
> > > > > > > > > > >>> > > > > > > > >> record
> > > > > > > > > > >>> > > > > > > > >> > > > > > > representing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state?
> > > Should
> > > > we
> > > > > > > also
> > > > > > > > > > >>> > distinguish
> > > > > > > > > > >>> > > > > > whether
> > > > > > > > > > >>> > > > > > > > it's
> > > > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > > > >>> > > > > > > > >> > > > > by
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > old
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new
> > client?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In
> > the
> > > > > > overflow
> > > > > > > > > case,
> > > > > > > > > > >>> it's
> > > > > > > > > > >>> > > > still
> > > > > > > > > > >>> > > > > > not
> > > > > > > > > > >>> > > > > > > > >> clear
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > me
> > > > > > > > > > >>> > > > > > > > >> > > > > why
> > > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous
> > > > produce
> > > > > > Id
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >>> > prepare
> > > > > > > > > > >>> > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> while
> > > > > > > > > > >>> > > > > > > > >> > > > writing
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the
> > > > complete
> > > > > > > > marker.
> > > > > > > > > > You
> > > > > > > > > > >>> > > > mentioned
> > > > > > > > > > >>> > > > > > that
> > > > > > > > > > >>> > > > > > > > >> it's
> > > > > > > > > > >>> > > > > > > > >> > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could
> > > > > downgrade
> > > > > > > > with
> > > > > > > > > > >>> either
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > prepare
> > > > > > > > > > >>> > > > > > > > >> marker
> > > > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In
> either
> > > > case,
> > > > > > the
> > > > > > > > > > >>> downgraded
> > > > > > > > > > >>> > > > > > coordinator
> > > > > > > > > > >>> > > > > > > > >> should
> > > > > > > > > > >>> > > > > > > > >> > > see
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> (probably
> > > the
> > > > > > > previous
> > > > > > > > > > >>> produce
> > > > > > > > > > >>> > > Id),
> > > > > > > > > > >>> > > > > > right?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed,
> > Dec
> > > > 20,
> > > > > > 2023
> > > > > > > > at
> > > > > > > > > > >>> 6:00 PM
> > > > > > > > > > >>> > > > > Justine
> > > > > > > > > > >>> > > > > > > > Olshan
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey
> Jun,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks
> > for
> > > > > > taking
> > > > > > > a
> > > > > > > > > look
> > > > > > > > > > >>> at
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > KIP
> > > > > > > > > > >>> > > > > > > again.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100.
> For
> > > the
> > > > > > epoch
> > > > > > > > > > >>> overflow
> > > > > > > > > > >>> > > case,
> > > > > > > > > > >>> > > > > only
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > will
> > > > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > max
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps
> > the
> > > > > > behavior
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >>> rest
> > > > > > > > > > >>> > > of
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> markers
> > > > > > > > > > >>> > > > > > > > >> > > where
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > last
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch
> of
> > > the
> > > > > > > > > transaction
> > > > > > > > > > >>> > > records +
> > > > > > > > > > >>> > > > > 1.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101.
> You
> > > are
> > > > > > > correct
> > > > > > > > > > that
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > don't
> > > > > > > > > > >>> > > > > > need
> > > > > > > > > > >>> > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > write
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the
> > > same.
> > > > > > > > However,
> > > > > > > > > I
> > > > > > > > > > >>> was
> > > > > > > > > > >>> > > > writing
> > > > > > > > > > >>> > > > > it
> > > > > > > > > > >>> > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> that
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old
> > > clients
> > > > > > where
> > > > > > > we
> > > > > > > > > > don't
> > > > > > > > > > >>> > have
> > > > > > > > > > >>> > > > the
> > > > > > > > > > >>> > > > > > > > ability
> > > > > > > > > > >>> > > > > > > > >> do
> > > > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > operation
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> clients
> > > that
> > > > > > can.
> > > > > > > > (Old
> > > > > > > > > > >>> clients
> > > > > > > > > > >>> > > > don't
> > > > > > > > > > >>> > > > > > > bump
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say
> for
> > > sure
> > > > > the
> > > > > > > > write
> > > > > > > > > > >>> belongs
> > > > > > > > > > >>> > > to
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > given
> > > > > > > > > > >>> > > > > > > > >> > > > > > transaction).
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn
> > > > request
> > > > > > > from
> > > > > > > > a
> > > > > > > > > > new
> > > > > > > > > > >>> > > client,
> > > > > > > > > > >>> > > > we
> > > > > > > > > > >>> > > > > > > will
> > > > > > > > > > >>> > > > > > > > >> fill
> > > > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > > > >>> > > > > > > > >> > > > > > > field.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > We
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> guarantee
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that
> any
> > > > > EndTxn
> > > > > > > > > requests
> > > > > > > > > > >>> with
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > same
> > > > > > > > > > >>> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > are
> > > > > > > > > > >>> > > > > > > > >> > > > from
> > > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the
> same
> > > > > > > > transaction.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102.
> In
> > > > > prepare
> > > > > > > > phase,
> > > > > > > > > > we
> > > > > > > > > > >>> have
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > same
> > > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had.
> It
> > is
> > > > the
> > > > > > > > > producer
> > > > > > > > > > >>> ID and
> > > > > > > > > > >>> > > > epoch
> > > > > > > > > > >>> > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> are
> > > > > > > > > > >>> > > > > > > > >> > on
> > > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase,
> > we
> > > > stay
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > >>> unless
> > > > > > > > > > >>> > > it
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > overflow
> > > > > > > > > > >>> > > > > > > > >> > > > > case.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set
> the
> > > > > producer
> > > > > > > ID
> > > > > > > > to
> > > > > > > > > > >>> the new
> > > > > > > > > > >>> > > one
> > > > > > > > > > >>> > > > > we
> > > > > > > > > > >>> > > > > > > > >> generated
> > > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > complete.
> > > > This
> > > > > > is
> > > > > > > > for
> > > > > > > > > > >>> > downgrade
> > > > > > > > > > >>> > > > > > > > >> compatibility.
> > > > > > > > > > >>> > > > > > > > >> > > The
> > > > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety
> > > > guards
> > > > > > for
> > > > > > > > > > retries
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > > > > > failovers.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In
> > prepare
> > > > > phase
> > > > > > > for
> > > > > > > > > > epoch
> > > > > > > > > > >>> > > > overflow
> > > > > > > > > > >>> > > > > > case
> > > > > > > > > > >>> > > > > > > > >> only
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > > store
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID.
> This
> > > is
> > > > > for
> > > > > > > the
> > > > > > > > > case
> > > > > > > > > > >>> where
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > > reload
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > transaction
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> prepare
> > > > state.
> > > > > > > Once
> > > > > > > > > the
> > > > > > > > > > >>> > > > transaction
> > > > > > > > > > >>> > > > > is
> > > > > > > > > > >>> > > > > > > > >> > committed,
> > > > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > can
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > use
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the
> > > > client
> > > > > > > > already
> > > > > > > > > is
> > > > > > > > > > >>> > using.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In
> > commit
> > > > > phase,
> > > > > > > we
> > > > > > > > > > store
> > > > > > > > > > >>> the
> > > > > > > > > > >>> > > > > previous
> > > > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I
> think
> > it
> > > > is
> > > > > > > easier
> > > > > > > > > to
> > > > > > > > > > >>> think
> > > > > > > > > > >>> > of
> > > > > > > > > > >>> > > > it
> > > > > > > > > > >>> > > > > as
> > > > > > > > > > >>> > > > > > > > just
> > > > > > > > > > >>> > > > > > > > >> how
> > > > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > were
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > epoch
> > > > > > before,
> > > > > > > > with
> > > > > > > > > > >>> some
> > > > > > > > > > >>> > > extra
> > > > > > > > > > >>> > > > > > > > bookeeping
> > > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > > >>> > > > > > > > >> > > > edge
> > > > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged
> > > > fields.
> > > > > > We
> > > > > > > > have
> > > > > > > > > > to
> > > > > > > > > > >>> do
> > > > > > > > > > >>> > it
> > > > > > > > > > >>> > > > this
> > > > > > > > > > >>> > > > > > way
> > > > > > > > > > >>> > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > with
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > downgrades.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103.
> > Next
> > > > > > producer
> > > > > > > > ID
> > > > > > > > > is
> > > > > > > > > > >>> for
> > > > > > > > > > >>> > > > prepare
> > > > > > > > > > >>> > > > > > > > status
> > > > > > > > > > >>> > > > > > > > >> and
> > > > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after
> > > > > complete.
> > > > > > > The
> > > > > > > > > > >>> reason why
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > need
> > > > > > > > > > >>> > > > > > > two
> > > > > > > > > > >>> > > > > > > > >> > > separate
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > backwards
> > > > > > > > > compatibility.
> > > > > > > > > > >>> We
> > > > > > > > > > >>> > need
> > > > > > > > > > >>> > > > to
> > > > > > > > > > >>> > > > > > keep
> > > > > > > > > > >>> > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > > > >>> > > > > > > > >> > > > > > > semantics
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > for
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > non-tagged
> > > > > field
> > > > > > > in
> > > > > > > > > case
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > > > > downgrade.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104.
> We
> > > set
> > > > > the
> > > > > > > > fields
> > > > > > > > > > as
> > > > > > > > > > >>> we
> > > > > > > > > > >>> > do
> > > > > > > > > > >>> > > in
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > transactional
> > > > > > > > > > >>> > > > > > > > >> > > > > > > state
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do
> this
> > > for
> > > > > > > > > > compatibility
> > > > > > > > > > >>> --
> > > > > > > > > > >>> > if
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > > > > >> downgrade,
> > > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > > >>> > > > > > > > >> > > > will
> > > > > > > > > > >>> > > > > > > > >> > > > > > > only
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > non-tagged
> > > > > > fields)
> > > > > > > > It
> > > > > > > > > > >>> will be
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > old
> > > > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > > > >>> > > > > > > > >> > ID
> > > > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > max
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope
> > this
> > > > > helps.
> > > > > > > Let
> > > > > > > > > me
> > > > > > > > > > >>> know
> > > > > > > > > > >>> > if
> > > > > > > > > > >>> > > > you
> > > > > > > > > > >>> > > > > > have
> > > > > > > > > > >>> > > > > > > > >> > further
> > > > > > > > > > >>> > > > > > > > >> > > > > > > questions.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> Justine
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On
> Wed,
> > > Dec
> > > > > 20,
> > > > > > > 2023
> > > > > > > > > at
> > > > > > > > > > >>> > 3:33 PM
> > > > > > > > > > >>> > > > Jun
> > > > > > > > > > >>> > > > > > Rao
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > <jun@confluent.io.invalid
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi,
> > > > Justine,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It
> > seems
> > > > > that
> > > > > > > you
> > > > > > > > > have
> > > > > > > > > > >>> made
> > > > > > > > > > >>> > > some
> > > > > > > > > > >>> > > > > > > changes
> > > > > > > > > > >>> > > > > > > > >> to
> > > > > > > > > > >>> > > > > > > > >> > > > KIP-890
> > > > > > > > > > >>> > > > > > > > >> > > > > > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > particular,
> > > > > we
> > > > > > > are
> > > > > > > > > > >>> changing
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > > > > format
> > > > > > > > > > >>> > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> TransactionLogValue.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > comments
> > > > > > related
> > > > > > > > to
> > > > > > > > > > >>> that.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100.
> > > Just
> > > > to
> > > > > > be
> > > > > > > > > clear.
> > > > > > > > > > >>> The
> > > > > > > > > > >>> > > > > overflow
> > > > > > > > > > >>> > > > > > > case
> > > > > > > > > > >>> > > > > > > > >> > (i.e.
> > > > > > > > > > >>> > > > > > > > >> > > > > when a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > generated)
> > > > > is
> > > > > > > when
> > > > > > > > > the
> > > > > > > > > > >>> > current
> > > > > > > > > > >>> > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> equals
> > > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > > >>> > > > > > > > >> > > > max
> > > > > > > > > > >>> > > > > > > > >> > > > > -
> > > > > > > > > > >>> > > > > > > > >> > > > > > 1
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101.
> > For
> > > > the
> > > > > > > "not
> > > > > > > > > > epoch
> > > > > > > > > > >>> > > > overflow"
> > > > > > > > > > >>> > > > > > > case,
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > write
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> tagged
> > > > field
> > > > > > in
> > > > > > > > the
> > > > > > > > > > >>> complete
> > > > > > > > > > >>> > > > > phase.
> > > > > > > > > > >>> > > > > > Do
> > > > > > > > > > >>> > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > need
> > > > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > do
> > > > > > > > > > >>> > > > > > > > >> > > > > > > that
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > doesn't
> > > > > change
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > > >>> case?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102.
> > It
> > > > > seems
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > >>> > meaning
> > > > > > > > > > >>> > > > for
> > > > > > > > > > >>> > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > ProducerId/ProducerEpoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > TransactionLogValue
> > > > > > > > > > >>> changes
> > > > > > > > > > >>> > > > > > depending
> > > > > > > > > > >>> > > > > > > on
> > > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > > > TransactionStatus
> > > > > > > > > > is
> > > > > > > > > > >>> > > > ongoing,
> > > > > > > > > > >>> > > > > > they
> > > > > > > > > > >>> > > > > > > > >> > > represent
> > > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > current
> > > > > > > > > > >>> ProducerEpoch.
> > > > > > > > > > >>> > > When
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > > > > > > > >>> > > > > > > > >> > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> PrepareCommit/PrepareAbort,
> > > > > > > > > > >>> > > they
> > > > > > > > > > >>> > > > > > > > represent
> > > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > > > > > > > ProducerEpoch.
> > > > > > > > > > >>> When the
> > > > > > > > > > >>> > > > > > > > >> > TransactionStatus
> > > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > further
> > > > > depend
> > > > > > > on
> > > > > > > > > > >>> whether
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> overflows
> > > > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > > > >>> > > > > > > > >> > > > > not.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > If
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > overflow,
> > > > > they
> > > > > > > > > > represent
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > ProducerEpoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> (max).
> > > > > > > Otherwise,
> > > > > > > > > they
> > > > > > > > > > >>> > > represent
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > > > > newly
> > > > > > > > > > >>> > > > > > > > >> > > > > generated
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > ProducerEpoch
> > > > > > of
> > > > > > > > 0.
> > > > > > > > > Is
> > > > > > > > > > >>> that
> > > > > > > > > > >>> > > > right?
> > > > > > > > > > >>> > > > > > > This
> > > > > > > > > > >>> > > > > > > > >> seems
> > > > > > > > > > >>> > > > > > > > >> > > not
> > > > > > > > > > >>> > > > > > > > >> > > > > > easy
> > > > > > > > > > >>> > > > > > > > >> > > > > > > to
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we
> > > provide
> > > > > > some
> > > > > > > > > > examples
> > > > > > > > > > >>> > like
> > > > > > > > > > >>> > > > what
> > > > > > > > > > >>> > > > > > > Artem
> > > > > > > > > > >>> > > > > > > > >> has
> > > > > > > > > > >>> > > > > > > > >> > > done
> > > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > considered a
> > > > > > > > simpler
> > > > > > > > > > >>> design
> > > > > > > > > > >>> > > > where
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> ProducerId/ProducerEpoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > represent
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > same
> > > > > value
> > > > > > > > (e.g.
> > > > > > > > > > >>> for the
> > > > > > > > > > >>> > > > > current
> > > > > > > > > > >>> > > > > > > > >> > > transaction)
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > TransactionStatus
> > > > > > > > > and
> > > > > > > > > > >>> epoch
> > > > > > > > > > >>> > > > > > overflow?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103.
> > > It's
> > > > > not
> > > > > > > > clear
> > > > > > > > > to
> > > > > > > > > > >>> me
> > > > > > > > > > >>> > why
> > > > > > > > > > >>> > > we
> > > > > > > > > > >>> > > > > > need
> > > > > > > > > > >>> > > > > > > 3
> > > > > > > > > > >>> > > > > > > > >> > fields:
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > PrevProducerId,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > NextProducerId.
> > > > > > > > > Could
> > > > > > > > > > we
> > > > > > > > > > >>> > just
> > > > > > > > > > >>> > > > have
> > > > > > > > > > >>> > > > > > > > >> ProducerId
> > > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104.
> > For
> > > > > > > > > > >>> > > WriteTxnMarkerRequests,
> > > > > > > > > > >>> > > > > if
> > > > > > > > > > >>> > > > > > > the
> > > > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we
> set
> > > the
> > > > > > > > > producerId
> > > > > > > > > > >>> and
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > > > > > > > >> producerEpoch?
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > Thanks,
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > > >>> > > > > > > > >
> > > > > > > > > > >>> > > > > > > >
> > > > > > > > > > >>> > > > > > >
> > > > > > > > > > >>> > > > > >
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> >
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Calvin Liu <ca...@confluent.io.INVALID>]

Hi Jun,
Thanks for the feedback, will update the name as suggested.
The default values are
add.partitions.to.txn.retry.backoff.ms=20
-- This is because the client side AddPartitionsToTxn retry backoff is 20ms.
add.partitions.to.txn.retry.backoff.max.ms=100
-- We want this value to be small to avoid client side timeout and retry,
but also long enough for the markers to propagate.


On Tue, Feb 11, 2025 at 2:36 PM Jun Rao <ju...@confluent.io.invalid> wrote:

> Hi, Calvin,
>
> Thanks for the update. The change looks good to me. Some minor comments.
>
> For retry timeout, our convention is to use retry.backoff.max.ms and
> retry.backoff.ms. So, to be consistent, the new configurations should be
> add.partitions.to.txn.retry.backoff.max.ms and
> add.partitions.to.txn.retry.backoff.ms.
>
> Also, could you specify the defaults for these two values?
>
> Thanks,
>
> Jun
>
>
>
>
> On Mon, Feb 10, 2025 at 2:45 PM Calvin Liu <ca...@confluent.io.invalid>
> wrote:
>
> > Hey team,
> > In order to address the performance issue in the
> > https://issues.apache.org/jira/browse/KAFKA-18654. We are making a
> change
> > to retry the AddPartitionToTxn requests on the server side when hitting
> the
> > CONCURRENT_TRANSACTION error. This error is common after the transaction
> is
> > committed but before the markers are fully propagated to all the involved
> > partitions. With the server side retry, the client can avoid the frequent
> > and expensive retry for the produce request. In order to better control
> the
> > retry behavior, we proposed to add 2 following configs.
> >
> >    - *add.partitions.to.txn.max.timeout.ms
> >    <http://add.partitions.to.txn.max.timeout.ms>* defines the maximum
> > retry
> >    timeout when the server attempts to add the partition to the
> > transaction.
> >    - *add.partitions.to.txn.retry.backoff.ms
> >    <http://add.partitions.to.txn.retry.backoff.ms>* defines how
> frequently
> >    the server will retry the AddPartitionToTxn.
> >
> > Thanks!
> >
> > On Thu, Nov 14, 2024 at 1:52 PM Justine Olshan
> > <jo...@confluent.io.invalid>
> > wrote:
> >
> > > Hey folks, just wanted to share another quick update with KIP-890.
> > >
> > > Making the change to bump epoch after every transaction means that we
> no
> > > longer need to call InitProducerId during the life of the producer to
> > fence
> > > requests from previous transactions. With KIP-890 part 2, the new
> client
> > > will no longer call InitProducerId except on startup to fence a
> previous
> > > instance.
> > >
> > > With this change, some other calls to InitProducerId were inspected
> > > including the call after receiving an InvalidPidMappingException. This
> > > exception was changed to abortable as part of KIP-360: Improve
> > reliability
> > > of idempotent/transactional producer
> > > <
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820
> > > >.
> > > However, this change means that we can violate EOS guarantees. As an
> > > example:
> > >
> > > Consider an application that is copying data from one partition to
> > another
> > >
> > >    - Application instance A processes to offset 4
> > >    - Application instance B comes up and fences application instance A
> > >    - Application instance B processes to offset 5
> > >    - Application instances A and B are idle for
> > > transaction.id.expiration.ms,
> > >    transaction id expires on server
> > >    - Application instance A attempts to process offset 5 (since in its
> > >    view, that is next) -- if we recover from invalid pid mapping, we
> can
> > >    duplicate this processing
> > >
> > > Thus, INVALID_PID_MAPPING should be fatal to the producer and as part
> of
> > > KIP-890 the new client in 4.0 will treat the exception as fatal.
> > >
> > > This is consistent with KIP-1050: Consistent error handling for
> > > Transactions
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions
> > > >
> > > where
> > > errors that are fatal to the producer are in the "application
> > recoverable"
> > > category. This is a grouping that indicates to the client that the
> > producer
> > > needs to restart and recovery on the application side is necessary.
> > > KIP-1050 is approved so we are consistent with that decision.
> > >
> > > Let me know if there are any questions. I've updated the KIP to reflect
> > > this change.
> > >
> > > Justine
> > >
> > > On Wed, Jul 10, 2024 at 3:34 PM Jun Rao <ju...@confluent.io.invalid>
> > wrote:
> > >
> > > > Sounds good, Justine. It would be useful to document that in the KIP.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Jul 10, 2024 at 2:59 PM Justine Olshan
> > > > <jo...@confluent.io.invalid>
> > > > wrote:
> > > >
> > > > > The client will send the newest EndTxn request version if and only
> if
> > > > both
> > > > > the client and the server support kip-890 part 2.
> > > > > We set the value in the record based on the EndTxn version.
> > > > >
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <ju...@confluent.io.invalid>
> > > > wrote:
> > > > >
> > > > > > Hi, Justine,
> > > > > >
> > > > > > Thanks for the reply.
> > > > > >
> > > > > > 120. If the broker sends TV Y for the finalized version in
> > > > > > ApiVersionResponse, but the client doesn't support Y, how does
> the
> > > > broker
> > > > > > know the TV that the client supports?
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> > > > > > <jo...@confluent.io.invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Jun,
> > > > > > >
> > > > > > > No worries. Work on this KIP has been blocked for a bit anyways
> > --
> > > > > > catching
> > > > > > > up and rereading what I wrote :)
> > > > > > >
> > > > > > > 120. ClientTransactionProtocolVersion is the transaction
> version
> > as
> > > > > > defined
> > > > > > > by the highest transaction version (feature version value)
> > > supported
> > > > by
> > > > > > the
> > > > > > > client and the server. This works by the broker sending an
> > > > > > > ApiVersionsRequest to the client with the finalized version.
> > > Assuming
> > > > > > > kip-890 part 2 is enabled by transaction version Y, if this
> > request
> > > > > > > contains finalized version Y and the client has the logic to
> set
> > > this
> > > > > > > field, it will set Y. If the server has Y - 1 (kip 890 part 2
> not
> > > > > enable)
> > > > > > > the client will send Y - 1, even though the client has the
> > ability
> > > to
> > > > > > > support kip-890 part 2.
> > > > > > >
> > > > > > > 121. You are correct that this is not needed. However,
> currently
> > > that
> > > > > > field
> > > > > > > is already being set in memory -- just not written to disk. I
> > think
> > > > it
> > > > > is
> > > > > > > ok to write it to disk though. Let me know if you think
> > otherwise.
> > > > > > >
> > > > > > > Justine
> > > > > > >
> > > > > > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao
> <jun@confluent.io.invalid
> > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi, Justine,
> > > > > > > >
> > > > > > > > Thanks for the update and sorry for the late reply.
> > > > > > > >
> > > > > > > > 120. I am wondering what value is used for
> > > > > > > > ClientTransactionProtocolVersion. Is it the version of the
> > > > > > EndTxnRequest?
> > > > > > > >
> > > > > > > > 121. Earlier, you made the change to set lastProducerId in
> > > PREPARE
> > > > to
> > > > > > > > indicate that the marker is written for the new client. With
> > the
> > > > new
> > > > > > > > ClientTransactionProtocolVersion field, it seems this is no
> > > longer
> > > > > > > > necessary.
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi there -- another update!
> > > > > > > > >
> > > > > > > > > When looking into the implementation for the safe epoch
> > bumps I
> > > > > > > realized
> > > > > > > > > that we are already populating previousProducerID in memory
> > as
> > > > part
> > > > > > of
> > > > > > > > > KIP-360.
> > > > > > > > > If we are to start using flexible fields, it is better to
> > > always
> > > > > use
> > > > > > > this
> > > > > > > > > information and have an explicit (tagged) field to indicate
> > > > whether
> > > > > > the
> > > > > > > > > client supports KIP-890 part 2.
> > > > > > > > >
> > > > > > > > > I've included the extra field and how it is set in the KIP.
> > > I've
> > > > > also
> > > > > > > > > updated the KIP to explain that we will be setting the
> tagged
> > > > > fields
> > > > > > > when
> > > > > > > > > they are available for all transitions.
> > > > > > > > >
> > > > > > > > > Finally, I added clearer text about the transaction
> protocol
> > > > > versions
> > > > > > > > > included with this KIP. 1 for flexible transaction state
> > > records
> > > > > and
> > > > > > 2
> > > > > > > > for
> > > > > > > > > KIP-890 part 2 enablement.
> > > > > > > > >
> > > > > > > > > Justine
> > > > > > > > >
> > > > > > > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <
> > > > > jolshan@confluent.io
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hey there -- small update to the KIP,
> > > > > > > > > >
> > > > > > > > > > The KIP mentioned introducing ABORTABLE_ERROR and bumping
> > > > > > > > TxnOffsetCommit
> > > > > > > > > > and Produce requests. I've changed the name in the KIP to
> > > > > > > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > > > > > > AbortableTransactionException to match the pattern we had
> > for
> > > > > other
> > > > > > > > > errors.
> > > > > > > > > > I also mentioned bumping all 6 transactional APIs so we
> can
> > > > > future
> > > > > > > > > > proof/support the error on the client going forward. If a
> > > > future
> > > > > > > change
> > > > > > > > > > wants to have an error scenario that requires us to abort
> > the
> > > > > > > > > transaction,
> > > > > > > > > > we can rely on the 3.8+ clients to support it. We ran
> into
> > > > issues
> > > > > > > > finding
> > > > > > > > > > good/generic error codes that older clients could support
> > > while
> > > > > > > working
> > > > > > > > > on
> > > > > > > > > > this KIP, so this should help in the future.
> > > > > > > > > >
> > > > > > > > > > The features discussion is still ongoing in KIP-1022.
> Will
> > > > update
> > > > > > > again
> > > > > > > > > > here when that concludes.
> > > > > > > > > >
> > > > > > > > > > Justine
> > > > > > > > > >
> > > > > > > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> > > > > > jolshan@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> I don't think AddPartitions is a good example since we
> > > > currenly
> > > > > > > don't
> > > > > > > > > >> gate the version on TV or MV. (We only set a different
> > flag
> > > > > > > depending
> > > > > > > > on
> > > > > > > > > >> the TV)
> > > > > > > > > >>
> > > > > > > > > >> Even if we did want to gate it on TV, I think the idea
> is
> > to
> > > > > move
> > > > > > > away
> > > > > > > > > >> from MV gating inter broker protocols. Ideally we can
> get
> > > to a
> > > > > > state
> > > > > > > > > where
> > > > > > > > > >> MV is just used for metadata changes.
> > > > > > > > > >>
> > > > > > > > > >> I think some of this discussion might fit more with the
> > > > feature
> > > > > > > > version
> > > > > > > > > >> KIP, so I can try to open that up soon. Until we settle
> > > that,
> > > > > some
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > >> work in KIP-890 is blocked.
> > > > > > > > > >>
> > > > > > > > > >> Justine
> > > > > > > > > >>
> > > > > > > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao
> > > > <jun@confluent.io.invalid
> > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Hi, Justine,
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks for the reply.
> > > > > > > > > >>>
> > > > > > > > > >>> Since AddPartitions is an inter broker request, will
> its
> > > > > version
> > > > > > be
> > > > > > > > > gated
> > > > > > > > > >>> only by TV or other features like MV too? For example,
> if
> > > we
> > > > > need
> > > > > > > to
> > > > > > > > > >>> change
> > > > > > > > > >>> the protocol for AddPartitions for reasons other than
> txn
> > > > > > > > verification
> > > > > > > > > in
> > > > > > > > > >>> the future, will the new version be gated by a new MV?
> If
> > > so,
> > > > > > does
> > > > > > > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > > > > > > >>>
> > > > > > > > > >>> Jun
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > > >>> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> > One TV gates the flexible feature version (no rpcs
> > > > involved,
> > > > > > only
> > > > > > > > the
> > > > > > > > > >>> > transactional records that should only be gated by
> TV)
> > > > > > > > > >>> > Another TV gates the ability to turn on kip-890 part
> 2.
> > > > This
> > > > > > > would
> > > > > > > > > >>> gate the
> > > > > > > > > >>> > version of Produce and EndTxn (likely only used by
> > > > > > transactions),
> > > > > > > > and
> > > > > > > > > >>> > specifies a flag in AddPartitionsToTxn though the
> > version
> > > > is
> > > > > > > > already
> > > > > > > > > >>> used
> > > > > > > > > >>> > without TV.
> > > > > > > > > >>> >
> > > > > > > > > >>> > I think the only concern is the Produce request and
> we
> > > > could
> > > > > > > > consider
> > > > > > > > > >>> work
> > > > > > > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > > > > > > >>> >
> > > > > > > > > >>> > Justine
> > > > > > > > > >>> >
> > > > > > > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> > > > > > <jun@confluent.io.invalid
> > > > > > > >
> > > > > > > > > >>> wrote:
> > > > > > > > > >>> >
> > > > > > > > > >>> > > Hi, Justine,
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > Which PRC/record protocols will TV guard? Going
> > > forward,
> > > > > will
> > > > > > > > those
> > > > > > > > > >>> > > PRC/record protocols only be guarded by TV and not
> by
> > > > other
> > > > > > > > > features
> > > > > > > > > >>> like
> > > > > > > > > >>> > > MV?
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > Thanks,
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > Jun
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > > > > > > >>> > <jolshan@confluent.io.invalid
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > wrote:
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > > Hi Jun,
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > Sorry I think I misunderstood your question or
> > > answered
> > > > > > > > > >>> incorrectly.
> > > > > > > > > >>> > The
> > > > > > > > > >>> > > TV
> > > > > > > > > >>> > > > version should ideally be fully independent from
> > MV.
> > > > > > > > > >>> > > > At least for the changes I proposed, TV should
> not
> > > > affect
> > > > > > MV
> > > > > > > > and
> > > > > > > > > MV
> > > > > > > > > >>> > > should
> > > > > > > > > >>> > > > not affect TV/
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > I think if we downgrade TV, only that feature
> > should
> > > > > > > downgrade.
> > > > > > > > > >>> > Likewise
> > > > > > > > > >>> > > > the same with MV. The finalizedFeatures should
> just
> > > > > reflect
> > > > > > > the
> > > > > > > > > >>> feature
> > > > > > > > > >>> > > > downgrade we made.
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > I also plan to write a new KIP for managing the
> > disk
> > > > > format
> > > > > > > and
> > > > > > > > > >>> upgrade
> > > > > > > > > >>> > > > tool as we will need new flags to support these
> > > > features.
> > > > > > > That
> > > > > > > > > >>> should
> > > > > > > > > >>> > > help
> > > > > > > > > >>> > > > clarify some things.
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > Justine
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > > > > > > <jun@confluent.io.invalid
> > > > > > > > > >
> > > > > > > > > >>> > > wrote:
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > > Hi, Justine,
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > Thanks for the reply.
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > So, if we downgrade TV, we could implicitly
> > > downgrade
> > > > > > > another
> > > > > > > > > >>> feature
> > > > > > > > > >>> > > > (say
> > > > > > > > > >>> > > > > MV) that has dependency (e.g. RPC). What would
> we
> > > > > return
> > > > > > > for
> > > > > > > > > >>> > > > > FinalizedFeatures for MV in ApiVersionsResponse
> > in
> > > > that
> > > > > > > case?
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > Thanks,
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > Jun
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > > > > > > >>> > > > <jolshan@confluent.io.invalid
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > > wrote:
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > > Hey Jun,
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > > > Yes, the idea is that if we downgrade TV
> > > > (transaction
> > > > > > > > > version)
> > > > > > > > > >>> we
> > > > > > > > > >>> > > will
> > > > > > > > > >>> > > > > stop
> > > > > > > > > >>> > > > > > using the add partitions to txn optimization
> > and
> > > > stop
> > > > > > > > writing
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > > flexible
> > > > > > > > > >>> > > > > > feature version of the log.
> > > > > > > > > >>> > > > > > In the compatibility section I included some
> > > > > > explanations
> > > > > > > > on
> > > > > > > > > >>> how
> > > > > > > > > >>> > this
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > done.
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > > > Thanks,
> > > > > > > > > >>> > > > > > Justine
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > > > > > > >>> <ju...@confluent.io.invalid>
> > > > > > > > > >>> > > > > wrote:
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > > > > Hi, Justine,
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > > > Thanks for the update.
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > > > If we ever downgrade the transaction
> feature,
> > > any
> > > > > > > feature
> > > > > > > > > >>> > depending
> > > > > > > > > >>> > > > on
> > > > > > > > > >>> > > > > > > changes on top of those RPC/record
> > > > > > > > > >>> > > > > > >
> > (AddPartitionsToTxnRequest/TransactionLogValue)
> > > > > > changes
> > > > > > > > > made
> > > > > > > > > >>> in
> > > > > > > > > >>> > > > KIP-890
> > > > > > > > > >>> > > > > > > will be automatically downgraded too?
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > > > Jun
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine
> > Olshan
> > > > > > > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > > >>> > > > > > > wrote:
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > > > > Hey Jun,
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > > > I wanted to get back to you about your
> > > > questions
> > > > > > > about
> > > > > > > > > >>> MV/IBP.
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > > > Looking at the options, I think it makes
> > the
> > > > most
> > > > > > > sense
> > > > > > > > > to
> > > > > > > > > >>> > > create a
> > > > > > > > > >>> > > > > > > > separate feature for transactions and use
> > > that
> > > > to
> > > > > > > > version
> > > > > > > > > >>> gate
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > features
> > > > > > > > > >>> > > > > > > > we need to version gate (flexible
> > > transactional
> > > > > > state
> > > > > > > > > >>> records
> > > > > > > > > >>> > and
> > > > > > > > > >>> > > > > using
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > new protocol)
> > > > > > > > > >>> > > > > > > > I've updated the KIP to include this
> > change.
> > > > > > > Hopefully
> > > > > > > > > >>> that's
> > > > > > > > > >>> > > > > > everything
> > > > > > > > > >>> > > > > > > we
> > > > > > > > > >>> > > > > > > > need for this KIP :)
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > > > Justine
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine
> > > Olshan
> > > > <
> > > > > > > > > >>> > > > jolshan@confluent.io
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > > > > Thanks Jun,
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > > I will update the KIP with the prev
> field
> > > for
> > > > > > > prepare
> > > > > > > > > as
> > > > > > > > > >>> > well.
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > > PREPARE
> > > > > > > > > >>> > > > > > > > > producerId: x
> > > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged
> field): x
> > > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> or z
> > > if
> > > > y
> > > > > > will
> > > > > > > > > >>> overflow
> > > > > > > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > > COMPLETE
> > > > > > > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged
> field): x
> > > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > > > > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we
> > overflowed
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > > Thanks again,
> > > > > > > > > >>> > > > > > > > > Justine
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun Rao
> > > > > > > > > >>> > > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> Hi, Justine,
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > > > > > > >>> > > > > > > > >> (1) My point was that the coordinator
> > > could
> > > > > fail
> > > > > > > > right
> > > > > > > > > >>> after
> > > > > > > > > >>> > > > > writing
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> prepare marker. When the new txn
> > > coordinator
> > > > > > > > generates
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > > complete
> > > > > > > > > >>> > > > > > > > marker
> > > > > > > > > >>> > > > > > > > >> after the failover, it needs some
> field
> > > from
> > > > > the
> > > > > > > > > prepare
> > > > > > > > > >>> > > marker
> > > > > > > > > >>> > > > to
> > > > > > > > > >>> > > > > > > > >> determine whether it's written by the
> > new
> > > > > > client.
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >> (2) The changing of the behavior
> sounds
> > > good
> > > > > to
> > > > > > > me.
> > > > > > > > We
> > > > > > > > > >>> only
> > > > > > > > > >>> > > want
> > > > > > > > > >>> > > > > to
> > > > > > > > > >>> > > > > > > > return
> > > > > > > > > >>> > > > > > > > >> success if the prepare state is
> written
> > by
> > > > the
> > > > > > new
> > > > > > > > > >>> client.
> > > > > > > > > >>> > So,
> > > > > > > > > >>> > > > in
> > > > > > > > > >>> > > > > > the
> > > > > > > > > >>> > > > > > > > >> non-overflow case, it seems that we
> also
> > > > need
> > > > > > sth
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >>> > > prepare
> > > > > > > > > >>> > > > > > > marker
> > > > > > > > > >>> > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> tell us whether it's written by the
> new
> > > > > client.
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >> 112. Thanks for the explanation. That
> > > sounds
> > > > > > good
> > > > > > > to
> > > > > > > > > me.
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >> Jun
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM
> Justine
> > > > > Olshan
> > > > > > > > > >>> > > > > > > > >> <jo...@confluent.io.invalid> wrote:
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >> > 101.3 I realized that I actually
> have
> > > two
> > > > > > > > questions.
> > > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> > need
> > > to
> > > > > > write
> > > > > > > > the
> > > > > > > > > >>> > > previous
> > > > > > > > > >>> > > > > > > produce
> > > > > > > > > >>> > > > > > > > Id
> > > > > > > > > >>> > > > > > > > >> > tagged field in the end maker so
> that
> > we
> > > > > know
> > > > > > if
> > > > > > > > the
> > > > > > > > > >>> > marker
> > > > > > > > > >>> > > is
> > > > > > > > > >>> > > > > > from
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> new
> > > > > > > > > >>> > > > > > > > >> > client. Since the end maker is
> derived
> > > > from
> > > > > > the
> > > > > > > > > >>> prepare
> > > > > > > > > >>> > > > marker,
> > > > > > > > > >>> > > > > > > should
> > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > >>> > > > > > > > >> > write the previous produce Id in the
> > > > prepare
> > > > > > > > marker
> > > > > > > > > >>> field
> > > > > > > > > >>> > > too?
> > > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > > >>> > > > > > > > >> > we will lose this information when
> > > > deriving
> > > > > > the
> > > > > > > > end
> > > > > > > > > >>> > marker.
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > The "previous" producer ID is in the
> > > > normal
> > > > > > > > producer
> > > > > > > > > >>> ID
> > > > > > > > > >>> > > field.
> > > > > > > > > >>> > > > > So
> > > > > > > > > >>> > > > > > > yes,
> > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > >>> > > > > > > > >> > need it in prepare and that was
> always
> > > the
> > > > > > plan.
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I will
> > > > > enumerate
> > > > > > > the
> > > > > > > > > >>> fields
> > > > > > > > > >>> > and
> > > > > > > > > >>> > > > add
> > > > > > > > > >>> > > > > > > them
> > > > > > > > > >>> > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > > > > > > >>> > > > > > > > >> > Say we have producer ID x and epoch
> y.
> > > > When
> > > > > we
> > > > > > > > > >>> overflow
> > > > > > > > > >>> > > epoch
> > > > > > > > > >>> > > > y
> > > > > > > > > >>> > > > > we
> > > > > > > > > >>> > > > > > > get
> > > > > > > > > >>> > > > > > > > >> > producer ID Z.
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > PREPARE
> > > > > > > > > >>> > > > > > > > >> > producerId: x
> > > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> > field):
> > > > > empty
> > > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > or
> > > z
> > > > > if y
> > > > > > > > will
> > > > > > > > > >>> > overflow
> > > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > COMPLETE
> > > > > > > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> > field):
> > > x
> > > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we
> > > overflowed
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > (2) In the prepare phase, if we
> retry
> > > and
> > > > > see
> > > > > > > > epoch
> > > > > > > > > -
> > > > > > > > > >>> 1 +
> > > > > > > > > >>> > ID
> > > > > > > > > >>> > > > in
> > > > > > > > > >>> > > > > > last
> > > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > > >>> > > > > > > > >> > fields and are issuing the same
> > command
> > > > (ie
> > > > > > > commit
> > > > > > > > > not
> > > > > > > > > >>> > > abort),
> > > > > > > > > >>> > > > > we
> > > > > > > > > >>> > > > > > > > return
> > > > > > > > > >>> > > > > > > > >> > success. The logic before KIP-890
> > seems
> > > to
> > > > > > > return
> > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > >>> > > > > > > > >> > in this case. Are we intentionally
> > > making
> > > > > this
> > > > > > > > > change?
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > Hmm -- we would fence the producer
> if
> > > the
> > > > > > epoch
> > > > > > > is
> > > > > > > > > >>> bumped
> > > > > > > > > >>> > > and
> > > > > > > > > >>> > > > we
> > > > > > > > > >>> > > > > > > get a
> > > > > > > > > >>> > > > > > > > >> > lower epoch. Yes -- we are
> > intentionally
> > > > > > adding
> > > > > > > > this
> > > > > > > > > >>> to
> > > > > > > > > >>> > > > prevent
> > > > > > > > > >>> > > > > > > > fencing.
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > 112. We already merged the code that
> > > adds
> > > > > the
> > > > > > > > > >>> VerifyOnly
> > > > > > > > > >>> > > field
> > > > > > > > > >>> > > > > in
> > > > > > > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which is
> an
> > > > inter
> > > > > > > > broker
> > > > > > > > > >>> > request.
> > > > > > > > > >>> > > > It
> > > > > > > > > >>> > > > > > > seems
> > > > > > > > > >>> > > > > > > > >> that
> > > > > > > > > >>> > > > > > > > >> > we didn't bump up the IBP for that.
> Do
> > > you
> > > > > > know
> > > > > > > > why?
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > We no longer need IBP for all
> > > interbroker
> > > > > > > requests
> > > > > > > > > as
> > > > > > > > > >>> > > > > ApiVersions
> > > > > > > > > >>> > > > > > > > should
> > > > > > > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > > > > > > >>> > > > > > > > >> > We also handle unsupported version
> > > errors
> > > > > > > > correctly
> > > > > > > > > >>> if we
> > > > > > > > > >>> > > > > receive
> > > > > > > > > >>> > > > > > > them
> > > > > > > > > >>> > > > > > > > >> in
> > > > > > > > > >>> > > > > > > > >> > edge cases like upgrades/downgrades.
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > Justine
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM Jun
> > Rao
> > > > > > > > > >>> > > > > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > > > > >> wrote:
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> > > 101.3 I realized that I actually
> > have
> > > > two
> > > > > > > > > questions.
> > > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> > need
> > > to
> > > > > > write
> > > > > > > > the
> > > > > > > > > >>> > > previous
> > > > > > > > > >>> > > > > > > produce
> > > > > > > > > >>> > > > > > > > Id
> > > > > > > > > >>> > > > > > > > >> > > tagged field in the end maker so
> > that
> > > we
> > > > > > know
> > > > > > > if
> > > > > > > > > the
> > > > > > > > > >>> > > marker
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > > from
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > new
> > > > > > > > > >>> > > > > > > > >> > > client. Since the end maker is
> > derived
> > > > > from
> > > > > > > the
> > > > > > > > > >>> prepare
> > > > > > > > > >>> > > > > marker,
> > > > > > > > > >>> > > > > > > > >> should we
> > > > > > > > > >>> > > > > > > > >> > > write the previous produce Id in
> the
> > > > > prepare
> > > > > > > > > marker
> > > > > > > > > >>> > field
> > > > > > > > > >>> > > > too?
> > > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > > >>> > > > > > > > >> > > we will lose this information when
> > > > > deriving
> > > > > > > the
> > > > > > > > > end
> > > > > > > > > >>> > > marker.
> > > > > > > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we
> > retry
> > > > and
> > > > > > see
> > > > > > > > > epoch
> > > > > > > > > >>> - 1
> > > > > > > > > >>> > +
> > > > > > > > > >>> > > ID
> > > > > > > > > >>> > > > > in
> > > > > > > > > >>> > > > > > > last
> > > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > > >>> > > > > > > > >> > > fields and are issuing the same
> > > command
> > > > > (ie
> > > > > > > > commit
> > > > > > > > > >>> not
> > > > > > > > > >>> > > > abort),
> > > > > > > > > >>> > > > > > we
> > > > > > > > > >>> > > > > > > > >> return
> > > > > > > > > >>> > > > > > > > >> > > success. The logic before KIP-890
> > > seems
> > > > to
> > > > > > > > return
> > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > >>> > > > > > > > >> > > in this case. Are we intentionally
> > > > making
> > > > > > this
> > > > > > > > > >>> change?
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> > > 112. We already merged the code
> that
> > > > adds
> > > > > > the
> > > > > > > > > >>> VerifyOnly
> > > > > > > > > >>> > > > field
> > > > > > > > > >>> > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which
> is
> > an
> > > > > inter
> > > > > > > > > broker
> > > > > > > > > >>> > > request.
> > > > > > > > > >>> > > > > It
> > > > > > > > > >>> > > > > > > > seems
> > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > >>> > > > > > > > >> > > we didn't bump up the IBP for
> that.
> > Do
> > > > you
> > > > > > > know
> > > > > > > > > why?
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> > > Jun
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM
> > > Justine
> > > > > > Olshan
> > > > > > > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen"
> to
> > > > > "current
> > > > > > > > > >>> producer id
> > > > > > > > > >>> > > and
> > > > > > > > > >>> > > > > > > epoch"
> > > > > > > > > >>> > > > > > > > if
> > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > >>> > > > > > > > >> > > > was the part that was confusing
> > > > > > > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > > > > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > > > > > > >>> > > > > > > > >> > > > 112 We still need it. But I am
> > still
> > > > > > > > finalizing
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > design.
> > > > > > > > > >>> > > > > I
> > > > > > > > > >>> > > > > > > will
> > > > > > > > > >>> > > > > > > > >> > update
> > > > > > > > > >>> > > > > > > > >> > > > the KIP once I get the
> information
> > > > > > > finalized.
> > > > > > > > > >>> Sorry
> > > > > > > > > >>> > for
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > > delays.
> > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > >>> > > > > > > > >> > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at 10:50 AM
> > Jun
> > > > Rao
> > > > > > > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow
> case,
> > > the
> > > > > > > previous
> > > > > > > > > ID
> > > > > > > > > >>> is
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > same
> > > > > > > > > >>> > > > > > as
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > ID for the complete marker
> too,
> > > but
> > > > we
> > > > > > set
> > > > > > > > the
> > > > > > > > > >>> > > previous
> > > > > > > > > >>> > > > ID
> > > > > > > > > >>> > > > > > in
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > > >>> > > > > > > > >> > > > > marker. Earlier you mentioned
> > that
> > > > > this
> > > > > > is
> > > > > > > > to
> > > > > > > > > >>> know
> > > > > > > > > >>> > > that
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > > >>> > > > > > > > >> > > > > written by the new client so
> > that
> > > we
> > > > > > could
> > > > > > > > > >>> return
> > > > > > > > > >>> > > > success
> > > > > > > > > >>> > > > > on
> > > > > > > > > >>> > > > > > > > >> retried
> > > > > > > > > >>> > > > > > > > >> > > > > endMarker requests. I was
> trying
> > > to
> > > > > > > > understand
> > > > > > > > > >>> why
> > > > > > > > > >>> > > this
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > not
> > > > > > > > > >>> > > > > > > > >> needed
> > > > > > > > > >>> > > > > > > > >> > > for
> > > > > > > > > >>> > > > > > > > >> > > > > the prepare marker since retry
> > can
> > > > > > happen
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >>> > > prepare
> > > > > > > > > >>> > > > > > state
> > > > > > > > > >>> > > > > > > > >> too.
> > > > > > > > > >>> > > > > > > > >> > Is
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > reason that in the prepare
> > state,
> > > we
> > > > > > > return
> > > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > > >>> > > > > > > > >> > > > instead
> > > > > > > > > >>> > > > > > > > >> > > > > of success on retried endMaker
> > > > > requests?
> > > > > > > If
> > > > > > > > > so,
> > > > > > > > > >>> > should
> > > > > > > > > >>> > > > we
> > > > > > > > > >>> > > > > > > change
> > > > > > > > > >>> > > > > > > > >> "If
> > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID
> in
> > > last
> > > > > > seen
> > > > > > > > > >>> fields and
> > > > > > > > > >>> > > are
> > > > > > > > > >>> > > > > > > issuing
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > same
> > > > > > > > > >>> > > > > > > > >> > > > > command (ie commit not abort)
> we
> > > can
> > > > > > > return
> > > > > > > > > >>> (with
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > new
> > > > > > > > > >>> > > > > > > > epoch)"
> > > > > > > > > >>> > > > > > > > >> > > > > accordingly?
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP, a
> > > > delayed
> > > > > > > > endMaker
> > > > > > > > > >>> > request
> > > > > > > > > >>> > > > > > carries
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > > >>> > > > > > > > >> > > > > epoch and won't be fenced.
> This
> > > can
> > > > > > > > > >>> commit/abort a
> > > > > > > > > >>> > > > future
> > > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure if
> > we
> > > > have
> > > > > > > seen
> > > > > > > > > >>> this in
> > > > > > > > > >>> > > > > practice
> > > > > > > > > >>> > > > > > > > >> though.
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would be
> > > useful
> > > > > to
> > > > > > > make
> > > > > > > > > it
> > > > > > > > > >>> > clear
> > > > > > > > > >>> > > > that
> > > > > > > > > >>> > > > > > we
> > > > > > > > > >>> > > > > > > > can
> > > > > > > > > >>> > > > > > > > >> now
> > > > > > > > > >>> > > > > > > > >> > > > > populate the lastSeen field
> from
> > > the
> > > > > log
> > > > > > > > > >>> reliably.
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > > >>> > > > > since
> > > > > > > > > >>> > > > > > > > it's
> > > > > > > > > >>> > > > > > > > >> > > called
> > > > > > > > > >>> > > > > > > > >> > > > > across brokers and we are
> > changing
> > > > its
> > > > > > > > schema.
> > > > > > > > > >>> Are
> > > > > > > > > >>> > you
> > > > > > > > > >>> > > > > > saying
> > > > > > > > > >>> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > don't
> > > > > > > > > >>> > > > > > > > >> > > > need
> > > > > > > > > >>> > > > > > > > >> > > > > it any more? I thought that we
> > > > already
> > > > > > > > > >>> implemented
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > > server
> > > > > > > > > >>> > > > > > > > side
> > > > > > > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > > >>> > > > > across
> > > > > > > > > >>> > > > > > > > >> brokers.
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at
> 5:05 PM
> > > > > Justine
> > > > > > > > Olshan
> > > > > > > > > >>> > > > > > > > >> > > > > <jolshan@confluent.io.invalid
> >
> > > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the
> > previous
> > > ID
> > > > > in
> > > > > > > the
> > > > > > > > > >>> Prepare
> > > > > > > > > >>> > > > field
> > > > > > > > > >>> > > > > > > since
> > > > > > > > > >>> > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > don't
> > > > > > > > > >>> > > > > > > > >> > > > > need
> > > > > > > > > >>> > > > > > > > >> > > > > > it. It is the same producer
> ID
> > > as
> > > > > the
> > > > > > > main
> > > > > > > > > >>> > producer
> > > > > > > > > >>> > > ID
> > > > > > > > > >>> > > > > > > field.
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to
> > > reread
> > > > > your
> > > > > > > > > message
> > > > > > > > > >>> > about
> > > > > > > > > >>> > > > > > delayed
> > > > > > > > > >>> > > > > > > > >> > markers.
> > > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn
> > marker
> > > > > after
> > > > > > > the
> > > > > > > > > >>> > > transaction
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > > > already
> > > > > > > > > >>> > > > > > > > >> > > > > complete?
> > > > > > > > > >>> > > > > > > > >> > > > > > So we will commit the next
> > > > > transaction
> > > > > > > > early
> > > > > > > > > >>> > without
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > fixes
> > > > > > > > > >>> > > > > > > > >> in
> > > > > > > > > >>> > > > > > > > >> > > part
> > > > > > > > > >>> > > > > > > > >> > > > 2?
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > 111 Yes -- this terminology
> > was
> > > > used
> > > > > > in
> > > > > > > a
> > > > > > > > > >>> previous
> > > > > > > > > >>> > > KIP
> > > > > > > > > >>> > > > > and
> > > > > > > > > >>> > > > > > > > never
> > > > > > > > > >>> > > > > > > > >> > > > > > implemented it in the log --
> > > only
> > > > in
> > > > > > > > memory
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which interbroker
> > > > > protocol
> > > > > > > are
> > > > > > > > > you
> > > > > > > > > >>> > > > referring
> > > > > > > > > >>> > > > > > to?
> > > > > > > > > >>> > > > > > > I
> > > > > > > > > >>> > > > > > > > am
> > > > > > > > > >>> > > > > > > > >> > > > working
> > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > the design for the work to
> > > remove
> > > > > the
> > > > > > > > extra
> > > > > > > > > >>> add
> > > > > > > > > >>> > > > > partitions
> > > > > > > > > >>> > > > > > > > call
> > > > > > > > > >>> > > > > > > > >> > and I
> > > > > > > > > >>> > > > > > > > >> > > > > right
> > > > > > > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I
> > have
> > > > yet
> > > > > to
> > > > > > > > > update
> > > > > > > > > >>> that
> > > > > > > > > >>> > > > > section
> > > > > > > > > >>> > > > > > > as
> > > > > > > > > >>> > > > > > > > I
> > > > > > > > > >>> > > > > > > > >> > > > finalize
> > > > > > > > > >>> > > > > > > > >> > > > > > the design so please stay
> > tuned.
> > > > Was
> > > > > > > there
> > > > > > > > > >>> > anything
> > > > > > > > > >>> > > > else
> > > > > > > > > >>> > > > > > you
> > > > > > > > > >>> > > > > > > > >> > thought
> > > > > > > > > >>> > > > > > > > >> > > > > needed
> > > > > > > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at
> > 3:07 PM
> > > > Jun
> > > > > > Rao
> > > > > > > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > I don't see this create
> any
> > > > issue.
> > > > > > It
> > > > > > > > just
> > > > > > > > > >>> makes
> > > > > > > > > >>> > > it
> > > > > > > > > >>> > > > a
> > > > > > > > > >>> > > > > > bit
> > > > > > > > > >>> > > > > > > > >> hard to
> > > > > > > > > >>> > > > > > > > >> > > > > explain
> > > > > > > > > >>> > > > > > > > >> > > > > > > what this non-tagged
> produce
> > > id
> > > > > > field
> > > > > > > > > >>> means. We
> > > > > > > > > >>> > > are
> > > > > > > > > >>> > > > > > > > >> essentially
> > > > > > > > > >>> > > > > > > > >> > > > trying
> > > > > > > > > >>> > > > > > > > >> > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > combine two actions
> > > (completing
> > > > a
> > > > > > txn
> > > > > > > > and
> > > > > > > > > >>> init a
> > > > > > > > > >>> > > new
> > > > > > > > > >>> > > > > > > produce
> > > > > > > > > >>> > > > > > > > >> Id)
> > > > > > > > > >>> > > > > > > > >> > > in a
> > > > > > > > > >>> > > > > > > > >> > > > > > > single record. But, this
> may
> > > be
> > > > > fine
> > > > > > > > too.
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > A few other follow up
> > > comments.
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason
> > that
> > > we
> > > > > > only
> > > > > > > > set
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > previous
> > > > > > > > > >>> > > > > > > > >> produce id
> > > > > > > > > >>> > > > > > > > >> > > > > tagged
> > > > > > > > > >>> > > > > > > > >> > > > > > > field in the complete
> > marker,
> > > > but
> > > > > > not
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > >>> > > prepare
> > > > > > > > > >>> > > > > > > marker,
> > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > prepare state, we always
> > > return
> > > > > > > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > > > > > > >>> > > > > > on
> > > > > > > > > >>> > > > > > > > >> > retried
> > > > > > > > > >>> > > > > > > > >> > > > > > endMaker
> > > > > > > > > >>> > > > > > > > >> > > > > > > requests?
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > 110. "I believe your
> second
> > > > point
> > > > > is
> > > > > > > > > >>> mentioned
> > > > > > > > > >>> > in
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > KIP. I
> > > > > > > > > >>> > > > > > > > >> can
> > > > > > > > > >>> > > > > > > > >> > > add
> > > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > > >>> > > > > > > > >> > > > > > > text on
> > > > > > > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > The delayed message case
> > can
> > > > > also
> > > > > > > > > violate
> > > > > > > > > >>> EOS
> > > > > > > > > >>> > if
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > > delayed
> > > > > > > > > >>> > > > > > > > >> > > > message
> > > > > > > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > > > > > > addPartitionsToTxn
> > > > > > > > > >>> > request
> > > > > > > > > >>> > > > > comes
> > > > > > > > > >>> > > > > > > in.
> > > > > > > > > >>> > > > > > > > >> > > > > Effectively
> > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > may see a message from a
> > > > previous
> > > > > > > > > (aborted)
> > > > > > > > > >>> > > > > transaction
> > > > > > > > > >>> > > > > > > > become
> > > > > > > > > >>> > > > > > > > >> > part
> > > > > > > > > >>> > > > > > > > >> > > > of
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > The above is the case
> when a
> > > > > delayed
> > > > > > > > > >>> message is
> > > > > > > > > >>> > > > > appended
> > > > > > > > > >>> > > > > > > to
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > data
> > > > > > > > > >>> > > > > > > > >> > > > > > > partition. What I
> mentioned
> > > is a
> > > > > > > > slightly
> > > > > > > > > >>> > > different
> > > > > > > > > >>> > > > > case
> > > > > > > > > >>> > > > > > > > when
> > > > > > > > > >>> > > > > > > > >> a
> > > > > > > > > >>> > > > > > > > >> > > > delayed
> > > > > > > > > >>> > > > > > > > >> > > > > > > marker is appended to the
> > > > > > transaction
> > > > > > > > log
> > > > > > > > > >>> > > partition.
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once we
> > > move
> > > > > past
> > > > > > > the
> > > > > > > > > >>> Prepare
> > > > > > > > > >>> > > and
> > > > > > > > > >>> > > > > > > > Complete
> > > > > > > > > >>> > > > > > > > >> > > states,
> > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > don’t need to worry about
> > > > lastSeen
> > > > > > > > fields
> > > > > > > > > >>> and
> > > > > > > > > >>> > > clear
> > > > > > > > > >>> > > > > > them,
> > > > > > > > > >>> > > > > > > > just
> > > > > > > > > >>> > > > > > > > >> > > handle
> > > > > > > > > >>> > > > > > > > >> > > > > > state
> > > > > > > > > >>> > > > > > > > >> > > > > > > transitions as normal.".
> Is
> > > the
> > > > > > > lastSeen
> > > > > > > > > >>> field
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > same
> > > > > > > > > >>> > > > > > as
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field in
> > > > > > > > > >>> TransactionLogValue?
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > 112. Since the kip changes
> > the
> > > > > > > > > inter-broker
> > > > > > > > > >>> > > > protocol,
> > > > > > > > > >>> > > > > > > should
> > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > >>> > > > > > > > >> > > bump
> > > > > > > > > >>> > > > > > > > >> > > > up
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this
> > > feature
> > > > > only
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > >>> > KRaft
> > > > > > > > > >>> > > > > mode?
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at
> > > 11:13 AM
> > > > > > > Justine
> > > > > > > > > >>> Olshan
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > <jolshan@confluent.io.invalid
> > > >
> > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > I'm glad we are getting
> to
> > > > > > > convergence
> > > > > > > > > on
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > > design.
> > > > > > > > > >>> > > > > > :)
> > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > While I understand it
> > seems
> > > a
> > > > > > little
> > > > > > > > > >>> "weird".
> > > > > > > > > >>> > > I'm
> > > > > > > > > >>> > > > > not
> > > > > > > > > >>> > > > > > > sure
> > > > > > > > > >>> > > > > > > > >> what
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > benefit
> > > > > > > > > >>> > > > > > > > >> > > > > > > > of writing an extra
> record
> > > to
> > > > > the
> > > > > > > log.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool to
> > > > > describe
> > > > > > > > > >>> transactions
> > > > > > > > > >>> > > > won't
> > > > > > > > > >>> > > > > > > work
> > > > > > > > > >>> > > > > > > > >> (ie,
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > complete
> > > > > > > > > >>> > > > > > > > >> > > > > > > > state is needed to
> > calculate
> > > > the
> > > > > > > time
> > > > > > > > > >>> since
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > > > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > completed?)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > If we have a reason like
> > > this,
> > > > > it
> > > > > > is
> > > > > > > > > >>> enough to
> > > > > > > > > >>> > > > > > convince
> > > > > > > > > >>> > > > > > > me
> > > > > > > > > >>> > > > > > > > >> we
> > > > > > > > > >>> > > > > > > > >> > > need
> > > > > > > > > >>> > > > > > > > >> > > > > such
> > > > > > > > > >>> > > > > > > > >> > > > > > > an
> > > > > > > > > >>> > > > > > > > >> > > > > > > > extra record. It seems
> > like
> > > it
> > > > > > would
> > > > > > > > be
> > > > > > > > > >>> > > replacing
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > record
> > > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is this
> > > > correct?
> > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024 at
> > > > 5:14 PM
> > > > > > Jun
> > > > > > > > Rao
> > > > > > > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks for the
> > > explanation.
> > > > I
> > > > > > > > > >>> understand the
> > > > > > > > > >>> > > > > > intention
> > > > > > > > > >>> > > > > > > > >> now.
> > > > > > > > > >>> > > > > > > > >> > In
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > case, we set the
> > > non-tagged
> > > > > > field
> > > > > > > to
> > > > > > > > > >>> the old
> > > > > > > > > >>> > > pid
> > > > > > > > > >>> > > > > > (and
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> max
> > > > > > > > > >>> > > > > > > > >> > > > > epoch)
> > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > prepare marker so that
> > we
> > > > > could
> > > > > > > > > >>> correctly
> > > > > > > > > >>> > > write
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> marker to
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > data
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > partition if the
> broker
> > > > > > > downgrades.
> > > > > > > > > When
> > > > > > > > > >>> > > writing
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> complete
> > > > > > > > > >>> > > > > > > > >> > > > > marker,
> > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > know the marker has
> > > already
> > > > > been
> > > > > > > > > >>> written to
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > data
> > > > > > > > > >>> > > > > > > > >> > partition.
> > > > > > > > > >>> > > > > > > > >> > > > We
> > > > > > > > > >>> > > > > > > > >> > > > > > set
> > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to
> the
> > > new
> > > > > pid
> > > > > > to
> > > > > > > > > avoid
> > > > > > > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > client if the broker
> > > > > downgrades.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > The above seems to
> work.
> > > > It's
> > > > > > > just a
> > > > > > > > > bit
> > > > > > > > > >>> > > > > > inconsistent
> > > > > > > > > >>> > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> a
> > > > > > > > > >>> > > > > > > > >> > > > prepare
> > > > > > > > > >>> > > > > > > > >> > > > > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > and a complete marker
> to
> > > use
> > > > > > > > different
> > > > > > > > > >>> pids
> > > > > > > > > >>> > in
> > > > > > > > > >>> > > > > this
> > > > > > > > > >>> > > > > > > > >> special
> > > > > > > > > >>> > > > > > > > >> > > case.
> > > > > > > > > >>> > > > > > > > >> > > > > If
> > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrade with the
> > > complete
> > > > > > > marker,
> > > > > > > > it
> > > > > > > > > >>> seems
> > > > > > > > > >>> > > > that
> > > > > > > > > >>> > > > > we
> > > > > > > > > >>> > > > > > > > will
> > > > > > > > > >>> > > > > > > > >> > never
> > > > > > > > > >>> > > > > > > > >> > > > be
> > > > > > > > > >>> > > > > > > > >> > > > > > able
> > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > write the complete
> > marker
> > > > with
> > > > > > the
> > > > > > > > old
> > > > > > > > > >>> pid.
> > > > > > > > > >>> > > Not
> > > > > > > > > >>> > > > > sure
> > > > > > > > > >>> > > > > > > if
> > > > > > > > > >>> > > > > > > > it
> > > > > > > > > >>> > > > > > > > >> > > causes
> > > > > > > > > >>> > > > > > > > >> > > > > any
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems a
> > bit
> > > > > weird.
> > > > > > > > > >>> Instead of
> > > > > > > > > >>> > > > > writing
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > complete
> > > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > with the new pid,
> could
> > we
> > > > > write
> > > > > > > two
> > > > > > > > > >>> > records:
> > > > > > > > > >>> > > a
> > > > > > > > > >>> > > > > > > complete
> > > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > > >>> > > > > > > > >> > > > > with
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > old pid followed by a
> > > > > > > > > >>> TransactionLogValue
> > > > > > > > > >>> > with
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > new
> > > > > > > > > >>> > > > > > > > pid
> > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > >>> > > > > > > > >> > > an
> > > > > > > > > >>> > > > > > > > >> > > > > > empty
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > state? We could make
> the
> > > two
> > > > > > > records
> > > > > > > > > in
> > > > > > > > > >>> the
> > > > > > > > > >>> > > same
> > > > > > > > > >>> > > > > > batch
> > > > > > > > > >>> > > > > > > > so
> > > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > > >>> > > > > > > > >> > > > they
> > > > > > > > > >>> > > > > > > > >> > > > > > > will
> > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > added to the log
> > > atomically.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024
> at
> > > > > 5:40 PM
> > > > > > > > > Justine
> > > > > > > > > >>> > Olshan
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > <jolshan@confluent.io.invalid
> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare
> marker
> > > is
> > > > > > > written,
> > > > > > > > > >>> but the
> > > > > > > > > >>> > > > > endTxn
> > > > > > > > > >>> > > > > > > > >> response
> > > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > > >>> > > > > > > > >> > > > not
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > received
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > by the client when
> the
> > > > > server
> > > > > > > > > >>> downgrades
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare
> > marker
> > > is
> > > > > > > > written,
> > > > > > > > > >>> the
> > > > > > > > > >>> > > endTxn
> > > > > > > > > >>> > > > > > > > response
> > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > >>> > > > > > > > >> > > > > received
> > > > > > > > > >>> > > > > > > > >> > > > > > > by
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > client when the
> server
> > > > > > > downgrades.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > I think I am still a
> > > > little
> > > > > > > > > confused.
> > > > > > > > > >>> In
> > > > > > > > > >>> > > both
> > > > > > > > > >>> > > > of
> > > > > > > > > >>> > > > > > > these
> > > > > > > > > >>> > > > > > > > >> > cases,
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction log has
> > the
> > > > old
> > > > > > > > producer
> > > > > > > > > >>> ID.
> > > > > > > > > >>> > We
> > > > > > > > > >>> > > > > don't
> > > > > > > > > >>> > > > > > > > write
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > in the prepare
> > marker's
> > > > non
> > > > > > > tagged
> > > > > > > > > >>> fields.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > If the server
> > downgrades
> > > > > now,
> > > > > > it
> > > > > > > > > would
> > > > > > > > > >>> > read
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > > records
> > > > > > > > > >>> > > > > > > > >> not
> > > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields and the
> > complete
> > > > > marker
> > > > > > > > will
> > > > > > > > > >>> also
> > > > > > > > > >>> > > have
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > old
> > > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > ID.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > (If we had used the
> > new
> > > > > > producer
> > > > > > > > ID,
> > > > > > > > > >>> we
> > > > > > > > > >>> > > would
> > > > > > > > > >>> > > > > not
> > > > > > > > > >>> > > > > > > have
> > > > > > > > > >>> > > > > > > > >> > > > > > transactional
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > correctness since
> the
> > > > > producer
> > > > > > > id
> > > > > > > > > >>> doesn't
> > > > > > > > > >>> > > > match
> > > > > > > > > >>> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > state would not be
> > > correct
> > > > > on
> > > > > > > the
> > > > > > > > > data
> > > > > > > > > >>> > > > > partition.)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > In the overflow
> case,
> > > I'd
> > > > > > expect
> > > > > > > > the
> > > > > > > > > >>> > > following
> > > > > > > > > >>> > > > > to
> > > > > > > > > >>> > > > > > > > >> happen on
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > client
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > side
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we retry
> > > EndTxn
> > > > > --
> > > > > > it
> > > > > > > > is
> > > > > > > > > >>> the
> > > > > > > > > >>> > same
> > > > > > > > > >>> > > > > > > producer
> > > > > > > > > >>> > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > >>> > > > > > > > >> > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > -
> > > > > > > > > >>> > > > > > > > >> > > > > > > 1
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > this
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > would fence the
> > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't
> > retry
> > > > > > EndTxn
> > > > > > > > and
> > > > > > > > > >>> use
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > new
> > > > > > > > > >>> > > > > > > > >> producer id
> > > > > > > > > >>> > > > > > > > >> > > > which
> > > > > > > > > >>> > > > > > > > >> > > > > > > would
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > > > > > > InvalidPidMappingException
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have
> > > special
> > > > > > > handling
> > > > > > > > > for
> > > > > > > > > >>> > when
> > > > > > > > > >>> > > a
> > > > > > > > > >>> > > > > > server
> > > > > > > > > >>> > > > > > > > >> > > > downgrades.
> > > > > > > > > >>> > > > > > > > >> > > > > > When
> > > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > reconnects we could
> > get
> > > an
> > > > > API
> > > > > > > > > version
> > > > > > > > > >>> > > request
> > > > > > > > > >>> > > > > > > showing
> > > > > > > > > >>> > > > > > > > >> > > KIP-890
> > > > > > > > > >>> > > > > > > > >> > > > > > part 2
> > > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > not supported. In
> that
> > > > case,
> > > > > > we
> > > > > > > > can
> > > > > > > > > >>> call
> > > > > > > > > >>> > > > > > > > initProducerId
> > > > > > > > > >>> > > > > > > > >> to
> > > > > > > > > >>> > > > > > > > >> > > > abort
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In the
> > > > > overflow
> > > > > > > > case,
> > > > > > > > > >>> this
> > > > > > > > > >>> > > > > correctly
> > > > > > > > > >>> > > > > > > > gives
> > > > > > > > > >>> > > > > > > > >> > us a
> > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > I guess the
> > > corresponding
> > > > > case
> > > > > > > > would
> > > > > > > > > >>> be
> > > > > > > > > >>> > > where
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> *complete
> > > > > > > > > >>> > > > > > > > >> > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > *is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > written but the
> endTxn
> > > is
> > > > > not
> > > > > > > > > >>> received by
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > client
> > > > > > > > > >>> > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > > server
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This
> would
> > > > > result
> > > > > > in
> > > > > > > > the
> > > > > > > > > >>> > > > transaction
> > > > > > > > > >>> > > > > > > > >> > coordinator
> > > > > > > > > >>> > > > > > > > >> > > > > having
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old
> > one.
> > > > If
> > > > > > the
> > > > > > > > > client
> > > > > > > > > >>> > > > retries,
> > > > > > > > > >>> > > > > it
> > > > > > > > > >>> > > > > > > > will
> > > > > > > > > >>> > > > > > > > >> > > receive
> > > > > > > > > >>> > > > > > > > >> > > > > an
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > InvalidPidMappingException.
> > > > > > The
> > > > > > > > > >>> > > InitProducerId
> > > > > > > > > >>> > > > > > > > scenario
> > > > > > > > > >>> > > > > > > > >> > above
> > > > > > > > > >>> > > > > > > > >> > > > > would
> > > > > > > > > >>> > > > > > > > >> > > > > > > > help
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my
> > > > > compatibility
> > > > > > > > story
> > > > > > > > > is
> > > > > > > > > >>> > meant
> > > > > > > > > >>> > > > to
> > > > > > > > > >>> > > > > > > > support
> > > > > > > > > >>> > > > > > > > >> > > > > downgrades
> > > > > > > > > >>> > > > > > > > >> > > > > > > > server
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > side in keeping the
> > > > > > > transactional
> > > > > > > > > >>> > > correctness.
> > > > > > > > > >>> > > > > > > Keeping
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > > client
> > > > > > > > > >>> > > > > > > > >> > > > > > > from
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is
> not
> > > the
> > > > > > > > priority.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I
> can
> > > > also
> > > > > > add
> > > > > > > > text
> > > > > > > > > >>> in
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > KIP
> > > > > > > > > >>> > > > > > > about
> > > > > > > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > > > > > > >>> > > > > > > > >> > > > > > > if
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > think that fixes
> some
> > > edge
> > > > > > > cases.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12, 2024
> > at
> > > > > > 4:10 PM
> > > > > > > > Jun
> > > > > > > > > >>> Rao
> > > > > > > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the
> > reply.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we
> > don't
> > > > need
> > > > > > to
> > > > > > > > > >>> optimize
> > > > > > > > > >>> > for
> > > > > > > > > >>> > > > > > fencing
> > > > > > > > > >>> > > > > > > > >> during
> > > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Regarding
> > consistency,
> > > > > there
> > > > > > > are
> > > > > > > > > two
> > > > > > > > > >>> > > > possible
> > > > > > > > > >>> > > > > > > cases:
> > > > > > > > > >>> > > > > > > > >> (1)
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > prepare
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is written, but
> the
> > > > endTxn
> > > > > > > > > response
> > > > > > > > > >>> is
> > > > > > > > > >>> > not
> > > > > > > > > >>> > > > > > > received
> > > > > > > > > >>> > > > > > > > by
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > > client
> > > > > > > > > >>> > > > > > > > >> > > > > > > > when
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > server downgrades;
> > (2)
> > > > > the
> > > > > > > > > prepare
> > > > > > > > > >>> > marker
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > > > written,
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > > endTxn
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is received by the
> > > > client
> > > > > > when
> > > > > > > > the
> > > > > > > > > >>> > server
> > > > > > > > > >>> > > > > > > > downgrades.
> > > > > > > > > >>> > > > > > > > >> In
> > > > > > > > > >>> > > > > > > > >> > > (1),
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > client
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > will have the old
> > > > produce
> > > > > Id
> > > > > > > and
> > > > > > > > > in
> > > > > > > > > >>> (2),
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > client
> > > > > > > > > >>> > > > > > > > >> will
> > > > > > > > > >>> > > > > > > > >> > > have
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If we
> > > > > downgrade
> > > > > > > > right
> > > > > > > > > >>> after
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > prepare
> > > > > > > > > >>> > > > > > > > >> > marker,
> > > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > can't
> > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > consistent to both
> > (1)
> > > > and
> > > > > > (2)
> > > > > > > > > >>> since we
> > > > > > > > > >>> > > can
> > > > > > > > > >>> > > > > only
> > > > > > > > > >>> > > > > > > put
> > > > > > > > > >>> > > > > > > > >> one
> > > > > > > > > >>> > > > > > > > >> > > > value
> > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > existing produce
> Id
> > > > field.
> > > > > > > It's
> > > > > > > > > >>> also not
> > > > > > > > > >>> > > > clear
> > > > > > > > > >>> > > > > > > which
> > > > > > > > > >>> > > > > > > > >> case
> > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > So we could
> probably
> > > be
> > > > > > > > consistent
> > > > > > > > > >>> with
> > > > > > > > > >>> > > > either
> > > > > > > > > >>> > > > > > > case.
> > > > > > > > > >>> > > > > > > > >> By
> > > > > > > > > >>> > > > > > > > >> > > > putting
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in the
> > > > prepare
> > > > > > > > marker,
> > > > > > > > > >>> we
> > > > > > > > > >>> > are
> > > > > > > > > >>> > > > > > > consistent
> > > > > > > > > >>> > > > > > > > >> with
> > > > > > > > > >>> > > > > > > > >> > > > case
> > > > > > > > > >>> > > > > > > > >> > > > > > (2)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > it
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > also has the
> slight
> > > > > benefit
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > >>> > > produce
> > > > > > > > > >>> > > > > > field
> > > > > > > > > >>> > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > > > prepare
> > > > > > > > > >>> > > > > > > > >> > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > complete marker
> are
> > > > > > consistent
> > > > > > > > in
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > overflow
> > > > > > > > > >>> > > > > > > case.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12,
> 2024
> > > at
> > > > > > > 3:11 PM
> > > > > > > > > >>> Justine
> > > > > > > > > >>> > > > Olshan
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > <jolshan@confluent.io.invalid
> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the case you
> > > > > describe,
> > > > > > we
> > > > > > > > > would
> > > > > > > > > >>> > need
> > > > > > > > > >>> > > to
> > > > > > > > > >>> > > > > > have
> > > > > > > > > >>> > > > > > > a
> > > > > > > > > >>> > > > > > > > >> > delayed
> > > > > > > > > >>> > > > > > > > >> > > > > > > request,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > successful
> EndTxn,
> > > > and a
> > > > > > > > > >>> successful
> > > > > > > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > >>> > > > > > > > >> > > > > then
> > > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn
> > > request
> > > > > go
> > > > > > > > > through
> > > > > > > > > >>> for
> > > > > > > > > >>> > a
> > > > > > > > > >>> > > > > given
> > > > > > > > > >>> > > > > > > > >> producer.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to
> > figure
> > > > out
> > > > > > if
> > > > > > > it
> > > > > > > > > is
> > > > > > > > > >>> > > possible
> > > > > > > > > >>> > > > > for
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > client
> > > > > > > > > >>> > > > > > > > >> > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > a previous
> request
> > > is
> > > > > > > delayed
> > > > > > > > > >>> > somewhere.
> > > > > > > > > >>> > > > But
> > > > > > > > > >>> > > > > > > yes,
> > > > > > > > > >>> > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > > >>> > > > > > > > >> > > > > case
> > > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > think
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > would fence the
> > > > client.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the
> > overflow
> > > > > case.
> > > > > > > In
> > > > > > > > > the
> > > > > > > > > >>> > > overflow
> > > > > > > > > >>> > > > > > case,
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are
> > different
> > > on
> > > > > the
> > > > > > > > > marker
> > > > > > > > > >>> and
> > > > > > > > > >>> > on
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > transaction.
> > > > > > > > > >>> > > > > > > > >> > > > > > So
> > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to
> use
> > > the
> > > > > max
> > > > > > > > epoch
> > > > > > > > > >>> but
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > new
> > > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > > >>> > > > > > > > >> > > > > > > > start
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and
> > > epoch 0
> > > > > in
> > > > > > > the
> > > > > > > > > >>> > > > transactional
> > > > > > > > > >>> > > > > > > state.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the server
> > > > downgrade
> > > > > > > case,
> > > > > > > > we
> > > > > > > > > >>> want
> > > > > > > > > >>> > to
> > > > > > > > > >>> > > > see
> > > > > > > > > >>> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > >>> > > > > > > > >> > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > > >>> > > > > > > > >> > > > > > > > that
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > what the client
> > will
> > > > > have.
> > > > > > > If
> > > > > > > > we
> > > > > > > > > >>> > > complete
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> commit,
> > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > state is
> reloaded,
> > > we
> > > > > need
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > >>> > > > producer
> > > > > > > > > >>> > > > > ID
> > > > > > > > > >>> > > > > > > in
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > > >>> > > > > > > > >> > > > > so
> > > > > > > > > >>> > > > > > > > >> > > > > > > > there
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid
> > producer
> > > ID
> > > > > > > > mapping.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > The server
> > downgrade
> > > > > cases
> > > > > > > are
> > > > > > > > > >>> > > considering
> > > > > > > > > >>> > > > > > > > >> > transactional
> > > > > > > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing
> > from
> > > > > > previous
> > > > > > > > > >>> behavior
> > > > > > > > > >>> > --
> > > > > > > > > >>> > > > and
> > > > > > > > > >>> > > > > > are
> > > > > > > > > >>> > > > > > > > not
> > > > > > > > > >>> > > > > > > > >> > > > > concerned
> > > > > > > > > >>> > > > > > > > >> > > > > > > > about
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the
> > > safety
> > > > > from
> > > > > > > > > fencing
> > > > > > > > > >>> > > retries
> > > > > > > > > >>> > > > > (as
> > > > > > > > > >>> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> have
> > > > > > > > > >>> > > > > > > > >> > > > > > downgraded
> > > > > > > > > >>> > > > > > > > >> > > > > > > > so
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to
> > > > support).
> > > > > > > > Perhaps
> > > > > > > > > >>> this
> > > > > > > > > >>> > is
> > > > > > > > > >>> > > a
> > > > > > > > > >>> > > > > > trade
> > > > > > > > > >>> > > > > > > > off,
> > > > > > > > > >>> > > > > > > > >> > but
> > > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > > >>> > > > > > > > >> > > > > > think
> > > > > > > > > >>> > > > > > > > >> > > > > > > it
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client
> > > > > downgrades,
> > > > > > > it
> > > > > > > > > will
> > > > > > > > > >>> > have
> > > > > > > > > >>> > > > > > > restarted
> > > > > > > > > >>> > > > > > > > >> and
> > > > > > > > > >>> > > > > > > > >> > it
> > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > >>> > > > > > > > >> > > > > ok
> > > > > > > > > >>> > > > > > > > >> > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > have a new
> > producer
> > > ID
> > > > > > too).
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan 12,
> > 2024
> > > > at
> > > > > > > > 11:42 AM
> > > > > > > > > >>> Jun
> > > > > > > > > >>> > Rao
> > > > > > > > > >>> > > > > > > > >> > > > > > > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for the
> > > > reply.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If the
> > > marker
> > > > > is
> > > > > > > > > written
> > > > > > > > > >>> by
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > new
> > > > > > > > > >>> > > > > > > > >> client, we
> > > > > > > > > >>> > > > > > > > >> > > can
> > > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the last email
> > > > > guarantee
> > > > > > > > that
> > > > > > > > > >>> any
> > > > > > > > > >>> > > EndTxn
> > > > > > > > > >>> > > > > > > > requests
> > > > > > > > > >>> > > > > > > > >> > with
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from the same
> > > > producer
> > > > > > and
> > > > > > > > the
> > > > > > > > > >>> same
> > > > > > > > > >>> > > > > > > transaction.
> > > > > > > > > >>> > > > > > > > >> Then
> > > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > don't
> > > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > return a
> fenced
> > > > error
> > > > > > but
> > > > > > > > can
> > > > > > > > > >>> handle
> > > > > > > > > >>> > > > > > > gracefully
> > > > > > > > > >>> > > > > > > > as
> > > > > > > > > >>> > > > > > > > >> > > > > described
> > > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When a delayed
> > > > EndTnx
> > > > > > > > request
> > > > > > > > > is
> > > > > > > > > >>> > > > > processed,
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> txn
> > > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > > >>> > > > > > > > >> > > > > > could
> > > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next
> > txn.
> > > I
> > > > > > guess
> > > > > > > in
> > > > > > > > > >>> this
> > > > > > > > > >>> > case
> > > > > > > > > >>> > > > we
> > > > > > > > > >>> > > > > > > still
> > > > > > > > > >>> > > > > > > > >> > return
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > fenced
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed
> > > request?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry, my
> > > > > question
> > > > > > > was
> > > > > > > > > >>> > > inaccurate.
> > > > > > > > > >>> > > > > What
> > > > > > > > > >>> > > > > > > you
> > > > > > > > > >>> > > > > > > > >> > > > described
> > > > > > > > > >>> > > > > > > > >> > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > "The downgrade
> > > > > > > > compatibility I
> > > > > > > > > >>> > mention
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > that
> > > > > > > > > >>> > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > keep
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in
> the
> > > > main
> > > > > > > > > >>> (non-tagged)
> > > > > > > > > >>> > > > fields
> > > > > > > > > >>> > > > > as
> > > > > > > > > >>> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> did
> > > > > > > > > >>> > > > > > > > >> > > > before
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > code
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > server side."
> If
> > > we
> > > > > want
> > > > > > > to
> > > > > > > > do
> > > > > > > > > >>> this,
> > > > > > > > > >>> > > it
> > > > > > > > > >>> > > > > > seems
> > > > > > > > > >>> > > > > > > > >> that we
> > > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > > >>> > > > > > > > >> > > > > > > use
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > current
> produce
> > Id
> > > > and
> > > > > > max
> > > > > > > > > >>> epoch in
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > existing
> > > > > > > > > >>> > > > > > > > >> > > > producerId
> > > > > > > > > >>> > > > > > > > >> > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerEpoch
> > > fields
> > > > > for
> > > > > > > > both
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > prepare
> > > > > > > > > >>> > > > > > and
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > The downgrade
> > can
> > > > > happen
> > > > > > > > after
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > > complete
> > > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > > >>> > > > > > > > >> > > > > > written.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > With
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > you described,
> > the
> > > > > > > > downgraded
> > > > > > > > > >>> > > > coordinator
> > > > > > > > > >>> > > > > > will
> > > > > > > > > >>> > > > > > > > see
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of the
> > old
> > > > > one.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan
> 12,
> > > 2024
> > > > > at
> > > > > > > > > 10:44 AM
> > > > > > > > > >>> > > Justine
> > > > > > > > > >>> > > > > > > Olshan
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > <jolshan@confluent.io.invalid
> > > > > > > > > >
> > > > > > > > > >>> > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can update
> > the
> > > > > > > > > description.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe
> your
> > > > > second
> > > > > > > > point
> > > > > > > > > is
> > > > > > > > > >>> > > > mentioned
> > > > > > > > > >>> > > > > > in
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > KIP.
> > > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > > >>> > > > > > > > >> > > > > can
> > > > > > > > > >>> > > > > > > > >> > > > > > > add
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > more
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it
> is
> > > > > helpful.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The
> delayed
> > > > > message
> > > > > > > case
> > > > > > > > > can
> > > > > > > > > >>> > also
> > > > > > > > > >>> > > > > > violate
> > > > > > > > > >>> > > > > > > > EOS
> > > > > > > > > >>> > > > > > > > >> if
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > delayed
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in
> after
> > > the
> > > > > > next
> > > > > > > > > >>> > > > > addPartitionsToTxn
> > > > > > > > > >>> > > > > > > > >> request
> > > > > > > > > >>> > > > > > > > >> > > > comes
> > > > > > > > > >>> > > > > > > > >> > > > > > in.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a
> > > message
> > > > > > from a
> > > > > > > > > >>> previous
> > > > > > > > > >>> > > > > > (aborted)
> > > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > > > become
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > part
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > next
> > > transaction.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the
> marker
> > is
> > > > > > written
> > > > > > > > by
> > > > > > > > > >>> the
> > > > > > > > > >>> > new
> > > > > > > > > >>> > > > > > client,
> > > > > > > > > >>> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> can
> > > > > > > > > >>> > > > > > > > >> > > as I
> > > > > > > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email
> > > > guarantee
> > > > > > > that
> > > > > > > > > any
> > > > > > > > > >>> > EndTxn
> > > > > > > > > >>> > > > > > > requests
> > > > > > > > > >>> > > > > > > > >> with
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same
> > > producer
> > > > > and
> > > > > > > the
> > > > > > > > > same
> > > > > > > > > >>> > > > > > transaction.
> > > > > > > > > >>> > > > > > > > >> Then we
> > > > > > > > > >>> > > > > > > > >> > > > don't
> > > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced error
> > but
> > > > can
> > > > > > > > handle
> > > > > > > > > >>> > > gracefully
> > > > > > > > > >>> > > > > as
> > > > > > > > > >>> > > > > > > > >> described
> > > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't
> think
> > a
> > > > > > boolean
> > > > > > > is
> > > > > > > > > >>> useful
> > > > > > > > > >>> > > > since
> > > > > > > > > >>> > > > > it
> > > > > > > > > >>> > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > directly
> > > > > > > > > >>> > > > > > > > >> > > > > > > encoded
> > > > > > > > > >>> > > > > > > > >> > > > > > > > by
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence or
> > > lack
> > > > of
> > > > > > the
> > > > > > > > > >>> tagged
> > > > > > > > > >>> > > field
> > > > > > > > > >>> > > > > > being
> > > > > > > > > >>> > > > > > > > >> > written.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> prepare
> > > > > marker
> > > > > > we
> > > > > > > > > will
> > > > > > > > > >>> have
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > same
> > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In
> the
> > > > > Complete
> > > > > > > > state
> > > > > > > > > >>> we
> > > > > > > > > >>> > may
> > > > > > > > > >>> > > > not.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure
> > why
> > > > the
> > > > > > > > ongoing
> > > > > > > > > >>> state
> > > > > > > > > >>> > > > > matters
> > > > > > > > > >>> > > > > > > for
> > > > > > > > > >>> > > > > > > > >> this
> > > > > > > > > >>> > > > > > > > >> > > > KIP.
> > > > > > > > > >>> > > > > > > > >> > > > > It
> > > > > > > > > >>> > > > > > > > >> > > > > > > > does
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure
> > > what
> > > > > you
> > > > > > > are
> > > > > > > > > >>> > referring
> > > > > > > > > >>> > > to
> > > > > > > > > >>> > > > > > about
> > > > > > > > > >>> > > > > > > > >> > writing
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > previous
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the
> > > prepare
> > > > > > > marker.
> > > > > > > > > >>> This is
> > > > > > > > > >>> > > not
> > > > > > > > > >>> > > > in
> > > > > > > > > >>> > > > > > the
> > > > > > > > > >>> > > > > > > > >> KIP.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> > overflow
> > > > > case,
> > > > > > we
> > > > > > > > > >>> write the
> > > > > > > > > >>> > > > > > > > >> nextProducerId
> > > > > > > > > >>> > > > > > > > >> > in
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so
> we
> > > know
> > > > > > what
> > > > > > > we
> > > > > > > > > >>> > assigned
> > > > > > > > > >>> > > > when
> > > > > > > > > >>> > > > > > we
> > > > > > > > > >>> > > > > > > > >> reload
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we
> > > complete,
> > > > we
> > > > > > > > > >>> transition
> > > > > > > > > >>> > this
> > > > > > > > > >>> > > > ID
> > > > > > > > > >>> > > > > to
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> main
> > > > > > > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the
> > > previous
> > > > > > > producer
> > > > > > > > > ID
> > > > > > > > > >>> > field
> > > > > > > > > >>> > > > > filled
> > > > > > > > > >>> > > > > > > in.
> > > > > > > > > >>> > > > > > > > >> This
> > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > >>> > > > > > > > >> > > > so
> > > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > can
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry
> > case
> > > > the
> > > > > > > > > operation
> > > > > > > > > >>> > > > completed
> > > > > > > > > >>> > > > > > > > >> > successfully
> > > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > don't
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer.
> The
> > > > > > downgrade
> > > > > > > > > >>> > > compatibility
> > > > > > > > > >>> > > > I
> > > > > > > > > >>> > > > > > > > mention
> > > > > > > > > >>> > > > > > > > >> is
> > > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > keep
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer ID
> > and
> > > > > epoch
> > > > > > in
> > > > > > > > the
> > > > > > > > > >>> main
> > > > > > > > > >>> > > > > > > (non-tagged)
> > > > > > > > > >>> > > > > > > > >> > fields
> > > > > > > > > >>> > > > > > > > >> > > > as
> > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > did
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on the
> > > server
> > > > > > side.
> > > > > > > > If
> > > > > > > > > >>> the
> > > > > > > > > >>> > > server
> > > > > > > > > >>> > > > > > > > >> downgrades,
> > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > >>> > > > > > > > >> > > > are
> > > > > > > > > >>> > > > > > > > >> > > > > > > still
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> addresses
> > > > both
> > > > > > the
> > > > > > > > > >>> prepare
> > > > > > > > > >>> > and
> > > > > > > > > >>> > > > > > complete
> > > > > > > > > >>> > > > > > > > >> state
> > > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri, Jan
> > 12,
> > > > 2024
> > > > > > at
> > > > > > > > > >>> 10:21 AM
> > > > > > > > > >>> > Jun
> > > > > > > > > >>> > > > Rao
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi,
> Justine,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks for
> > the
> > > > > > reply.
> > > > > > > > > Sorry
> > > > > > > > > >>> for
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > delay. I
> > > > > > > > > >>> > > > > > > > >> > have a
> > > > > > > > > >>> > > > > > > > >> > > > few
> > > > > > > > > >>> > > > > > > > >> > > > > > > more
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I
> think
> > > the
> > > > > > > > > motivation
> > > > > > > > > >>> > > section
> > > > > > > > > >>> > > > > > could
> > > > > > > > > >>> > > > > > > be
> > > > > > > > > >>> > > > > > > > >> > > > improved.
> > > > > > > > > >>> > > > > > > > >> > > > > > One
> > > > > > > > > >>> > > > > > > > >> > > > > > > of
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> motivations
> > > > listed
> > > > > > by
> > > > > > > > the
> > > > > > > > > >>> KIP is
> > > > > > > > > >>> > > > "This
> > > > > > > > > >>> > > > > > can
> > > > > > > > > >>> > > > > > > > >> happen
> > > > > > > > > >>> > > > > > > > >> > > > when
> > > > > > > > > >>> > > > > > > > >> > > > > a
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > message
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or delayed
> > due
> > > > to
> > > > > > > > > networking
> > > > > > > > > >>> > > issues
> > > > > > > > > >>> > > > > or a
> > > > > > > > > >>> > > > > > > > >> network
> > > > > > > > > >>> > > > > > > > >> > > > > > partition,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts,
> and
> > > then
> > > > > the
> > > > > > > > > delayed
> > > > > > > > > >>> > > message
> > > > > > > > > >>> > > > > > > finally
> > > > > > > > > >>> > > > > > > > >> > comes
> > > > > > > > > >>> > > > > > > > >> > > > > in.".
> > > > > > > > > >>> > > > > > > > >> > > > > > > This
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very
> > accurate.
> > > > > > Without
> > > > > > > > > >>> KIP-890,
> > > > > > > > > >>> > > > > > currently,
> > > > > > > > > >>> > > > > > > > if
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts an
> > > > ongoing
> > > > > > > > > >>> transaction,
> > > > > > > > > >>> > it
> > > > > > > > > >>> > > > > > already
> > > > > > > > > >>> > > > > > > > >> bumps
> > > > > > > > > >>> > > > > > > > >> > up
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which
> > prevents
> > > > the
> > > > > > > > delayed
> > > > > > > > > >>> > produce
> > > > > > > > > >>> > > > > > message
> > > > > > > > > >>> > > > > > > > >> from
> > > > > > > > > >>> > > > > > > > >> > > being
> > > > > > > > > >>> > > > > > > > >> > > > > > added
> > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > partition.
> > > What
> > > > > can
> > > > > > > > cause
> > > > > > > > > a
> > > > > > > > > >>> > > hanging
> > > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > > >>> > > > > > > > >> > > > that
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completes
> > > > (either
> > > > > > > aborts
> > > > > > > > > or
> > > > > > > > > >>> > > > commits) a
> > > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > before
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > successful
> > ack
> > > > on
> > > > > > > > messages
> > > > > > > > > >>> > > published
> > > > > > > > > >>> > > > > in
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> same
> > > > > > > > > >>> > > > > > > > >> > > txn.
> > > > > > > > > >>> > > > > > > > >> > > > > In
> > > > > > > > > >>> > > > > > > > >> > > > > > > this
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible
> for
> > > the
> > > > > > > delayed
> > > > > > > > > >>> message
> > > > > > > > > >>> > > to
> > > > > > > > > >>> > > > be
> > > > > > > > > >>> > > > > > > > >> appended
> > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker,
> > > causing
> > > > a
> > > > > > > > > >>> transaction to
> > > > > > > > > >>> > > > hang.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A similar
> > > issue
> > > > > (not
> > > > > > > > > >>> mentioned
> > > > > > > > > >>> > in
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > > >> motivation)
> > > > > > > > > >>> > > > > > > > >> > > > could
> > > > > > > > > >>> > > > > > > > >> > > > > > > > happen
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker in
> > the
> > > > > > > > > coordinator's
> > > > > > > > > >>> log.
> > > > > > > > > >>> > > For
> > > > > > > > > >>> > > > > > > > example,
> > > > > > > > > >>> > > > > > > > >> > it's
> > > > > > > > > >>> > > > > > > > >> > > > > > possible
> > > > > > > > > >>> > > > > > > > >> > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > EndTxnRequest
> > > to
> > > > > be
> > > > > > > > > delayed
> > > > > > > > > >>> on
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > > > > >> coordinator.
> > > > > > > > > >>> > > > > > > > >> > By
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > time
> > > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > EndTxnRequest
> > > is
> > > > > > > > > processed,
> > > > > > > > > >>> it's
> > > > > > > > > >>> > > > > > possible
> > > > > > > > > >>> > > > > > > > that
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > > >>> > > > > > > > >> > > > > > > > txn
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completed
> > and
> > > a
> > > > > new
> > > > > > > txn
> > > > > > > > > has
> > > > > > > > > >>> > > started.
> > > > > > > > > >>> > > > > > > > >> Currently,
> > > > > > > > > >>> > > > > > > > >> > > since
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped on
> > > every
> > > > > txn,
> > > > > > > the
> > > > > > > > > >>> delayed
> > > > > > > > > >>> > > > > > > > EndTxnRequest
> > > > > > > > > >>> > > > > > > > >> > will
> > > > > > > > > >>> > > > > > > > >> > > > add
> > > > > > > > > >>> > > > > > > > >> > > > > > an
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare
> > marker
> > > > > (and
> > > > > > > > > >>> eventually a
> > > > > > > > > >>> > > > > > complete
> > > > > > > > > >>> > > > > > > > >> marker)
> > > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't
> cause
> > > the
> > > > > > > > > transaction
> > > > > > > > > >>> to
> > > > > > > > > >>> > > hang,
> > > > > > > > > >>> > > > > but
> > > > > > > > > >>> > > > > > > it
> > > > > > > > > >>> > > > > > > > >> will
> > > > > > > > > >>> > > > > > > > >> > > > break
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The
> proposal
> > > in
> > > > > this
> > > > > > > KIP
> > > > > > > > > >>> will
> > > > > > > > > >>> > > > address
> > > > > > > > > >>> > > > > > this
> > > > > > > > > >>> > > > > > > > >> issue
> > > > > > > > > >>> > > > > > > > >> > > too.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.
> > > "However, I
> > > > > was
> > > > > > > > > >>> writing it
> > > > > > > > > >>> > so
> > > > > > > > > >>> > > > > that
> > > > > > > > > >>> > > > > > we
> > > > > > > > > >>> > > > > > > > can
> > > > > > > > > >>> > > > > > > > >> > > > > > distinguish
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old
> clients
> > > > where
> > > > > we
> > > > > > > > don't
> > > > > > > > > >>> have
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > ability
> > > > > > > > > >>> > > > > > > > do
> > > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > > >>> > > > > > > > >> > > > > > > operation
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients
> that
> > > > can.
> > > > > > (Old
> > > > > > > > > >>> clients
> > > > > > > > > >>> > > don't
> > > > > > > > > >>> > > > > > bump
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > epoch
> > > > > > > > > >>> > > > > > > > >> > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > so
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for
> sure
> > > the
> > > > > > write
> > > > > > > > > >>> belongs
> > > > > > > > > >>> > to
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > given
> > > > > > > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I am
> > > > > wondering
> > > > > > > why
> > > > > > > > > we
> > > > > > > > > >>> need
> > > > > > > > > >>> > > to
> > > > > > > > > >>> > > > > > > > >> distinguish
> > > > > > > > > >>> > > > > > > > >> > > > whether
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written by
> > the
> > > > old
> > > > > > and
> > > > > > > > the
> > > > > > > > > >>> new
> > > > > > > > > >>> > > > client.
> > > > > > > > > >>> > > > > > > Could
> > > > > > > > > >>> > > > > > > > >> you
> > > > > > > > > >>> > > > > > > > >> > > > > describe
> > > > > > > > > >>> > > > > > > > >> > > > > > > > what
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> differently
> > if
> > > > we
> > > > > > know
> > > > > > > > the
> > > > > > > > > >>> > marker
> > > > > > > > > >>> > > is
> > > > > > > > > >>> > > > > > > written
> > > > > > > > > >>> > > > > > > > >> by
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > client?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If
> we
> > do
> > > > > need
> > > > > > a
> > > > > > > > way
> > > > > > > > > to
> > > > > > > > > >>> > > > > distinguish
> > > > > > > > > >>> > > > > > > > >> whether
> > > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old
> and
> > > the
> > > > > new
> > > > > > > > > client.
> > > > > > > > > >>> > Would
> > > > > > > > > >>> > > it
> > > > > > > > > >>> > > > > be
> > > > > > > > > >>> > > > > > > > >> simpler
> > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > >>> > > > > > > > >> > > > just
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field
> > instead
> > > of
> > > > > > > > > indirectly
> > > > > > > > > >>> > > through
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> previous
> > > > > > > > > >>> > > > > > > > >> > > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3 It's
> > not
> > > > > clear
> > > > > > > to
> > > > > > > > me
> > > > > > > > > >>> why
> > > > > > > > > >>> > we
> > > > > > > > > >>> > > > only
> > > > > > > > > >>> > > > > > add
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> complete
> > > > > marker,
> > > > > > > but
> > > > > > > > > >>> not in
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > > prepare
> > > > > > > > > >>> > > > > > > > >> > marker.
> > > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > want
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether a
> > > marker
> > > > > is
> > > > > > > > > written
> > > > > > > > > >>> by
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > new
> > > > > > > > > >>> > > > > > > > client
> > > > > > > > > >>> > > > > > > > >> or
> > > > > > > > > >>> > > > > > > > >> > > not,
> > > > > > > > > >>> > > > > > > > >> > > > > it
> > > > > > > > > >>> > > > > > > > >> > > > > > > > seems
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do this
> > > > > > > consistently
> > > > > > > > > for
> > > > > > > > > >>> all
> > > > > > > > > >>> > > > > markers.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4 What
> > > about
> > > > > the
> > > > > > > > > >>> > > > > TransactionLogValue
> > > > > > > > > >>> > > > > > > > >> record
> > > > > > > > > >>> > > > > > > > >> > > > > > > representing
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state?
> > Should
> > > we
> > > > > > also
> > > > > > > > > >>> > distinguish
> > > > > > > > > >>> > > > > > whether
> > > > > > > > > >>> > > > > > > > it's
> > > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > > >>> > > > > > > > >> > > > > by
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > old
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new
> client?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In
> the
> > > > > overflow
> > > > > > > > case,
> > > > > > > > > >>> it's
> > > > > > > > > >>> > > > still
> > > > > > > > > >>> > > > > > not
> > > > > > > > > >>> > > > > > > > >> clear
> > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > >>> > > > > > > > >> > > me
> > > > > > > > > >>> > > > > > > > >> > > > > why
> > > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous
> > > produce
> > > > > Id
> > > > > > in
> > > > > > > > the
> > > > > > > > > >>> > prepare
> > > > > > > > > >>> > > > > > marker
> > > > > > > > > >>> > > > > > > > >> while
> > > > > > > > > >>> > > > > > > > >> > > > writing
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the
> > > complete
> > > > > > > marker.
> > > > > > > > > You
> > > > > > > > > >>> > > > mentioned
> > > > > > > > > >>> > > > > > that
> > > > > > > > > >>> > > > > > > > >> it's
> > > > > > > > > >>> > > > > > > > >> > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could
> > > > downgrade
> > > > > > > with
> > > > > > > > > >>> either
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > > prepare
> > > > > > > > > >>> > > > > > > > >> marker
> > > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In either
> > > case,
> > > > > the
> > > > > > > > > >>> downgraded
> > > > > > > > > >>> > > > > > coordinator
> > > > > > > > > >>> > > > > > > > >> should
> > > > > > > > > >>> > > > > > > > >> > > see
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > (probably
> > the
> > > > > > previous
> > > > > > > > > >>> produce
> > > > > > > > > >>> > > Id),
> > > > > > > > > >>> > > > > > right?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed,
> Dec
> > > 20,
> > > > > 2023
> > > > > > > at
> > > > > > > > > >>> 6:00 PM
> > > > > > > > > >>> > > > > Justine
> > > > > > > > > >>> > > > > > > > Olshan
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey Jun,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks
> for
> > > > > taking
> > > > > > a
> > > > > > > > look
> > > > > > > > > >>> at
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > KIP
> > > > > > > > > >>> > > > > > > again.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100. For
> > the
> > > > > epoch
> > > > > > > > > >>> overflow
> > > > > > > > > >>> > > case,
> > > > > > > > > >>> > > > > only
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > > >>> > > > > > > > >> > > > > will
> > > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > max
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps
> the
> > > > > behavior
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > >>> rest
> > > > > > > > > >>> > > of
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> markers
> > > > > > > > > >>> > > > > > > > >> > > where
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > last
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch of
> > the
> > > > > > > > transaction
> > > > > > > > > >>> > > records +
> > > > > > > > > >>> > > > > 1.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101. You
> > are
> > > > > > correct
> > > > > > > > > that
> > > > > > > > > >>> we
> > > > > > > > > >>> > > don't
> > > > > > > > > >>> > > > > > need
> > > > > > > > > >>> > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > write
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the
> > same.
> > > > > > > However,
> > > > > > > > I
> > > > > > > > > >>> was
> > > > > > > > > >>> > > > writing
> > > > > > > > > >>> > > > > it
> > > > > > > > > >>> > > > > > > so
> > > > > > > > > >>> > > > > > > > >> that
> > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > >>> > > > > > > > >> > > > can
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old
> > clients
> > > > > where
> > > > > > we
> > > > > > > > > don't
> > > > > > > > > >>> > have
> > > > > > > > > >>> > > > the
> > > > > > > > > >>> > > > > > > > ability
> > > > > > > > > >>> > > > > > > > >> do
> > > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > > >>> > > > > > > > >> > > > > > > > operation
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > clients
> > that
> > > > > can.
> > > > > > > (Old
> > > > > > > > > >>> clients
> > > > > > > > > >>> > > > don't
> > > > > > > > > >>> > > > > > > bump
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say for
> > sure
> > > > the
> > > > > > > write
> > > > > > > > > >>> belongs
> > > > > > > > > >>> > > to
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > given
> > > > > > > > > >>> > > > > > > > >> > > > > > transaction).
> > > > > > > > > >>> > > > > > > > >> > > > > > > > If
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn
> > > request
> > > > > > from
> > > > > > > a
> > > > > > > > > new
> > > > > > > > > >>> > > client,
> > > > > > > > > >>> > > > we
> > > > > > > > > >>> > > > > > > will
> > > > > > > > > >>> > > > > > > > >> fill
> > > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > > >>> > > > > > > > >> > > > > > > field.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > We
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > guarantee
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that any
> > > > EndTxn
> > > > > > > > requests
> > > > > > > > > >>> with
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > same
> > > > > > > > > >>> > > > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > are
> > > > > > > > > >>> > > > > > > > >> > > > from
> > > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the same
> > > > > > > transaction.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102. In
> > > > prepare
> > > > > > > phase,
> > > > > > > > > we
> > > > > > > > > >>> have
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > same
> > > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had. It
> is
> > > the
> > > > > > > > producer
> > > > > > > > > >>> ID and
> > > > > > > > > >>> > > > epoch
> > > > > > > > > >>> > > > > > > that
> > > > > > > > > >>> > > > > > > > >> are
> > > > > > > > > >>> > > > > > > > >> > on
> > > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > In
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase,
> we
> > > stay
> > > > > the
> > > > > > > > same
> > > > > > > > > >>> unless
> > > > > > > > > >>> > > it
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > overflow
> > > > > > > > > >>> > > > > > > > >> > > > > case.
> > > > > > > > > >>> > > > > > > > >> > > > > > > In
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > that
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set the
> > > > producer
> > > > > > ID
> > > > > > > to
> > > > > > > > > >>> the new
> > > > > > > > > >>> > > one
> > > > > > > > > >>> > > > > we
> > > > > > > > > >>> > > > > > > > >> generated
> > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> complete.
> > > This
> > > > > is
> > > > > > > for
> > > > > > > > > >>> > downgrade
> > > > > > > > > >>> > > > > > > > >> compatibility.
> > > > > > > > > >>> > > > > > > > >> > > The
> > > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety
> > > guards
> > > > > for
> > > > > > > > > retries
> > > > > > > > > >>> and
> > > > > > > > > >>> > > > > > failovers.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In
> prepare
> > > > phase
> > > > > > for
> > > > > > > > > epoch
> > > > > > > > > >>> > > > overflow
> > > > > > > > > >>> > > > > > case
> > > > > > > > > >>> > > > > > > > >> only
> > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > >>> > > > > > > > >> > > > > store
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID. This
> > is
> > > > for
> > > > > > the
> > > > > > > > case
> > > > > > > > > >>> where
> > > > > > > > > >>> > > we
> > > > > > > > > >>> > > > > > reload
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > transaction
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > prepare
> > > state.
> > > > > > Once
> > > > > > > > the
> > > > > > > > > >>> > > > transaction
> > > > > > > > > >>> > > > > is
> > > > > > > > > >>> > > > > > > > >> > committed,
> > > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > can
> > > > > > > > > >>> > > > > > > > >> > > > > > > > use
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the
> > > client
> > > > > > > already
> > > > > > > > is
> > > > > > > > > >>> > using.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In
> commit
> > > > phase,
> > > > > > we
> > > > > > > > > store
> > > > > > > > > >>> the
> > > > > > > > > >>> > > > > previous
> > > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > > >>> > > > > > > > >> > > > > > > > of
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I think
> it
> > > is
> > > > > > easier
> > > > > > > > to
> > > > > > > > > >>> think
> > > > > > > > > >>> > of
> > > > > > > > > >>> > > > it
> > > > > > > > > >>> > > > > as
> > > > > > > > > >>> > > > > > > > just
> > > > > > > > > >>> > > > > > > > >> how
> > > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > > >>> > > > > > > > >> > > > > were
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> epoch
> > > > > before,
> > > > > > > with
> > > > > > > > > >>> some
> > > > > > > > > >>> > > extra
> > > > > > > > > >>> > > > > > > > bookeeping
> > > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > > >>> > > > > > > > >> > > > edge
> > > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged
> > > fields.
> > > > > We
> > > > > > > have
> > > > > > > > > to
> > > > > > > > > >>> do
> > > > > > > > > >>> > it
> > > > > > > > > >>> > > > this
> > > > > > > > > >>> > > > > > way
> > > > > > > > > >>> > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > with
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> downgrades.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103.
> Next
> > > > > producer
> > > > > > > ID
> > > > > > > > is
> > > > > > > > > >>> for
> > > > > > > > > >>> > > > prepare
> > > > > > > > > >>> > > > > > > > status
> > > > > > > > > >>> > > > > > > > >> and
> > > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after
> > > > complete.
> > > > > > The
> > > > > > > > > >>> reason why
> > > > > > > > > >>> > > we
> > > > > > > > > >>> > > > > need
> > > > > > > > > >>> > > > > > > two
> > > > > > > > > >>> > > > > > > > >> > > separate
> > > > > > > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> backwards
> > > > > > > > compatibility.
> > > > > > > > > >>> We
> > > > > > > > > >>> > need
> > > > > > > > > >>> > > > to
> > > > > > > > > >>> > > > > > keep
> > > > > > > > > >>> > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > > >>> > > > > > > > >> > > > > > > semantics
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > for
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> non-tagged
> > > > field
> > > > > > in
> > > > > > > > case
> > > > > > > > > >>> we
> > > > > > > > > >>> > > > > downgrade.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104. We
> > set
> > > > the
> > > > > > > fields
> > > > > > > > > as
> > > > > > > > > >>> we
> > > > > > > > > >>> > do
> > > > > > > > > >>> > > in
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > transactional
> > > > > > > > > >>> > > > > > > > >> > > > > > > state
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do this
> > for
> > > > > > > > > compatibility
> > > > > > > > > >>> --
> > > > > > > > > >>> > if
> > > > > > > > > >>> > > we
> > > > > > > > > >>> > > > > > > > >> downgrade,
> > > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > > >>> > > > > > > > >> > > > will
> > > > > > > > > >>> > > > > > > > >> > > > > > > only
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > have
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> non-tagged
> > > > > fields)
> > > > > > > It
> > > > > > > > > >>> will be
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > old
> > > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > > >>> > > > > > > > >> > ID
> > > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > max
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope
> this
> > > > helps.
> > > > > > Let
> > > > > > > > me
> > > > > > > > > >>> know
> > > > > > > > > >>> > if
> > > > > > > > > >>> > > > you
> > > > > > > > > >>> > > > > > have
> > > > > > > > > >>> > > > > > > > >> > further
> > > > > > > > > >>> > > > > > > > >> > > > > > > questions.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Justine
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On Wed,
> > Dec
> > > > 20,
> > > > > > 2023
> > > > > > > > at
> > > > > > > > > >>> > 3:33 PM
> > > > > > > > > >>> > > > Jun
> > > > > > > > > >>> > > > > > Rao
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > <jun@confluent.io.invalid
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi,
> > > Justine,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It
> seems
> > > > that
> > > > > > you
> > > > > > > > have
> > > > > > > > > >>> made
> > > > > > > > > >>> > > some
> > > > > > > > > >>> > > > > > > changes
> > > > > > > > > >>> > > > > > > > >> to
> > > > > > > > > >>> > > > > > > > >> > > > KIP-890
> > > > > > > > > >>> > > > > > > > >> > > > > > > since
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > particular,
> > > > we
> > > > > > are
> > > > > > > > > >>> changing
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > > > > format
> > > > > > > > > >>> > > > > > > > of
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionLogValue.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> comments
> > > > > related
> > > > > > > to
> > > > > > > > > >>> that.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100.
> > Just
> > > to
> > > > > be
> > > > > > > > clear.
> > > > > > > > > >>> The
> > > > > > > > > >>> > > > > overflow
> > > > > > > > > >>> > > > > > > case
> > > > > > > > > >>> > > > > > > > >> > (i.e.
> > > > > > > > > >>> > > > > > > > >> > > > > when a
> > > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > generated)
> > > > is
> > > > > > when
> > > > > > > > the
> > > > > > > > > >>> > current
> > > > > > > > > >>> > > > > epoch
> > > > > > > > > >>> > > > > > > > >> equals
> > > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > > >>> > > > > > > > >> > > > max
> > > > > > > > > >>> > > > > > > > >> > > > > -
> > > > > > > > > >>> > > > > > > > >> > > > > > 1
> > > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101.
> For
> > > the
> > > > > > "not
> > > > > > > > > epoch
> > > > > > > > > >>> > > > overflow"
> > > > > > > > > >>> > > > > > > case,
> > > > > > > > > >>> > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > write
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > tagged
> > > field
> > > > > in
> > > > > > > the
> > > > > > > > > >>> complete
> > > > > > > > > >>> > > > > phase.
> > > > > > > > > >>> > > > > > Do
> > > > > > > > > >>> > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > need
> > > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > > >>> > > > > > > > >> > > > > do
> > > > > > > > > >>> > > > > > > > >> > > > > > > that
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> doesn't
> > > > change
> > > > > > in
> > > > > > > > this
> > > > > > > > > >>> case?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102.
> It
> > > > seems
> > > > > > that
> > > > > > > > the
> > > > > > > > > >>> > meaning
> > > > > > > > > >>> > > > for
> > > > > > > > > >>> > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > ProducerId/ProducerEpoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > TransactionLogValue
> > > > > > > > > >>> changes
> > > > > > > > > >>> > > > > > depending
> > > > > > > > > >>> > > > > > > on
> > > > > > > > > >>> > > > > > > > >> the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > > TransactionStatus
> > > > > > > > > is
> > > > > > > > > >>> > > > ongoing,
> > > > > > > > > >>> > > > > > they
> > > > > > > > > >>> > > > > > > > >> > > represent
> > > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > current
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > current
> > > > > > > > > >>> ProducerEpoch.
> > > > > > > > > >>> > > When
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > > > > > > >>> > > > > > > > >> > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> PrepareCommit/PrepareAbort,
> > > > > > > > > >>> > > they
> > > > > > > > > >>> > > > > > > > represent
> > > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > > >>> > > > > > > > >> > > > > > current
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > > > > > > ProducerEpoch.
> > > > > > > > > >>> When the
> > > > > > > > > >>> > > > > > > > >> > TransactionStatus
> > > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> further
> > > > depend
> > > > > > on
> > > > > > > > > >>> whether
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > epoch
> > > > > > > > > >>> > > > > > > > >> overflows
> > > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > > >>> > > > > > > > >> > > > > not.
> > > > > > > > > >>> > > > > > > > >> > > > > > > If
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > overflow,
> > > > they
> > > > > > > > > represent
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > > current
> > > > > > > > > >>> > > > > > > > >> > > ProducerId
> > > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > ProducerEpoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > (max).
> > > > > > Otherwise,
> > > > > > > > they
> > > > > > > > > >>> > > represent
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > > > > newly
> > > > > > > > > >>> > > > > > > > >> > > > > generated
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > ProducerEpoch
> > > > > of
> > > > > > > 0.
> > > > > > > > Is
> > > > > > > > > >>> that
> > > > > > > > > >>> > > > right?
> > > > > > > > > >>> > > > > > > This
> > > > > > > > > >>> > > > > > > > >> seems
> > > > > > > > > >>> > > > > > > > >> > > not
> > > > > > > > > >>> > > > > > > > >> > > > > > easy
> > > > > > > > > >>> > > > > > > > >> > > > > > > to
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we
> > provide
> > > > > some
> > > > > > > > > examples
> > > > > > > > > >>> > like
> > > > > > > > > >>> > > > what
> > > > > > > > > >>> > > > > > > Artem
> > > > > > > > > >>> > > > > > > > >> has
> > > > > > > > > >>> > > > > > > > >> > > done
> > > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > considered a
> > > > > > > simpler
> > > > > > > > > >>> design
> > > > > > > > > >>> > > > where
> > > > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId/ProducerEpoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> represent
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> same
> > > > value
> > > > > > > (e.g.
> > > > > > > > > >>> for the
> > > > > > > > > >>> > > > > current
> > > > > > > > > >>> > > > > > > > >> > > transaction)
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > TransactionStatus
> > > > > > > > and
> > > > > > > > > >>> epoch
> > > > > > > > > >>> > > > > > overflow?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103.
> > It's
> > > > not
> > > > > > > clear
> > > > > > > > to
> > > > > > > > > >>> me
> > > > > > > > > >>> > why
> > > > > > > > > >>> > > we
> > > > > > > > > >>> > > > > > need
> > > > > > > > > >>> > > > > > > 3
> > > > > > > > > >>> > > > > > > > >> > fields:
> > > > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > PrevProducerId,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > NextProducerId.
> > > > > > > > Could
> > > > > > > > > we
> > > > > > > > > >>> > just
> > > > > > > > > >>> > > > have
> > > > > > > > > >>> > > > > > > > >> ProducerId
> > > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104.
> For
> > > > > > > > > >>> > > WriteTxnMarkerRequests,
> > > > > > > > > >>> > > > > if
> > > > > > > > > >>> > > > > > > the
> > > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we set
> > the
> > > > > > > > producerId
> > > > > > > > > >>> and
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > > > > > > >> producerEpoch?
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> Thanks,
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > > >>> > > > > > > > >> >
> > > > > > > > > >>> > > > > > > > >>
> > > > > > > > > >>> > > > > > > > >
> > > > > > > > > >>> > > > > > > >
> > > > > > > > > >>> > > > > > >
> > > > > > > > > >>> > > > > >
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > >
> > > > > > > > > >>> >
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Calvin,

Thanks for the update. The change looks good to me. Some minor comments.

For retry timeout, our convention is to use retry.backoff.max.ms and
retry.backoff.ms. So, to be consistent, the new configurations should be
add.partitions.to.txn.retry.backoff.max.ms and
add.partitions.to.txn.retry.backoff.ms.

Also, could you specify the defaults for these two values?

Thanks,

Jun




On Mon, Feb 10, 2025 at 2:45 PM Calvin Liu <ca...@confluent.io.invalid>
wrote:

> Hey team,
> In order to address the performance issue in the
> https://issues.apache.org/jira/browse/KAFKA-18654. We are making a change
> to retry the AddPartitionToTxn requests on the server side when hitting the
> CONCURRENT_TRANSACTION error. This error is common after the transaction is
> committed but before the markers are fully propagated to all the involved
> partitions. With the server side retry, the client can avoid the frequent
> and expensive retry for the produce request. In order to better control the
> retry behavior, we proposed to add 2 following configs.
>
>    - *add.partitions.to.txn.max.timeout.ms
>    <http://add.partitions.to.txn.max.timeout.ms>* defines the maximum
> retry
>    timeout when the server attempts to add the partition to the
> transaction.
>    - *add.partitions.to.txn.retry.backoff.ms
>    <http://add.partitions.to.txn.retry.backoff.ms>* defines how frequently
>    the server will retry the AddPartitionToTxn.
>
> Thanks!
>
> On Thu, Nov 14, 2024 at 1:52 PM Justine Olshan
> <jo...@confluent.io.invalid>
> wrote:
>
> > Hey folks, just wanted to share another quick update with KIP-890.
> >
> > Making the change to bump epoch after every transaction means that we no
> > longer need to call InitProducerId during the life of the producer to
> fence
> > requests from previous transactions. With KIP-890 part 2, the new client
> > will no longer call InitProducerId except on startup to fence a previous
> > instance.
> >
> > With this change, some other calls to InitProducerId were inspected
> > including the call after receiving an InvalidPidMappingException. This
> > exception was changed to abortable as part of KIP-360: Improve
> reliability
> > of idempotent/transactional producer
> > <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820
> > >.
> > However, this change means that we can violate EOS guarantees. As an
> > example:
> >
> > Consider an application that is copying data from one partition to
> another
> >
> >    - Application instance A processes to offset 4
> >    - Application instance B comes up and fences application instance A
> >    - Application instance B processes to offset 5
> >    - Application instances A and B are idle for
> > transaction.id.expiration.ms,
> >    transaction id expires on server
> >    - Application instance A attempts to process offset 5 (since in its
> >    view, that is next) -- if we recover from invalid pid mapping, we can
> >    duplicate this processing
> >
> > Thus, INVALID_PID_MAPPING should be fatal to the producer and as part of
> > KIP-890 the new client in 4.0 will treat the exception as fatal.
> >
> > This is consistent with KIP-1050: Consistent error handling for
> > Transactions
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions
> > >
> > where
> > errors that are fatal to the producer are in the "application
> recoverable"
> > category. This is a grouping that indicates to the client that the
> producer
> > needs to restart and recovery on the application side is necessary.
> > KIP-1050 is approved so we are consistent with that decision.
> >
> > Let me know if there are any questions. I've updated the KIP to reflect
> > this change.
> >
> > Justine
> >
> > On Wed, Jul 10, 2024 at 3:34 PM Jun Rao <ju...@confluent.io.invalid>
> wrote:
> >
> > > Sounds good, Justine. It would be useful to document that in the KIP.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Jul 10, 2024 at 2:59 PM Justine Olshan
> > > <jo...@confluent.io.invalid>
> > > wrote:
> > >
> > > > The client will send the newest EndTxn request version if and only if
> > > both
> > > > the client and the server support kip-890 part 2.
> > > > We set the value in the record based on the EndTxn version.
> > > >
> > > > Justine
> > > >
> > > > On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <ju...@confluent.io.invalid>
> > > wrote:
> > > >
> > > > > Hi, Justine,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > 120. If the broker sends TV Y for the finalized version in
> > > > > ApiVersionResponse, but the client doesn't support Y, how does the
> > > broker
> > > > > know the TV that the client supports?
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> > > > > <jo...@confluent.io.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hey Jun,
> > > > > >
> > > > > > No worries. Work on this KIP has been blocked for a bit anyways
> --
> > > > > catching
> > > > > > up and rereading what I wrote :)
> > > > > >
> > > > > > 120. ClientTransactionProtocolVersion is the transaction version
> as
> > > > > defined
> > > > > > by the highest transaction version (feature version value)
> > supported
> > > by
> > > > > the
> > > > > > client and the server. This works by the broker sending an
> > > > > > ApiVersionsRequest to the client with the finalized version.
> > Assuming
> > > > > > kip-890 part 2 is enabled by transaction version Y, if this
> request
> > > > > > contains finalized version Y and the client has the logic to set
> > this
> > > > > > field, it will set Y. If the server has Y - 1 (kip 890 part 2 not
> > > > enable)
> > > > > > the client will send Y - 1, even though the client has the
> ability
> > to
> > > > > > support kip-890 part 2.
> > > > > >
> > > > > > 121. You are correct that this is not needed. However, currently
> > that
> > > > > field
> > > > > > is already being set in memory -- just not written to disk. I
> think
> > > it
> > > > is
> > > > > > ok to write it to disk though. Let me know if you think
> otherwise.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao <jun@confluent.io.invalid
> >
> > > > > wrote:
> > > > > >
> > > > > > > Hi, Justine,
> > > > > > >
> > > > > > > Thanks for the update and sorry for the late reply.
> > > > > > >
> > > > > > > 120. I am wondering what value is used for
> > > > > > > ClientTransactionProtocolVersion. Is it the version of the
> > > > > EndTxnRequest?
> > > > > > >
> > > > > > > 121. Earlier, you made the change to set lastProducerId in
> > PREPARE
> > > to
> > > > > > > indicate that the marker is written for the new client. With
> the
> > > new
> > > > > > > ClientTransactionProtocolVersion field, it seems this is no
> > longer
> > > > > > > necessary.
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi there -- another update!
> > > > > > > >
> > > > > > > > When looking into the implementation for the safe epoch
> bumps I
> > > > > > realized
> > > > > > > > that we are already populating previousProducerID in memory
> as
> > > part
> > > > > of
> > > > > > > > KIP-360.
> > > > > > > > If we are to start using flexible fields, it is better to
> > always
> > > > use
> > > > > > this
> > > > > > > > information and have an explicit (tagged) field to indicate
> > > whether
> > > > > the
> > > > > > > > client supports KIP-890 part 2.
> > > > > > > >
> > > > > > > > I've included the extra field and how it is set in the KIP.
> > I've
> > > > also
> > > > > > > > updated the KIP to explain that we will be setting the tagged
> > > > fields
> > > > > > when
> > > > > > > > they are available for all transitions.
> > > > > > > >
> > > > > > > > Finally, I added clearer text about the transaction protocol
> > > > versions
> > > > > > > > included with this KIP. 1 for flexible transaction state
> > records
> > > > and
> > > > > 2
> > > > > > > for
> > > > > > > > KIP-890 part 2 enablement.
> > > > > > > >
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <
> > > > jolshan@confluent.io
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey there -- small update to the KIP,
> > > > > > > > >
> > > > > > > > > The KIP mentioned introducing ABORTABLE_ERROR and bumping
> > > > > > > TxnOffsetCommit
> > > > > > > > > and Produce requests. I've changed the name in the KIP to
> > > > > > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > > > > > AbortableTransactionException to match the pattern we had
> for
> > > > other
> > > > > > > > errors.
> > > > > > > > > I also mentioned bumping all 6 transactional APIs so we can
> > > > future
> > > > > > > > > proof/support the error on the client going forward. If a
> > > future
> > > > > > change
> > > > > > > > > wants to have an error scenario that requires us to abort
> the
> > > > > > > > transaction,
> > > > > > > > > we can rely on the 3.8+ clients to support it. We ran into
> > > issues
> > > > > > > finding
> > > > > > > > > good/generic error codes that older clients could support
> > while
> > > > > > working
> > > > > > > > on
> > > > > > > > > this KIP, so this should help in the future.
> > > > > > > > >
> > > > > > > > > The features discussion is still ongoing in KIP-1022. Will
> > > update
> > > > > > again
> > > > > > > > > here when that concludes.
> > > > > > > > >
> > > > > > > > > Justine
> > > > > > > > >
> > > > > > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> > > > > jolshan@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> I don't think AddPartitions is a good example since we
> > > currenly
> > > > > > don't
> > > > > > > > >> gate the version on TV or MV. (We only set a different
> flag
> > > > > > depending
> > > > > > > on
> > > > > > > > >> the TV)
> > > > > > > > >>
> > > > > > > > >> Even if we did want to gate it on TV, I think the idea is
> to
> > > > move
> > > > > > away
> > > > > > > > >> from MV gating inter broker protocols. Ideally we can get
> > to a
> > > > > state
> > > > > > > > where
> > > > > > > > >> MV is just used for metadata changes.
> > > > > > > > >>
> > > > > > > > >> I think some of this discussion might fit more with the
> > > feature
> > > > > > > version
> > > > > > > > >> KIP, so I can try to open that up soon. Until we settle
> > that,
> > > > some
> > > > > > of
> > > > > > > > the
> > > > > > > > >> work in KIP-890 is blocked.
> > > > > > > > >>
> > > > > > > > >> Justine
> > > > > > > > >>
> > > > > > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao
> > > <jun@confluent.io.invalid
> > > > >
> > > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Hi, Justine,
> > > > > > > > >>>
> > > > > > > > >>> Thanks for the reply.
> > > > > > > > >>>
> > > > > > > > >>> Since AddPartitions is an inter broker request, will its
> > > > version
> > > > > be
> > > > > > > > gated
> > > > > > > > >>> only by TV or other features like MV too? For example, if
> > we
> > > > need
> > > > > > to
> > > > > > > > >>> change
> > > > > > > > >>> the protocol for AddPartitions for reasons other than txn
> > > > > > > verification
> > > > > > > > in
> > > > > > > > >>> the future, will the new version be gated by a new MV? If
> > so,
> > > > > does
> > > > > > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > > > > > >>>
> > > > > > > > >>> Jun
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > >>> wrote:
> > > > > > > > >>>
> > > > > > > > >>> > One TV gates the flexible feature version (no rpcs
> > > involved,
> > > > > only
> > > > > > > the
> > > > > > > > >>> > transactional records that should only be gated by TV)
> > > > > > > > >>> > Another TV gates the ability to turn on kip-890 part 2.
> > > This
> > > > > > would
> > > > > > > > >>> gate the
> > > > > > > > >>> > version of Produce and EndTxn (likely only used by
> > > > > transactions),
> > > > > > > and
> > > > > > > > >>> > specifies a flag in AddPartitionsToTxn though the
> version
> > > is
> > > > > > > already
> > > > > > > > >>> used
> > > > > > > > >>> > without TV.
> > > > > > > > >>> >
> > > > > > > > >>> > I think the only concern is the Produce request and we
> > > could
> > > > > > > consider
> > > > > > > > >>> work
> > > > > > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > > > > > >>> >
> > > > > > > > >>> > Justine
> > > > > > > > >>> >
> > > > > > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> > > > > <jun@confluent.io.invalid
> > > > > > >
> > > > > > > > >>> wrote:
> > > > > > > > >>> >
> > > > > > > > >>> > > Hi, Justine,
> > > > > > > > >>> > >
> > > > > > > > >>> > > Which PRC/record protocols will TV guard? Going
> > forward,
> > > > will
> > > > > > > those
> > > > > > > > >>> > > PRC/record protocols only be guarded by TV and not by
> > > other
> > > > > > > > features
> > > > > > > > >>> like
> > > > > > > > >>> > > MV?
> > > > > > > > >>> > >
> > > > > > > > >>> > > Thanks,
> > > > > > > > >>> > >
> > > > > > > > >>> > > Jun
> > > > > > > > >>> > >
> > > > > > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > > > > > >>> > <jolshan@confluent.io.invalid
> > > > > > > > >>> > > >
> > > > > > > > >>> > > wrote:
> > > > > > > > >>> > >
> > > > > > > > >>> > > > Hi Jun,
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > Sorry I think I misunderstood your question or
> > answered
> > > > > > > > >>> incorrectly.
> > > > > > > > >>> > The
> > > > > > > > >>> > > TV
> > > > > > > > >>> > > > version should ideally be fully independent from
> MV.
> > > > > > > > >>> > > > At least for the changes I proposed, TV should not
> > > affect
> > > > > MV
> > > > > > > and
> > > > > > > > MV
> > > > > > > > >>> > > should
> > > > > > > > >>> > > > not affect TV/
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > I think if we downgrade TV, only that feature
> should
> > > > > > downgrade.
> > > > > > > > >>> > Likewise
> > > > > > > > >>> > > > the same with MV. The finalizedFeatures should just
> > > > reflect
> > > > > > the
> > > > > > > > >>> feature
> > > > > > > > >>> > > > downgrade we made.
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > I also plan to write a new KIP for managing the
> disk
> > > > format
> > > > > > and
> > > > > > > > >>> upgrade
> > > > > > > > >>> > > > tool as we will need new flags to support these
> > > features.
> > > > > > That
> > > > > > > > >>> should
> > > > > > > > >>> > > help
> > > > > > > > >>> > > > clarify some things.
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > Justine
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > > > > > <jun@confluent.io.invalid
> > > > > > > > >
> > > > > > > > >>> > > wrote:
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > > Hi, Justine,
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > Thanks for the reply.
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > So, if we downgrade TV, we could implicitly
> > downgrade
> > > > > > another
> > > > > > > > >>> feature
> > > > > > > > >>> > > > (say
> > > > > > > > >>> > > > > MV) that has dependency (e.g. RPC). What would we
> > > > return
> > > > > > for
> > > > > > > > >>> > > > > FinalizedFeatures for MV in ApiVersionsResponse
> in
> > > that
> > > > > > case?
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > Thanks,
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > Jun
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > > > > > >>> > > > <jolshan@confluent.io.invalid
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > > wrote:
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > > Hey Jun,
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > > > Yes, the idea is that if we downgrade TV
> > > (transaction
> > > > > > > > version)
> > > > > > > > >>> we
> > > > > > > > >>> > > will
> > > > > > > > >>> > > > > stop
> > > > > > > > >>> > > > > > using the add partitions to txn optimization
> and
> > > stop
> > > > > > > writing
> > > > > > > > >>> the
> > > > > > > > >>> > > > > flexible
> > > > > > > > >>> > > > > > feature version of the log.
> > > > > > > > >>> > > > > > In the compatibility section I included some
> > > > > explanations
> > > > > > > on
> > > > > > > > >>> how
> > > > > > > > >>> > this
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > done.
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > > > Thanks,
> > > > > > > > >>> > > > > > Justine
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > > > > > >>> <ju...@confluent.io.invalid>
> > > > > > > > >>> > > > > wrote:
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > > > > Hi, Justine,
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > > > Thanks for the update.
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > > > If we ever downgrade the transaction feature,
> > any
> > > > > > feature
> > > > > > > > >>> > depending
> > > > > > > > >>> > > > on
> > > > > > > > >>> > > > > > > changes on top of those RPC/record
> > > > > > > > >>> > > > > > >
> (AddPartitionsToTxnRequest/TransactionLogValue)
> > > > > changes
> > > > > > > > made
> > > > > > > > >>> in
> > > > > > > > >>> > > > KIP-890
> > > > > > > > >>> > > > > > > will be automatically downgraded too?
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > > > Jun
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine
> Olshan
> > > > > > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > > >>> > > > > > > wrote:
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > > > > Hey Jun,
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > > > I wanted to get back to you about your
> > > questions
> > > > > > about
> > > > > > > > >>> MV/IBP.
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > > > Looking at the options, I think it makes
> the
> > > most
> > > > > > sense
> > > > > > > > to
> > > > > > > > >>> > > create a
> > > > > > > > >>> > > > > > > > separate feature for transactions and use
> > that
> > > to
> > > > > > > version
> > > > > > > > >>> gate
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > features
> > > > > > > > >>> > > > > > > > we need to version gate (flexible
> > transactional
> > > > > state
> > > > > > > > >>> records
> > > > > > > > >>> > and
> > > > > > > > >>> > > > > using
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > new protocol)
> > > > > > > > >>> > > > > > > > I've updated the KIP to include this
> change.
> > > > > > Hopefully
> > > > > > > > >>> that's
> > > > > > > > >>> > > > > > everything
> > > > > > > > >>> > > > > > > we
> > > > > > > > >>> > > > > > > > need for this KIP :)
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > > > Justine
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine
> > Olshan
> > > <
> > > > > > > > >>> > > > jolshan@confluent.io
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > > > > Thanks Jun,
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > > I will update the KIP with the prev field
> > for
> > > > > > prepare
> > > > > > > > as
> > > > > > > > >>> > well.
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > > PREPARE
> > > > > > > > >>> > > > > > > > > producerId: x
> > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty or z
> > if
> > > y
> > > > > will
> > > > > > > > >>> overflow
> > > > > > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > > COMPLETE
> > > > > > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > > > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we
> overflowed
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > > Thanks again,
> > > > > > > > >>> > > > > > > > > Justine
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun Rao
> > > > > > > > >>> > > <jun@confluent.io.invalid
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> Hi, Justine,
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > > > > > >>> > > > > > > > >> (1) My point was that the coordinator
> > could
> > > > fail
> > > > > > > right
> > > > > > > > >>> after
> > > > > > > > >>> > > > > writing
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> prepare marker. When the new txn
> > coordinator
> > > > > > > generates
> > > > > > > > >>> the
> > > > > > > > >>> > > > > complete
> > > > > > > > >>> > > > > > > > marker
> > > > > > > > >>> > > > > > > > >> after the failover, it needs some field
> > from
> > > > the
> > > > > > > > prepare
> > > > > > > > >>> > > marker
> > > > > > > > >>> > > > to
> > > > > > > > >>> > > > > > > > >> determine whether it's written by the
> new
> > > > > client.
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >> (2) The changing of the behavior sounds
> > good
> > > > to
> > > > > > me.
> > > > > > > We
> > > > > > > > >>> only
> > > > > > > > >>> > > want
> > > > > > > > >>> > > > > to
> > > > > > > > >>> > > > > > > > return
> > > > > > > > >>> > > > > > > > >> success if the prepare state is written
> by
> > > the
> > > > > new
> > > > > > > > >>> client.
> > > > > > > > >>> > So,
> > > > > > > > >>> > > > in
> > > > > > > > >>> > > > > > the
> > > > > > > > >>> > > > > > > > >> non-overflow case, it seems that we also
> > > need
> > > > > sth
> > > > > > in
> > > > > > > > the
> > > > > > > > >>> > > prepare
> > > > > > > > >>> > > > > > > marker
> > > > > > > > >>> > > > > > > > to
> > > > > > > > >>> > > > > > > > >> tell us whether it's written by the new
> > > > client.
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >> 112. Thanks for the explanation. That
> > sounds
> > > > > good
> > > > > > to
> > > > > > > > me.
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >> Jun
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM Justine
> > > > Olshan
> > > > > > > > >>> > > > > > > > >> <jo...@confluent.io.invalid> wrote:
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >> > 101.3 I realized that I actually have
> > two
> > > > > > > questions.
> > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> need
> > to
> > > > > write
> > > > > > > the
> > > > > > > > >>> > > previous
> > > > > > > > >>> > > > > > > produce
> > > > > > > > >>> > > > > > > > Id
> > > > > > > > >>> > > > > > > > >> > tagged field in the end maker so that
> we
> > > > know
> > > > > if
> > > > > > > the
> > > > > > > > >>> > marker
> > > > > > > > >>> > > is
> > > > > > > > >>> > > > > > from
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> new
> > > > > > > > >>> > > > > > > > >> > client. Since the end maker is derived
> > > from
> > > > > the
> > > > > > > > >>> prepare
> > > > > > > > >>> > > > marker,
> > > > > > > > >>> > > > > > > should
> > > > > > > > >>> > > > > > > > >> we
> > > > > > > > >>> > > > > > > > >> > write the previous produce Id in the
> > > prepare
> > > > > > > marker
> > > > > > > > >>> field
> > > > > > > > >>> > > too?
> > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > >>> > > > > > > > >> > we will lose this information when
> > > deriving
> > > > > the
> > > > > > > end
> > > > > > > > >>> > marker.
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > The "previous" producer ID is in the
> > > normal
> > > > > > > producer
> > > > > > > > >>> ID
> > > > > > > > >>> > > field.
> > > > > > > > >>> > > > > So
> > > > > > > > >>> > > > > > > yes,
> > > > > > > > >>> > > > > > > > >> we
> > > > > > > > >>> > > > > > > > >> > need it in prepare and that was always
> > the
> > > > > plan.
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I will
> > > > enumerate
> > > > > > the
> > > > > > > > >>> fields
> > > > > > > > >>> > and
> > > > > > > > >>> > > > add
> > > > > > > > >>> > > > > > > them
> > > > > > > > >>> > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > > > > > >>> > > > > > > > >> > Say we have producer ID x and epoch y.
> > > When
> > > > we
> > > > > > > > >>> overflow
> > > > > > > > >>> > > epoch
> > > > > > > > >>> > > > y
> > > > > > > > >>> > > > > we
> > > > > > > > >>> > > > > > > get
> > > > > > > > >>> > > > > > > > >> > producer ID Z.
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > PREPARE
> > > > > > > > >>> > > > > > > > >> > producerId: x
> > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> field):
> > > > empty
> > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> or
> > z
> > > > if y
> > > > > > > will
> > > > > > > > >>> > overflow
> > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > COMPLETE
> > > > > > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged
> field):
> > x
> > > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we
> > overflowed
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > (2) In the prepare phase, if we retry
> > and
> > > > see
> > > > > > > epoch
> > > > > > > > -
> > > > > > > > >>> 1 +
> > > > > > > > >>> > ID
> > > > > > > > >>> > > > in
> > > > > > > > >>> > > > > > last
> > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > >>> > > > > > > > >> > fields and are issuing the same
> command
> > > (ie
> > > > > > commit
> > > > > > > > not
> > > > > > > > >>> > > abort),
> > > > > > > > >>> > > > > we
> > > > > > > > >>> > > > > > > > return
> > > > > > > > >>> > > > > > > > >> > success. The logic before KIP-890
> seems
> > to
> > > > > > return
> > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > >>> > > > > > > > >> > in this case. Are we intentionally
> > making
> > > > this
> > > > > > > > change?
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > Hmm -- we would fence the producer if
> > the
> > > > > epoch
> > > > > > is
> > > > > > > > >>> bumped
> > > > > > > > >>> > > and
> > > > > > > > >>> > > > we
> > > > > > > > >>> > > > > > > get a
> > > > > > > > >>> > > > > > > > >> > lower epoch. Yes -- we are
> intentionally
> > > > > adding
> > > > > > > this
> > > > > > > > >>> to
> > > > > > > > >>> > > > prevent
> > > > > > > > >>> > > > > > > > fencing.
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > 112. We already merged the code that
> > adds
> > > > the
> > > > > > > > >>> VerifyOnly
> > > > > > > > >>> > > field
> > > > > > > > >>> > > > > in
> > > > > > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which is an
> > > inter
> > > > > > > broker
> > > > > > > > >>> > request.
> > > > > > > > >>> > > > It
> > > > > > > > >>> > > > > > > seems
> > > > > > > > >>> > > > > > > > >> that
> > > > > > > > >>> > > > > > > > >> > we didn't bump up the IBP for that. Do
> > you
> > > > > know
> > > > > > > why?
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > We no longer need IBP for all
> > interbroker
> > > > > > requests
> > > > > > > > as
> > > > > > > > >>> > > > > ApiVersions
> > > > > > > > >>> > > > > > > > should
> > > > > > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > > > > > >>> > > > > > > > >> > We also handle unsupported version
> > errors
> > > > > > > correctly
> > > > > > > > >>> if we
> > > > > > > > >>> > > > > receive
> > > > > > > > >>> > > > > > > them
> > > > > > > > >>> > > > > > > > >> in
> > > > > > > > >>> > > > > > > > >> > edge cases like upgrades/downgrades.
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > Justine
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM Jun
> Rao
> > > > > > > > >>> > > > > <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > > > > >> wrote:
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> > > 101.3 I realized that I actually
> have
> > > two
> > > > > > > > questions.
> > > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we
> need
> > to
> > > > > write
> > > > > > > the
> > > > > > > > >>> > > previous
> > > > > > > > >>> > > > > > > produce
> > > > > > > > >>> > > > > > > > Id
> > > > > > > > >>> > > > > > > > >> > > tagged field in the end maker so
> that
> > we
> > > > > know
> > > > > > if
> > > > > > > > the
> > > > > > > > >>> > > marker
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > > from
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > new
> > > > > > > > >>> > > > > > > > >> > > client. Since the end maker is
> derived
> > > > from
> > > > > > the
> > > > > > > > >>> prepare
> > > > > > > > >>> > > > > marker,
> > > > > > > > >>> > > > > > > > >> should we
> > > > > > > > >>> > > > > > > > >> > > write the previous produce Id in the
> > > > prepare
> > > > > > > > marker
> > > > > > > > >>> > field
> > > > > > > > >>> > > > too?
> > > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > > >>> > > > > > > > >> > > we will lose this information when
> > > > deriving
> > > > > > the
> > > > > > > > end
> > > > > > > > >>> > > marker.
> > > > > > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we
> retry
> > > and
> > > > > see
> > > > > > > > epoch
> > > > > > > > >>> - 1
> > > > > > > > >>> > +
> > > > > > > > >>> > > ID
> > > > > > > > >>> > > > > in
> > > > > > > > >>> > > > > > > last
> > > > > > > > >>> > > > > > > > >> seen
> > > > > > > > >>> > > > > > > > >> > > fields and are issuing the same
> > command
> > > > (ie
> > > > > > > commit
> > > > > > > > >>> not
> > > > > > > > >>> > > > abort),
> > > > > > > > >>> > > > > > we
> > > > > > > > >>> > > > > > > > >> return
> > > > > > > > >>> > > > > > > > >> > > success. The logic before KIP-890
> > seems
> > > to
> > > > > > > return
> > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > >>> > > > > > > > >> > > in this case. Are we intentionally
> > > making
> > > > > this
> > > > > > > > >>> change?
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> > > 112. We already merged the code that
> > > adds
> > > > > the
> > > > > > > > >>> VerifyOnly
> > > > > > > > >>> > > > field
> > > > > > > > >>> > > > > > in
> > > > > > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which is
> an
> > > > inter
> > > > > > > > broker
> > > > > > > > >>> > > request.
> > > > > > > > >>> > > > > It
> > > > > > > > >>> > > > > > > > seems
> > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > >>> > > > > > > > >> > > we didn't bump up the IBP for that.
> Do
> > > you
> > > > > > know
> > > > > > > > why?
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> > > Jun
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM
> > Justine
> > > > > Olshan
> > > > > > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen" to
> > > > "current
> > > > > > > > >>> producer id
> > > > > > > > >>> > > and
> > > > > > > > >>> > > > > > > epoch"
> > > > > > > > >>> > > > > > > > if
> > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > >>> > > > > > > > >> > > > was the part that was confusing
> > > > > > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > > > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > > > > > >>> > > > > > > > >> > > > 112 We still need it. But I am
> still
> > > > > > > finalizing
> > > > > > > > >>> the
> > > > > > > > >>> > > > design.
> > > > > > > > >>> > > > > I
> > > > > > > > >>> > > > > > > will
> > > > > > > > >>> > > > > > > > >> > update
> > > > > > > > >>> > > > > > > > >> > > > the KIP once I get the information
> > > > > > finalized.
> > > > > > > > >>> Sorry
> > > > > > > > >>> > for
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > > delays.
> > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > >>> > > > > > > > >> > > > Justine
> > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at 10:50 AM
> Jun
> > > Rao
> > > > > > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow case,
> > the
> > > > > > previous
> > > > > > > > ID
> > > > > > > > >>> is
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > same
> > > > > > > > >>> > > > > > as
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > ID for the complete marker too,
> > but
> > > we
> > > > > set
> > > > > > > the
> > > > > > > > >>> > > previous
> > > > > > > > >>> > > > ID
> > > > > > > > >>> > > > > > in
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > >>> > > > > > > > >> > > > > marker. Earlier you mentioned
> that
> > > > this
> > > > > is
> > > > > > > to
> > > > > > > > >>> know
> > > > > > > > >>> > > that
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > >>> > > > > > > > >> > > > > written by the new client so
> that
> > we
> > > > > could
> > > > > > > > >>> return
> > > > > > > > >>> > > > success
> > > > > > > > >>> > > > > on
> > > > > > > > >>> > > > > > > > >> retried
> > > > > > > > >>> > > > > > > > >> > > > > endMarker requests. I was trying
> > to
> > > > > > > understand
> > > > > > > > >>> why
> > > > > > > > >>> > > this
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > not
> > > > > > > > >>> > > > > > > > >> needed
> > > > > > > > >>> > > > > > > > >> > > for
> > > > > > > > >>> > > > > > > > >> > > > > the prepare marker since retry
> can
> > > > > happen
> > > > > > in
> > > > > > > > the
> > > > > > > > >>> > > prepare
> > > > > > > > >>> > > > > > state
> > > > > > > > >>> > > > > > > > >> too.
> > > > > > > > >>> > > > > > > > >> > Is
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > reason that in the prepare
> state,
> > we
> > > > > > return
> > > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > > >>> > > > > > > > >> > > > instead
> > > > > > > > >>> > > > > > > > >> > > > > of success on retried endMaker
> > > > requests?
> > > > > > If
> > > > > > > > so,
> > > > > > > > >>> > should
> > > > > > > > >>> > > > we
> > > > > > > > >>> > > > > > > change
> > > > > > > > >>> > > > > > > > >> "If
> > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID in
> > last
> > > > > seen
> > > > > > > > >>> fields and
> > > > > > > > >>> > > are
> > > > > > > > >>> > > > > > > issuing
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > same
> > > > > > > > >>> > > > > > > > >> > > > > command (ie commit not abort) we
> > can
> > > > > > return
> > > > > > > > >>> (with
> > > > > > > > >>> > the
> > > > > > > > >>> > > > new
> > > > > > > > >>> > > > > > > > epoch)"
> > > > > > > > >>> > > > > > > > >> > > > > accordingly?
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP, a
> > > delayed
> > > > > > > endMaker
> > > > > > > > >>> > request
> > > > > > > > >>> > > > > > carries
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > >>> > > > > > > > >> > > > > epoch and won't be fenced. This
> > can
> > > > > > > > >>> commit/abort a
> > > > > > > > >>> > > > future
> > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure if
> we
> > > have
> > > > > > seen
> > > > > > > > >>> this in
> > > > > > > > >>> > > > > practice
> > > > > > > > >>> > > > > > > > >> though.
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would be
> > useful
> > > > to
> > > > > > make
> > > > > > > > it
> > > > > > > > >>> > clear
> > > > > > > > >>> > > > that
> > > > > > > > >>> > > > > > we
> > > > > > > > >>> > > > > > > > can
> > > > > > > > >>> > > > > > > > >> now
> > > > > > > > >>> > > > > > > > >> > > > > populate the lastSeen field from
> > the
> > > > log
> > > > > > > > >>> reliably.
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > >>> > > > > since
> > > > > > > > >>> > > > > > > > it's
> > > > > > > > >>> > > > > > > > >> > > called
> > > > > > > > >>> > > > > > > > >> > > > > across brokers and we are
> changing
> > > its
> > > > > > > schema.
> > > > > > > > >>> Are
> > > > > > > > >>> > you
> > > > > > > > >>> > > > > > saying
> > > > > > > > >>> > > > > > > we
> > > > > > > > >>> > > > > > > > >> > don't
> > > > > > > > >>> > > > > > > > >> > > > need
> > > > > > > > >>> > > > > > > > >> > > > > it any more? I thought that we
> > > already
> > > > > > > > >>> implemented
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > > server
> > > > > > > > >>> > > > > > > > side
> > > > > > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > > >>> > > > > across
> > > > > > > > >>> > > > > > > > >> brokers.
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at 5:05 PM
> > > > Justine
> > > > > > > Olshan
> > > > > > > > >>> > > > > > > > >> > > > > <jo...@confluent.io.invalid>
> > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the
> previous
> > ID
> > > > in
> > > > > > the
> > > > > > > > >>> Prepare
> > > > > > > > >>> > > > field
> > > > > > > > >>> > > > > > > since
> > > > > > > > >>> > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > don't
> > > > > > > > >>> > > > > > > > >> > > > > need
> > > > > > > > >>> > > > > > > > >> > > > > > it. It is the same producer ID
> > as
> > > > the
> > > > > > main
> > > > > > > > >>> > producer
> > > > > > > > >>> > > ID
> > > > > > > > >>> > > > > > > field.
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to
> > reread
> > > > your
> > > > > > > > message
> > > > > > > > >>> > about
> > > > > > > > >>> > > > > > delayed
> > > > > > > > >>> > > > > > > > >> > markers.
> > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn
> marker
> > > > after
> > > > > > the
> > > > > > > > >>> > > transaction
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > > > already
> > > > > > > > >>> > > > > > > > >> > > > > complete?
> > > > > > > > >>> > > > > > > > >> > > > > > So we will commit the next
> > > > transaction
> > > > > > > early
> > > > > > > > >>> > without
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > fixes
> > > > > > > > >>> > > > > > > > >> in
> > > > > > > > >>> > > > > > > > >> > > part
> > > > > > > > >>> > > > > > > > >> > > > 2?
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > 111 Yes -- this terminology
> was
> > > used
> > > > > in
> > > > > > a
> > > > > > > > >>> previous
> > > > > > > > >>> > > KIP
> > > > > > > > >>> > > > > and
> > > > > > > > >>> > > > > > > > never
> > > > > > > > >>> > > > > > > > >> > > > > > implemented it in the log --
> > only
> > > in
> > > > > > > memory
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which interbroker
> > > > protocol
> > > > > > are
> > > > > > > > you
> > > > > > > > >>> > > > referring
> > > > > > > > >>> > > > > > to?
> > > > > > > > >>> > > > > > > I
> > > > > > > > >>> > > > > > > > am
> > > > > > > > >>> > > > > > > > >> > > > working
> > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > the design for the work to
> > remove
> > > > the
> > > > > > > extra
> > > > > > > > >>> add
> > > > > > > > >>> > > > > partitions
> > > > > > > > >>> > > > > > > > call
> > > > > > > > >>> > > > > > > > >> > and I
> > > > > > > > >>> > > > > > > > >> > > > > right
> > > > > > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I
> have
> > > yet
> > > > to
> > > > > > > > update
> > > > > > > > >>> that
> > > > > > > > >>> > > > > section
> > > > > > > > >>> > > > > > > as
> > > > > > > > >>> > > > > > > > I
> > > > > > > > >>> > > > > > > > >> > > > finalize
> > > > > > > > >>> > > > > > > > >> > > > > > the design so please stay
> tuned.
> > > Was
> > > > > > there
> > > > > > > > >>> > anything
> > > > > > > > >>> > > > else
> > > > > > > > >>> > > > > > you
> > > > > > > > >>> > > > > > > > >> > thought
> > > > > > > > >>> > > > > > > > >> > > > > needed
> > > > > > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > Justine
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at
> 3:07 PM
> > > Jun
> > > > > Rao
> > > > > > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > I don't see this create any
> > > issue.
> > > > > It
> > > > > > > just
> > > > > > > > >>> makes
> > > > > > > > >>> > > it
> > > > > > > > >>> > > > a
> > > > > > > > >>> > > > > > bit
> > > > > > > > >>> > > > > > > > >> hard to
> > > > > > > > >>> > > > > > > > >> > > > > explain
> > > > > > > > >>> > > > > > > > >> > > > > > > what this non-tagged produce
> > id
> > > > > field
> > > > > > > > >>> means. We
> > > > > > > > >>> > > are
> > > > > > > > >>> > > > > > > > >> essentially
> > > > > > > > >>> > > > > > > > >> > > > trying
> > > > > > > > >>> > > > > > > > >> > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > combine two actions
> > (completing
> > > a
> > > > > txn
> > > > > > > and
> > > > > > > > >>> init a
> > > > > > > > >>> > > new
> > > > > > > > >>> > > > > > > produce
> > > > > > > > >>> > > > > > > > >> Id)
> > > > > > > > >>> > > > > > > > >> > > in a
> > > > > > > > >>> > > > > > > > >> > > > > > > single record. But, this may
> > be
> > > > fine
> > > > > > > too.
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > A few other follow up
> > comments.
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason
> that
> > we
> > > > > only
> > > > > > > set
> > > > > > > > >>> the
> > > > > > > > >>> > > > previous
> > > > > > > > >>> > > > > > > > >> produce id
> > > > > > > > >>> > > > > > > > >> > > > > tagged
> > > > > > > > >>> > > > > > > > >> > > > > > > field in the complete
> marker,
> > > but
> > > > > not
> > > > > > in
> > > > > > > > the
> > > > > > > > >>> > > prepare
> > > > > > > > >>> > > > > > > marker,
> > > > > > > > >>> > > > > > > > >> is
> > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > prepare state, we always
> > return
> > > > > > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > > > > > >>> > > > > > on
> > > > > > > > >>> > > > > > > > >> > retried
> > > > > > > > >>> > > > > > > > >> > > > > > endMaker
> > > > > > > > >>> > > > > > > > >> > > > > > > requests?
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > 110. "I believe your second
> > > point
> > > > is
> > > > > > > > >>> mentioned
> > > > > > > > >>> > in
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > KIP. I
> > > > > > > > >>> > > > > > > > >> can
> > > > > > > > >>> > > > > > > > >> > > add
> > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > >>> > > > > > > > >> > > > > > > text on
> > > > > > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > > > > > >>> > > > > > > > >> > > > > > > > The delayed message case
> can
> > > > also
> > > > > > > > violate
> > > > > > > > >>> EOS
> > > > > > > > >>> > if
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > > delayed
> > > > > > > > >>> > > > > > > > >> > > > message
> > > > > > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > > > > > addPartitionsToTxn
> > > > > > > > >>> > request
> > > > > > > > >>> > > > > comes
> > > > > > > > >>> > > > > > > in.
> > > > > > > > >>> > > > > > > > >> > > > > Effectively
> > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > may see a message from a
> > > previous
> > > > > > > > (aborted)
> > > > > > > > >>> > > > > transaction
> > > > > > > > >>> > > > > > > > become
> > > > > > > > >>> > > > > > > > >> > part
> > > > > > > > >>> > > > > > > > >> > > > of
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > The above is the case when a
> > > > delayed
> > > > > > > > >>> message is
> > > > > > > > >>> > > > > appended
> > > > > > > > >>> > > > > > > to
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > data
> > > > > > > > >>> > > > > > > > >> > > > > > > partition. What I mentioned
> > is a
> > > > > > > slightly
> > > > > > > > >>> > > different
> > > > > > > > >>> > > > > case
> > > > > > > > >>> > > > > > > > when
> > > > > > > > >>> > > > > > > > >> a
> > > > > > > > >>> > > > > > > > >> > > > delayed
> > > > > > > > >>> > > > > > > > >> > > > > > > marker is appended to the
> > > > > transaction
> > > > > > > log
> > > > > > > > >>> > > partition.
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once we
> > move
> > > > past
> > > > > > the
> > > > > > > > >>> Prepare
> > > > > > > > >>> > > and
> > > > > > > > >>> > > > > > > > Complete
> > > > > > > > >>> > > > > > > > >> > > states,
> > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > don’t need to worry about
> > > lastSeen
> > > > > > > fields
> > > > > > > > >>> and
> > > > > > > > >>> > > clear
> > > > > > > > >>> > > > > > them,
> > > > > > > > >>> > > > > > > > just
> > > > > > > > >>> > > > > > > > >> > > handle
> > > > > > > > >>> > > > > > > > >> > > > > > state
> > > > > > > > >>> > > > > > > > >> > > > > > > transitions as normal.". Is
> > the
> > > > > > lastSeen
> > > > > > > > >>> field
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > same
> > > > > > > > >>> > > > > > as
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field in
> > > > > > > > >>> TransactionLogValue?
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > 112. Since the kip changes
> the
> > > > > > > > inter-broker
> > > > > > > > >>> > > > protocol,
> > > > > > > > >>> > > > > > > should
> > > > > > > > >>> > > > > > > > >> we
> > > > > > > > >>> > > > > > > > >> > > bump
> > > > > > > > >>> > > > > > > > >> > > > up
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this
> > feature
> > > > only
> > > > > > for
> > > > > > > > the
> > > > > > > > >>> > KRaft
> > > > > > > > >>> > > > > mode?
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at
> > 11:13 AM
> > > > > > Justine
> > > > > > > > >>> Olshan
> > > > > > > > >>> > > > > > > > >> > > > > > >
> <jolshan@confluent.io.invalid
> > >
> > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > I'm glad we are getting to
> > > > > > convergence
> > > > > > > > on
> > > > > > > > >>> the
> > > > > > > > >>> > > > > design.
> > > > > > > > >>> > > > > > :)
> > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > While I understand it
> seems
> > a
> > > > > little
> > > > > > > > >>> "weird".
> > > > > > > > >>> > > I'm
> > > > > > > > >>> > > > > not
> > > > > > > > >>> > > > > > > sure
> > > > > > > > >>> > > > > > > > >> what
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > benefit
> > > > > > > > >>> > > > > > > > >> > > > > > > > of writing an extra record
> > to
> > > > the
> > > > > > log.
> > > > > > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool to
> > > > describe
> > > > > > > > >>> transactions
> > > > > > > > >>> > > > won't
> > > > > > > > >>> > > > > > > work
> > > > > > > > >>> > > > > > > > >> (ie,
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > complete
> > > > > > > > >>> > > > > > > > >> > > > > > > > state is needed to
> calculate
> > > the
> > > > > > time
> > > > > > > > >>> since
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > > > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > completed?)
> > > > > > > > >>> > > > > > > > >> > > > > > > > If we have a reason like
> > this,
> > > > it
> > > > > is
> > > > > > > > >>> enough to
> > > > > > > > >>> > > > > > convince
> > > > > > > > >>> > > > > > > me
> > > > > > > > >>> > > > > > > > >> we
> > > > > > > > >>> > > > > > > > >> > > need
> > > > > > > > >>> > > > > > > > >> > > > > such
> > > > > > > > >>> > > > > > > > >> > > > > > > an
> > > > > > > > >>> > > > > > > > >> > > > > > > > extra record. It seems
> like
> > it
> > > > > would
> > > > > > > be
> > > > > > > > >>> > > replacing
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > record
> > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is this
> > > correct?
> > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024 at
> > > 5:14 PM
> > > > > Jun
> > > > > > > Rao
> > > > > > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks for the
> > explanation.
> > > I
> > > > > > > > >>> understand the
> > > > > > > > >>> > > > > > intention
> > > > > > > > >>> > > > > > > > >> now.
> > > > > > > > >>> > > > > > > > >> > In
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > > > > > >>> > > > > > > > >> > > > > > > > > case, we set the
> > non-tagged
> > > > > field
> > > > > > to
> > > > > > > > >>> the old
> > > > > > > > >>> > > pid
> > > > > > > > >>> > > > > > (and
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> max
> > > > > > > > >>> > > > > > > > >> > > > > epoch)
> > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > prepare marker so that
> we
> > > > could
> > > > > > > > >>> correctly
> > > > > > > > >>> > > write
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> marker to
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > data
> > > > > > > > >>> > > > > > > > >> > > > > > > > > partition if the broker
> > > > > > downgrades.
> > > > > > > > When
> > > > > > > > >>> > > writing
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> complete
> > > > > > > > >>> > > > > > > > >> > > > > marker,
> > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > know the marker has
> > already
> > > > been
> > > > > > > > >>> written to
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > data
> > > > > > > > >>> > > > > > > > >> > partition.
> > > > > > > > >>> > > > > > > > >> > > > We
> > > > > > > > >>> > > > > > > > >> > > > > > set
> > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to the
> > new
> > > > pid
> > > > > to
> > > > > > > > avoid
> > > > > > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > client if the broker
> > > > downgrades.
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > The above seems to work.
> > > It's
> > > > > > just a
> > > > > > > > bit
> > > > > > > > >>> > > > > > inconsistent
> > > > > > > > >>> > > > > > > > for
> > > > > > > > >>> > > > > > > > >> a
> > > > > > > > >>> > > > > > > > >> > > > prepare
> > > > > > > > >>> > > > > > > > >> > > > > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > > > and a complete marker to
> > use
> > > > > > > different
> > > > > > > > >>> pids
> > > > > > > > >>> > in
> > > > > > > > >>> > > > > this
> > > > > > > > >>> > > > > > > > >> special
> > > > > > > > >>> > > > > > > > >> > > case.
> > > > > > > > >>> > > > > > > > >> > > > > If
> > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrade with the
> > complete
> > > > > > marker,
> > > > > > > it
> > > > > > > > >>> seems
> > > > > > > > >>> > > > that
> > > > > > > > >>> > > > > we
> > > > > > > > >>> > > > > > > > will
> > > > > > > > >>> > > > > > > > >> > never
> > > > > > > > >>> > > > > > > > >> > > > be
> > > > > > > > >>> > > > > > > > >> > > > > > able
> > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > write the complete
> marker
> > > with
> > > > > the
> > > > > > > old
> > > > > > > > >>> pid.
> > > > > > > > >>> > > Not
> > > > > > > > >>> > > > > sure
> > > > > > > > >>> > > > > > > if
> > > > > > > > >>> > > > > > > > it
> > > > > > > > >>> > > > > > > > >> > > causes
> > > > > > > > >>> > > > > > > > >> > > > > any
> > > > > > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems a
> bit
> > > > weird.
> > > > > > > > >>> Instead of
> > > > > > > > >>> > > > > writing
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > complete
> > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > > > with the new pid, could
> we
> > > > write
> > > > > > two
> > > > > > > > >>> > records:
> > > > > > > > >>> > > a
> > > > > > > > >>> > > > > > > complete
> > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > >>> > > > > > > > >> > > > > with
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > old pid followed by a
> > > > > > > > >>> TransactionLogValue
> > > > > > > > >>> > with
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > new
> > > > > > > > >>> > > > > > > > pid
> > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > >>> > > > > > > > >> > > an
> > > > > > > > >>> > > > > > > > >> > > > > > empty
> > > > > > > > >>> > > > > > > > >> > > > > > > > > state? We could make the
> > two
> > > > > > records
> > > > > > > > in
> > > > > > > > >>> the
> > > > > > > > >>> > > same
> > > > > > > > >>> > > > > > batch
> > > > > > > > >>> > > > > > > > so
> > > > > > > > >>> > > > > > > > >> > that
> > > > > > > > >>> > > > > > > > >> > > > they
> > > > > > > > >>> > > > > > > > >> > > > > > > will
> > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > >>> > > > > > > > >> > > > > > > > > added to the log
> > atomically.
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024 at
> > > > 5:40 PM
> > > > > > > > Justine
> > > > > > > > >>> > Olshan
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > <jolshan@confluent.io.invalid
> > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare marker
> > is
> > > > > > written,
> > > > > > > > >>> but the
> > > > > > > > >>> > > > > endTxn
> > > > > > > > >>> > > > > > > > >> response
> > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > >>> > > > > > > > >> > > > not
> > > > > > > > >>> > > > > > > > >> > > > > > > > > received
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > by the client when the
> > > > server
> > > > > > > > >>> downgrades
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare
> marker
> > is
> > > > > > > written,
> > > > > > > > >>> the
> > > > > > > > >>> > > endTxn
> > > > > > > > >>> > > > > > > > response
> > > > > > > > >>> > > > > > > > >> is
> > > > > > > > >>> > > > > > > > >> > > > > received
> > > > > > > > >>> > > > > > > > >> > > > > > > by
> > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > client when the server
> > > > > > downgrades.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > I think I am still a
> > > little
> > > > > > > > confused.
> > > > > > > > >>> In
> > > > > > > > >>> > > both
> > > > > > > > >>> > > > of
> > > > > > > > >>> > > > > > > these
> > > > > > > > >>> > > > > > > > >> > cases,
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction log has
> the
> > > old
> > > > > > > producer
> > > > > > > > >>> ID.
> > > > > > > > >>> > We
> > > > > > > > >>> > > > > don't
> > > > > > > > >>> > > > > > > > write
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > in the prepare
> marker's
> > > non
> > > > > > tagged
> > > > > > > > >>> fields.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > If the server
> downgrades
> > > > now,
> > > > > it
> > > > > > > > would
> > > > > > > > >>> > read
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > > records
> > > > > > > > >>> > > > > > > > >> not
> > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields and the
> complete
> > > > marker
> > > > > > > will
> > > > > > > > >>> also
> > > > > > > > >>> > > have
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > old
> > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > >>> > > > > > > > >> > > > > ID.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > (If we had used the
> new
> > > > > producer
> > > > > > > ID,
> > > > > > > > >>> we
> > > > > > > > >>> > > would
> > > > > > > > >>> > > > > not
> > > > > > > > >>> > > > > > > have
> > > > > > > > >>> > > > > > > > >> > > > > > transactional
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > correctness since the
> > > > producer
> > > > > > id
> > > > > > > > >>> doesn't
> > > > > > > > >>> > > > match
> > > > > > > > >>> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > state would not be
> > correct
> > > > on
> > > > > > the
> > > > > > > > data
> > > > > > > > >>> > > > > partition.)
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > In the overflow case,
> > I'd
> > > > > expect
> > > > > > > the
> > > > > > > > >>> > > following
> > > > > > > > >>> > > > > to
> > > > > > > > >>> > > > > > > > >> happen on
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > client
> > > > > > > > >>> > > > > > > > >> > > > > > > > > side
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we retry
> > EndTxn
> > > > --
> > > > > it
> > > > > > > is
> > > > > > > > >>> the
> > > > > > > > >>> > same
> > > > > > > > >>> > > > > > > producer
> > > > > > > > >>> > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > >>> > > > > > > > >> > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > -
> > > > > > > > >>> > > > > > > > >> > > > > > > 1
> > > > > > > > >>> > > > > > > > >> > > > > > > > > this
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > would fence the
> producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't
> retry
> > > > > EndTxn
> > > > > > > and
> > > > > > > > >>> use
> > > > > > > > >>> > the
> > > > > > > > >>> > > > new
> > > > > > > > >>> > > > > > > > >> producer id
> > > > > > > > >>> > > > > > > > >> > > > which
> > > > > > > > >>> > > > > > > > >> > > > > > > would
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > > > > > InvalidPidMappingException
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have
> > special
> > > > > > handling
> > > > > > > > for
> > > > > > > > >>> > when
> > > > > > > > >>> > > a
> > > > > > > > >>> > > > > > server
> > > > > > > > >>> > > > > > > > >> > > > downgrades.
> > > > > > > > >>> > > > > > > > >> > > > > > When
> > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > reconnects we could
> get
> > an
> > > > API
> > > > > > > > version
> > > > > > > > >>> > > request
> > > > > > > > >>> > > > > > > showing
> > > > > > > > >>> > > > > > > > >> > > KIP-890
> > > > > > > > >>> > > > > > > > >> > > > > > part 2
> > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > not supported. In that
> > > case,
> > > > > we
> > > > > > > can
> > > > > > > > >>> call
> > > > > > > > >>> > > > > > > > initProducerId
> > > > > > > > >>> > > > > > > > >> to
> > > > > > > > >>> > > > > > > > >> > > > abort
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In the
> > > > overflow
> > > > > > > case,
> > > > > > > > >>> this
> > > > > > > > >>> > > > > correctly
> > > > > > > > >>> > > > > > > > gives
> > > > > > > > >>> > > > > > > > >> > us a
> > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > I guess the
> > corresponding
> > > > case
> > > > > > > would
> > > > > > > > >>> be
> > > > > > > > >>> > > where
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> *complete
> > > > > > > > >>> > > > > > > > >> > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > *is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > written but the endTxn
> > is
> > > > not
> > > > > > > > >>> received by
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > client
> > > > > > > > >>> > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > > server
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This would
> > > > result
> > > > > in
> > > > > > > the
> > > > > > > > >>> > > > transaction
> > > > > > > > >>> > > > > > > > >> > coordinator
> > > > > > > > >>> > > > > > > > >> > > > > having
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old
> one.
> > > If
> > > > > the
> > > > > > > > client
> > > > > > > > >>> > > > retries,
> > > > > > > > >>> > > > > it
> > > > > > > > >>> > > > > > > > will
> > > > > > > > >>> > > > > > > > >> > > receive
> > > > > > > > >>> > > > > > > > >> > > > > an
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > InvalidPidMappingException.
> > > > > The
> > > > > > > > >>> > > InitProducerId
> > > > > > > > >>> > > > > > > > scenario
> > > > > > > > >>> > > > > > > > >> > above
> > > > > > > > >>> > > > > > > > >> > > > > would
> > > > > > > > >>> > > > > > > > >> > > > > > > > help
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my
> > > > compatibility
> > > > > > > story
> > > > > > > > is
> > > > > > > > >>> > meant
> > > > > > > > >>> > > > to
> > > > > > > > >>> > > > > > > > support
> > > > > > > > >>> > > > > > > > >> > > > > downgrades
> > > > > > > > >>> > > > > > > > >> > > > > > > > server
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > side in keeping the
> > > > > > transactional
> > > > > > > > >>> > > correctness.
> > > > > > > > >>> > > > > > > Keeping
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > > client
> > > > > > > > >>> > > > > > > > >> > > > > > > from
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is not
> > the
> > > > > > > priority.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I can
> > > also
> > > > > add
> > > > > > > text
> > > > > > > > >>> in
> > > > > > > > >>> > the
> > > > > > > > >>> > > > KIP
> > > > > > > > >>> > > > > > > about
> > > > > > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > > > > > >>> > > > > > > > >> > > > > > > if
> > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > think that fixes some
> > edge
> > > > > > cases.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12, 2024
> at
> > > > > 4:10 PM
> > > > > > > Jun
> > > > > > > > >>> Rao
> > > > > > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the
> reply.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we
> don't
> > > need
> > > > > to
> > > > > > > > >>> optimize
> > > > > > > > >>> > for
> > > > > > > > >>> > > > > > fencing
> > > > > > > > >>> > > > > > > > >> during
> > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Regarding
> consistency,
> > > > there
> > > > > > are
> > > > > > > > two
> > > > > > > > >>> > > > possible
> > > > > > > > >>> > > > > > > cases:
> > > > > > > > >>> > > > > > > > >> (1)
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > prepare
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is written, but the
> > > endTxn
> > > > > > > > response
> > > > > > > > >>> is
> > > > > > > > >>> > not
> > > > > > > > >>> > > > > > > received
> > > > > > > > >>> > > > > > > > by
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > > client
> > > > > > > > >>> > > > > > > > >> > > > > > > > when
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > server downgrades;
> (2)
> > > > the
> > > > > > > > prepare
> > > > > > > > >>> > marker
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > > > written,
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > > endTxn
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is received by the
> > > client
> > > > > when
> > > > > > > the
> > > > > > > > >>> > server
> > > > > > > > >>> > > > > > > > downgrades.
> > > > > > > > >>> > > > > > > > >> In
> > > > > > > > >>> > > > > > > > >> > > (1),
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > client
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > will have the old
> > > produce
> > > > Id
> > > > > > and
> > > > > > > > in
> > > > > > > > >>> (2),
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > client
> > > > > > > > >>> > > > > > > > >> will
> > > > > > > > >>> > > > > > > > >> > > have
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If we
> > > > downgrade
> > > > > > > right
> > > > > > > > >>> after
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > prepare
> > > > > > > > >>> > > > > > > > >> > marker,
> > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > can't
> > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > consistent to both
> (1)
> > > and
> > > > > (2)
> > > > > > > > >>> since we
> > > > > > > > >>> > > can
> > > > > > > > >>> > > > > only
> > > > > > > > >>> > > > > > > put
> > > > > > > > >>> > > > > > > > >> one
> > > > > > > > >>> > > > > > > > >> > > > value
> > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > existing produce Id
> > > field.
> > > > > > It's
> > > > > > > > >>> also not
> > > > > > > > >>> > > > clear
> > > > > > > > >>> > > > > > > which
> > > > > > > > >>> > > > > > > > >> case
> > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > So we could probably
> > be
> > > > > > > consistent
> > > > > > > > >>> with
> > > > > > > > >>> > > > either
> > > > > > > > >>> > > > > > > case.
> > > > > > > > >>> > > > > > > > >> By
> > > > > > > > >>> > > > > > > > >> > > > putting
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in the
> > > prepare
> > > > > > > marker,
> > > > > > > > >>> we
> > > > > > > > >>> > are
> > > > > > > > >>> > > > > > > consistent
> > > > > > > > >>> > > > > > > > >> with
> > > > > > > > >>> > > > > > > > >> > > > case
> > > > > > > > >>> > > > > > > > >> > > > > > (2)
> > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > it
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > also has the slight
> > > > benefit
> > > > > > that
> > > > > > > > the
> > > > > > > > >>> > > produce
> > > > > > > > >>> > > > > > field
> > > > > > > > >>> > > > > > > > in
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > > > prepare
> > > > > > > > >>> > > > > > > > >> > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > complete marker are
> > > > > consistent
> > > > > > > in
> > > > > > > > >>> the
> > > > > > > > >>> > > > overflow
> > > > > > > > >>> > > > > > > case.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12, 2024
> > at
> > > > > > 3:11 PM
> > > > > > > > >>> Justine
> > > > > > > > >>> > > > Olshan
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > <jolshan@confluent.io.invalid
> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the case you
> > > > describe,
> > > > > we
> > > > > > > > would
> > > > > > > > >>> > need
> > > > > > > > >>> > > to
> > > > > > > > >>> > > > > > have
> > > > > > > > >>> > > > > > > a
> > > > > > > > >>> > > > > > > > >> > delayed
> > > > > > > > >>> > > > > > > > >> > > > > > > request,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > successful EndTxn,
> > > and a
> > > > > > > > >>> successful
> > > > > > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > >>> > > > > > > > >> > > > > then
> > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn
> > request
> > > > go
> > > > > > > > through
> > > > > > > > >>> for
> > > > > > > > >>> > a
> > > > > > > > >>> > > > > given
> > > > > > > > >>> > > > > > > > >> producer.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to
> figure
> > > out
> > > > > if
> > > > > > it
> > > > > > > > is
> > > > > > > > >>> > > possible
> > > > > > > > >>> > > > > for
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > client
> > > > > > > > >>> > > > > > > > >> > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > a previous request
> > is
> > > > > > delayed
> > > > > > > > >>> > somewhere.
> > > > > > > > >>> > > > But
> > > > > > > > >>> > > > > > > yes,
> > > > > > > > >>> > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > >>> > > > > > > > >> > > > > case
> > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > >>> > > > > > > > >> > > > > > > > > think
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > would fence the
> > > client.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the
> overflow
> > > > case.
> > > > > > In
> > > > > > > > the
> > > > > > > > >>> > > overflow
> > > > > > > > >>> > > > > > case,
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are
> different
> > on
> > > > the
> > > > > > > > marker
> > > > > > > > >>> and
> > > > > > > > >>> > on
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > transaction.
> > > > > > > > >>> > > > > > > > >> > > > > > So
> > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to use
> > the
> > > > max
> > > > > > > epoch
> > > > > > > > >>> but
> > > > > > > > >>> > the
> > > > > > > > >>> > > > new
> > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > >>> > > > > > > > >> > > > > > > > start
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and
> > epoch 0
> > > > in
> > > > > > the
> > > > > > > > >>> > > > transactional
> > > > > > > > >>> > > > > > > state.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the server
> > > downgrade
> > > > > > case,
> > > > > > > we
> > > > > > > > >>> want
> > > > > > > > >>> > to
> > > > > > > > >>> > > > see
> > > > > > > > >>> > > > > > the
> > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > >>> > > > > > > > >> > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > >>> > > > > > > > >> > > > > > > > that
> > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > what the client
> will
> > > > have.
> > > > > > If
> > > > > > > we
> > > > > > > > >>> > > complete
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> commit,
> > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > state is reloaded,
> > we
> > > > need
> > > > > > the
> > > > > > > > new
> > > > > > > > >>> > > > producer
> > > > > > > > >>> > > > > ID
> > > > > > > > >>> > > > > > > in
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > >>> > > > > > > > >> > > > > so
> > > > > > > > >>> > > > > > > > >> > > > > > > > there
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid
> producer
> > ID
> > > > > > > mapping.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > The server
> downgrade
> > > > cases
> > > > > > are
> > > > > > > > >>> > > considering
> > > > > > > > >>> > > > > > > > >> > transactional
> > > > > > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing
> from
> > > > > previous
> > > > > > > > >>> behavior
> > > > > > > > >>> > --
> > > > > > > > >>> > > > and
> > > > > > > > >>> > > > > > are
> > > > > > > > >>> > > > > > > > not
> > > > > > > > >>> > > > > > > > >> > > > > concerned
> > > > > > > > >>> > > > > > > > >> > > > > > > > about
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the
> > safety
> > > > from
> > > > > > > > fencing
> > > > > > > > >>> > > retries
> > > > > > > > >>> > > > > (as
> > > > > > > > >>> > > > > > > we
> > > > > > > > >>> > > > > > > > >> have
> > > > > > > > >>> > > > > > > > >> > > > > > downgraded
> > > > > > > > >>> > > > > > > > >> > > > > > > > so
> > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to
> > > support).
> > > > > > > Perhaps
> > > > > > > > >>> this
> > > > > > > > >>> > is
> > > > > > > > >>> > > a
> > > > > > > > >>> > > > > > trade
> > > > > > > > >>> > > > > > > > off,
> > > > > > > > >>> > > > > > > > >> > but
> > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > >>> > > > > > > > >> > > > > > think
> > > > > > > > >>> > > > > > > > >> > > > > > > it
> > > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client
> > > > downgrades,
> > > > > > it
> > > > > > > > will
> > > > > > > > >>> > have
> > > > > > > > >>> > > > > > > restarted
> > > > > > > > >>> > > > > > > > >> and
> > > > > > > > >>> > > > > > > > >> > it
> > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > >>> > > > > > > > >> > > > > ok
> > > > > > > > >>> > > > > > > > >> > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > have a new
> producer
> > ID
> > > > > too).
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan 12,
> 2024
> > > at
> > > > > > > 11:42 AM
> > > > > > > > >>> Jun
> > > > > > > > >>> > Rao
> > > > > > > > >>> > > > > > > > >> > > > > > > <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for the
> > > reply.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If the
> > marker
> > > > is
> > > > > > > > written
> > > > > > > > >>> by
> > > > > > > > >>> > the
> > > > > > > > >>> > > > new
> > > > > > > > >>> > > > > > > > >> client, we
> > > > > > > > >>> > > > > > > > >> > > can
> > > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the last email
> > > > guarantee
> > > > > > > that
> > > > > > > > >>> any
> > > > > > > > >>> > > EndTxn
> > > > > > > > >>> > > > > > > > requests
> > > > > > > > >>> > > > > > > > >> > with
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from the same
> > > producer
> > > > > and
> > > > > > > the
> > > > > > > > >>> same
> > > > > > > > >>> > > > > > > transaction.
> > > > > > > > >>> > > > > > > > >> Then
> > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > >>> > > > > > > > >> > > > > > don't
> > > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > return a fenced
> > > error
> > > > > but
> > > > > > > can
> > > > > > > > >>> handle
> > > > > > > > >>> > > > > > > gracefully
> > > > > > > > >>> > > > > > > > as
> > > > > > > > >>> > > > > > > > >> > > > > described
> > > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When a delayed
> > > EndTnx
> > > > > > > request
> > > > > > > > is
> > > > > > > > >>> > > > > processed,
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> txn
> > > > > > > > >>> > > > > > > > >> > > state
> > > > > > > > >>> > > > > > > > >> > > > > > could
> > > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next
> txn.
> > I
> > > > > guess
> > > > > > in
> > > > > > > > >>> this
> > > > > > > > >>> > case
> > > > > > > > >>> > > > we
> > > > > > > > >>> > > > > > > still
> > > > > > > > >>> > > > > > > > >> > return
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > fenced
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed
> > request?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry, my
> > > > question
> > > > > > was
> > > > > > > > >>> > > inaccurate.
> > > > > > > > >>> > > > > What
> > > > > > > > >>> > > > > > > you
> > > > > > > > >>> > > > > > > > >> > > > described
> > > > > > > > >>> > > > > > > > >> > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > "The downgrade
> > > > > > > compatibility I
> > > > > > > > >>> > mention
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > that
> > > > > > > > >>> > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > keep
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in the
> > > main
> > > > > > > > >>> (non-tagged)
> > > > > > > > >>> > > > fields
> > > > > > > > >>> > > > > as
> > > > > > > > >>> > > > > > > we
> > > > > > > > >>> > > > > > > > >> did
> > > > > > > > >>> > > > > > > > >> > > > before
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > code
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > server side." If
> > we
> > > > want
> > > > > > to
> > > > > > > do
> > > > > > > > >>> this,
> > > > > > > > >>> > > it
> > > > > > > > >>> > > > > > seems
> > > > > > > > >>> > > > > > > > >> that we
> > > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > > >>> > > > > > > > >> > > > > > > use
> > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > current produce
> Id
> > > and
> > > > > max
> > > > > > > > >>> epoch in
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > existing
> > > > > > > > >>> > > > > > > > >> > > > producerId
> > > > > > > > >>> > > > > > > > >> > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerEpoch
> > fields
> > > > for
> > > > > > > both
> > > > > > > > >>> the
> > > > > > > > >>> > > > prepare
> > > > > > > > >>> > > > > > and
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > The downgrade
> can
> > > > happen
> > > > > > > after
> > > > > > > > >>> the
> > > > > > > > >>> > > > > complete
> > > > > > > > >>> > > > > > > > >> marker is
> > > > > > > > >>> > > > > > > > >> > > > > > written.
> > > > > > > > >>> > > > > > > > >> > > > > > > > With
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > you described,
> the
> > > > > > > downgraded
> > > > > > > > >>> > > > coordinator
> > > > > > > > >>> > > > > > will
> > > > > > > > >>> > > > > > > > see
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of the
> old
> > > > one.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan 12,
> > 2024
> > > > at
> > > > > > > > 10:44 AM
> > > > > > > > >>> > > Justine
> > > > > > > > >>> > > > > > > Olshan
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > <jolshan@confluent.io.invalid
> > > > > > > > >
> > > > > > > > >>> > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can update
> the
> > > > > > > > description.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe your
> > > > second
> > > > > > > point
> > > > > > > > is
> > > > > > > > >>> > > > mentioned
> > > > > > > > >>> > > > > > in
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > KIP.
> > > > > > > > >>> > > > > > > > >> > > I
> > > > > > > > >>> > > > > > > > >> > > > > can
> > > > > > > > >>> > > > > > > > >> > > > > > > add
> > > > > > > > >>> > > > > > > > >> > > > > > > > > more
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it is
> > > > helpful.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The delayed
> > > > message
> > > > > > case
> > > > > > > > can
> > > > > > > > >>> > also
> > > > > > > > >>> > > > > > violate
> > > > > > > > >>> > > > > > > > EOS
> > > > > > > > >>> > > > > > > > >> if
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > delayed
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in after
> > the
> > > > > next
> > > > > > > > >>> > > > > addPartitionsToTxn
> > > > > > > > >>> > > > > > > > >> request
> > > > > > > > >>> > > > > > > > >> > > > comes
> > > > > > > > >>> > > > > > > > >> > > > > > in.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a
> > message
> > > > > from a
> > > > > > > > >>> previous
> > > > > > > > >>> > > > > > (aborted)
> > > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > > > become
> > > > > > > > >>> > > > > > > > >> > > > > > > > > part
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > next
> > transaction.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the marker
> is
> > > > > written
> > > > > > > by
> > > > > > > > >>> the
> > > > > > > > >>> > new
> > > > > > > > >>> > > > > > client,
> > > > > > > > >>> > > > > > > we
> > > > > > > > >>> > > > > > > > >> can
> > > > > > > > >>> > > > > > > > >> > > as I
> > > > > > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email
> > > guarantee
> > > > > > that
> > > > > > > > any
> > > > > > > > >>> > EndTxn
> > > > > > > > >>> > > > > > > requests
> > > > > > > > >>> > > > > > > > >> with
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > same
> > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same
> > producer
> > > > and
> > > > > > the
> > > > > > > > same
> > > > > > > > >>> > > > > > transaction.
> > > > > > > > >>> > > > > > > > >> Then we
> > > > > > > > >>> > > > > > > > >> > > > don't
> > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced error
> but
> > > can
> > > > > > > handle
> > > > > > > > >>> > > gracefully
> > > > > > > > >>> > > > > as
> > > > > > > > >>> > > > > > > > >> described
> > > > > > > > >>> > > > > > > > >> > > in
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't think
> a
> > > > > boolean
> > > > > > is
> > > > > > > > >>> useful
> > > > > > > > >>> > > > since
> > > > > > > > >>> > > > > it
> > > > > > > > >>> > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > directly
> > > > > > > > >>> > > > > > > > >> > > > > > > encoded
> > > > > > > > >>> > > > > > > > >> > > > > > > > by
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence or
> > lack
> > > of
> > > > > the
> > > > > > > > >>> tagged
> > > > > > > > >>> > > field
> > > > > > > > >>> > > > > > being
> > > > > > > > >>> > > > > > > > >> > written.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the prepare
> > > > marker
> > > > > we
> > > > > > > > will
> > > > > > > > >>> have
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > same
> > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In the
> > > > Complete
> > > > > > > state
> > > > > > > > >>> we
> > > > > > > > >>> > may
> > > > > > > > >>> > > > not.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure
> why
> > > the
> > > > > > > ongoing
> > > > > > > > >>> state
> > > > > > > > >>> > > > > matters
> > > > > > > > >>> > > > > > > for
> > > > > > > > >>> > > > > > > > >> this
> > > > > > > > >>> > > > > > > > >> > > > KIP.
> > > > > > > > >>> > > > > > > > >> > > > > It
> > > > > > > > >>> > > > > > > > >> > > > > > > > does
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure
> > what
> > > > you
> > > > > > are
> > > > > > > > >>> > referring
> > > > > > > > >>> > > to
> > > > > > > > >>> > > > > > about
> > > > > > > > >>> > > > > > > > >> > writing
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > previous
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the
> > prepare
> > > > > > marker.
> > > > > > > > >>> This is
> > > > > > > > >>> > > not
> > > > > > > > >>> > > > in
> > > > > > > > >>> > > > > > the
> > > > > > > > >>> > > > > > > > >> KIP.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the
> overflow
> > > > case,
> > > > > we
> > > > > > > > >>> write the
> > > > > > > > >>> > > > > > > > >> nextProducerId
> > > > > > > > >>> > > > > > > > >> > in
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so we
> > know
> > > > > what
> > > > > > we
> > > > > > > > >>> > assigned
> > > > > > > > >>> > > > when
> > > > > > > > >>> > > > > > we
> > > > > > > > >>> > > > > > > > >> reload
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we
> > complete,
> > > we
> > > > > > > > >>> transition
> > > > > > > > >>> > this
> > > > > > > > >>> > > > ID
> > > > > > > > >>> > > > > to
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> main
> > > > > > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the
> > previous
> > > > > > producer
> > > > > > > > ID
> > > > > > > > >>> > field
> > > > > > > > >>> > > > > filled
> > > > > > > > >>> > > > > > > in.
> > > > > > > > >>> > > > > > > > >> This
> > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > >>> > > > > > > > >> > > > so
> > > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > can
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry
> case
> > > the
> > > > > > > > operation
> > > > > > > > >>> > > > completed
> > > > > > > > >>> > > > > > > > >> > successfully
> > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > don't
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer. The
> > > > > downgrade
> > > > > > > > >>> > > compatibility
> > > > > > > > >>> > > > I
> > > > > > > > >>> > > > > > > > mention
> > > > > > > > >>> > > > > > > > >> is
> > > > > > > > >>> > > > > > > > >> > > that
> > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > keep
> > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer ID
> and
> > > > epoch
> > > > > in
> > > > > > > the
> > > > > > > > >>> main
> > > > > > > > >>> > > > > > > (non-tagged)
> > > > > > > > >>> > > > > > > > >> > fields
> > > > > > > > >>> > > > > > > > >> > > > as
> > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > did
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on the
> > server
> > > > > side.
> > > > > > > If
> > > > > > > > >>> the
> > > > > > > > >>> > > server
> > > > > > > > >>> > > > > > > > >> downgrades,
> > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > >>> > > > > > > > >> > > > are
> > > > > > > > >>> > > > > > > > >> > > > > > > still
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This addresses
> > > both
> > > > > the
> > > > > > > > >>> prepare
> > > > > > > > >>> > and
> > > > > > > > >>> > > > > > complete
> > > > > > > > >>> > > > > > > > >> state
> > > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri, Jan
> 12,
> > > 2024
> > > > > at
> > > > > > > > >>> 10:21 AM
> > > > > > > > >>> > Jun
> > > > > > > > >>> > > > Rao
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi, Justine,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks for
> the
> > > > > reply.
> > > > > > > > Sorry
> > > > > > > > >>> for
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > delay. I
> > > > > > > > >>> > > > > > > > >> > have a
> > > > > > > > >>> > > > > > > > >> > > > few
> > > > > > > > >>> > > > > > > > >> > > > > > > more
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I think
> > the
> > > > > > > > motivation
> > > > > > > > >>> > > section
> > > > > > > > >>> > > > > > could
> > > > > > > > >>> > > > > > > be
> > > > > > > > >>> > > > > > > > >> > > > improved.
> > > > > > > > >>> > > > > > > > >> > > > > > One
> > > > > > > > >>> > > > > > > > >> > > > > > > of
> > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > motivations
> > > listed
> > > > > by
> > > > > > > the
> > > > > > > > >>> KIP is
> > > > > > > > >>> > > > "This
> > > > > > > > >>> > > > > > can
> > > > > > > > >>> > > > > > > > >> happen
> > > > > > > > >>> > > > > > > > >> > > > when
> > > > > > > > >>> > > > > > > > >> > > > > a
> > > > > > > > >>> > > > > > > > >> > > > > > > > > message
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or delayed
> due
> > > to
> > > > > > > > networking
> > > > > > > > >>> > > issues
> > > > > > > > >>> > > > > or a
> > > > > > > > >>> > > > > > > > >> network
> > > > > > > > >>> > > > > > > > >> > > > > > partition,
> > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts, and
> > then
> > > > the
> > > > > > > > delayed
> > > > > > > > >>> > > message
> > > > > > > > >>> > > > > > > finally
> > > > > > > > >>> > > > > > > > >> > comes
> > > > > > > > >>> > > > > > > > >> > > > > in.".
> > > > > > > > >>> > > > > > > > >> > > > > > > This
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very
> accurate.
> > > > > Without
> > > > > > > > >>> KIP-890,
> > > > > > > > >>> > > > > > currently,
> > > > > > > > >>> > > > > > > > if
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts an
> > > ongoing
> > > > > > > > >>> transaction,
> > > > > > > > >>> > it
> > > > > > > > >>> > > > > > already
> > > > > > > > >>> > > > > > > > >> bumps
> > > > > > > > >>> > > > > > > > >> > up
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which
> prevents
> > > the
> > > > > > > delayed
> > > > > > > > >>> > produce
> > > > > > > > >>> > > > > > message
> > > > > > > > >>> > > > > > > > >> from
> > > > > > > > >>> > > > > > > > >> > > being
> > > > > > > > >>> > > > > > > > >> > > > > > added
> > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > partition.
> > What
> > > > can
> > > > > > > cause
> > > > > > > > a
> > > > > > > > >>> > > hanging
> > > > > > > > >>> > > > > > > > >> transaction
> > > > > > > > >>> > > > > > > > >> > is
> > > > > > > > >>> > > > > > > > >> > > > that
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completes
> > > (either
> > > > > > aborts
> > > > > > > > or
> > > > > > > > >>> > > > commits) a
> > > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > > >>> > > > > > > > >> > > > > before
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > successful
> ack
> > > on
> > > > > > > messages
> > > > > > > > >>> > > published
> > > > > > > > >>> > > > > in
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> same
> > > > > > > > >>> > > > > > > > >> > > txn.
> > > > > > > > >>> > > > > > > > >> > > > > In
> > > > > > > > >>> > > > > > > > >> > > > > > > this
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible for
> > the
> > > > > > delayed
> > > > > > > > >>> message
> > > > > > > > >>> > > to
> > > > > > > > >>> > > > be
> > > > > > > > >>> > > > > > > > >> appended
> > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker,
> > causing
> > > a
> > > > > > > > >>> transaction to
> > > > > > > > >>> > > > hang.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A similar
> > issue
> > > > (not
> > > > > > > > >>> mentioned
> > > > > > > > >>> > in
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > > >> motivation)
> > > > > > > > >>> > > > > > > > >> > > > could
> > > > > > > > >>> > > > > > > > >> > > > > > > > happen
> > > > > > > > >>> > > > > > > > >> > > > > > > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker in
> the
> > > > > > > > coordinator's
> > > > > > > > >>> log.
> > > > > > > > >>> > > For
> > > > > > > > >>> > > > > > > > example,
> > > > > > > > >>> > > > > > > > >> > it's
> > > > > > > > >>> > > > > > > > >> > > > > > possible
> > > > > > > > >>> > > > > > > > >> > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> EndTxnRequest
> > to
> > > > be
> > > > > > > > delayed
> > > > > > > > >>> on
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > > > > >> coordinator.
> > > > > > > > >>> > > > > > > > >> > By
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > time
> > > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> EndTxnRequest
> > is
> > > > > > > > processed,
> > > > > > > > >>> it's
> > > > > > > > >>> > > > > > possible
> > > > > > > > >>> > > > > > > > that
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > >>> > > > > > > > >> > > > > > > > txn
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completed
> and
> > a
> > > > new
> > > > > > txn
> > > > > > > > has
> > > > > > > > >>> > > started.
> > > > > > > > >>> > > > > > > > >> Currently,
> > > > > > > > >>> > > > > > > > >> > > since
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped on
> > every
> > > > txn,
> > > > > > the
> > > > > > > > >>> delayed
> > > > > > > > >>> > > > > > > > EndTxnRequest
> > > > > > > > >>> > > > > > > > >> > will
> > > > > > > > >>> > > > > > > > >> > > > add
> > > > > > > > >>> > > > > > > > >> > > > > > an
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare
> marker
> > > > (and
> > > > > > > > >>> eventually a
> > > > > > > > >>> > > > > > complete
> > > > > > > > >>> > > > > > > > >> marker)
> > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't cause
> > the
> > > > > > > > transaction
> > > > > > > > >>> to
> > > > > > > > >>> > > hang,
> > > > > > > > >>> > > > > but
> > > > > > > > >>> > > > > > > it
> > > > > > > > >>> > > > > > > > >> will
> > > > > > > > >>> > > > > > > > >> > > > break
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The proposal
> > in
> > > > this
> > > > > > KIP
> > > > > > > > >>> will
> > > > > > > > >>> > > > address
> > > > > > > > >>> > > > > > this
> > > > > > > > >>> > > > > > > > >> issue
> > > > > > > > >>> > > > > > > > >> > > too.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.
> > "However, I
> > > > was
> > > > > > > > >>> writing it
> > > > > > > > >>> > so
> > > > > > > > >>> > > > > that
> > > > > > > > >>> > > > > > we
> > > > > > > > >>> > > > > > > > can
> > > > > > > > >>> > > > > > > > >> > > > > > distinguish
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old clients
> > > where
> > > > we
> > > > > > > don't
> > > > > > > > >>> have
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > ability
> > > > > > > > >>> > > > > > > > do
> > > > > > > > >>> > > > > > > > >> > this
> > > > > > > > >>> > > > > > > > >> > > > > > > operation
> > > > > > > > >>> > > > > > > > >> > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients that
> > > can.
> > > > > (Old
> > > > > > > > >>> clients
> > > > > > > > >>> > > don't
> > > > > > > > >>> > > > > > bump
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > epoch
> > > > > > > > >>> > > > > > > > >> > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > so
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for sure
> > the
> > > > > write
> > > > > > > > >>> belongs
> > > > > > > > >>> > to
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > given
> > > > > > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I am
> > > > wondering
> > > > > > why
> > > > > > > > we
> > > > > > > > >>> need
> > > > > > > > >>> > > to
> > > > > > > > >>> > > > > > > > >> distinguish
> > > > > > > > >>> > > > > > > > >> > > > whether
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written by
> the
> > > old
> > > > > and
> > > > > > > the
> > > > > > > > >>> new
> > > > > > > > >>> > > > client.
> > > > > > > > >>> > > > > > > Could
> > > > > > > > >>> > > > > > > > >> you
> > > > > > > > >>> > > > > > > > >> > > > > describe
> > > > > > > > >>> > > > > > > > >> > > > > > > > what
> > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > differently
> if
> > > we
> > > > > know
> > > > > > > the
> > > > > > > > >>> > marker
> > > > > > > > >>> > > is
> > > > > > > > >>> > > > > > > written
> > > > > > > > >>> > > > > > > > >> by
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > client?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If we
> do
> > > > need
> > > > > a
> > > > > > > way
> > > > > > > > to
> > > > > > > > >>> > > > > distinguish
> > > > > > > > >>> > > > > > > > >> whether
> > > > > > > > >>> > > > > > > > >> > > the
> > > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old and
> > the
> > > > new
> > > > > > > > client.
> > > > > > > > >>> > Would
> > > > > > > > >>> > > it
> > > > > > > > >>> > > > > be
> > > > > > > > >>> > > > > > > > >> simpler
> > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > >>> > > > > > > > >> > > > just
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field
> instead
> > of
> > > > > > > > indirectly
> > > > > > > > >>> > > through
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> previous
> > > > > > > > >>> > > > > > > > >> > > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3 It's
> not
> > > > clear
> > > > > > to
> > > > > > > me
> > > > > > > > >>> why
> > > > > > > > >>> > we
> > > > > > > > >>> > > > only
> > > > > > > > >>> > > > > > add
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the complete
> > > > marker,
> > > > > > but
> > > > > > > > >>> not in
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > > prepare
> > > > > > > > >>> > > > > > > > >> > marker.
> > > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > want
> > > > > > > > >>> > > > > > > > >> > > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether a
> > marker
> > > > is
> > > > > > > > written
> > > > > > > > >>> by
> > > > > > > > >>> > the
> > > > > > > > >>> > > > new
> > > > > > > > >>> > > > > > > > client
> > > > > > > > >>> > > > > > > > >> or
> > > > > > > > >>> > > > > > > > >> > > not,
> > > > > > > > >>> > > > > > > > >> > > > > it
> > > > > > > > >>> > > > > > > > >> > > > > > > > seems
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do this
> > > > > > consistently
> > > > > > > > for
> > > > > > > > >>> all
> > > > > > > > >>> > > > > markers.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4 What
> > about
> > > > the
> > > > > > > > >>> > > > > TransactionLogValue
> > > > > > > > >>> > > > > > > > >> record
> > > > > > > > >>> > > > > > > > >> > > > > > > representing
> > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state?
> Should
> > we
> > > > > also
> > > > > > > > >>> > distinguish
> > > > > > > > >>> > > > > > whether
> > > > > > > > >>> > > > > > > > it's
> > > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > > >>> > > > > > > > >> > > > > by
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > old
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new client?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In the
> > > > overflow
> > > > > > > case,
> > > > > > > > >>> it's
> > > > > > > > >>> > > > still
> > > > > > > > >>> > > > > > not
> > > > > > > > >>> > > > > > > > >> clear
> > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > >>> > > > > > > > >> > > me
> > > > > > > > >>> > > > > > > > >> > > > > why
> > > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous
> > produce
> > > > Id
> > > > > in
> > > > > > > the
> > > > > > > > >>> > prepare
> > > > > > > > >>> > > > > > marker
> > > > > > > > >>> > > > > > > > >> while
> > > > > > > > >>> > > > > > > > >> > > > writing
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the
> > complete
> > > > > > marker.
> > > > > > > > You
> > > > > > > > >>> > > > mentioned
> > > > > > > > >>> > > > > > that
> > > > > > > > >>> > > > > > > > >> it's
> > > > > > > > >>> > > > > > > > >> > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could
> > > downgrade
> > > > > > with
> > > > > > > > >>> either
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > > prepare
> > > > > > > > >>> > > > > > > > >> marker
> > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In either
> > case,
> > > > the
> > > > > > > > >>> downgraded
> > > > > > > > >>> > > > > > coordinator
> > > > > > > > >>> > > > > > > > >> should
> > > > > > > > >>> > > > > > > > >> > > see
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > same
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > (probably
> the
> > > > > previous
> > > > > > > > >>> produce
> > > > > > > > >>> > > Id),
> > > > > > > > >>> > > > > > right?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed, Dec
> > 20,
> > > > 2023
> > > > > > at
> > > > > > > > >>> 6:00 PM
> > > > > > > > >>> > > > > Justine
> > > > > > > > >>> > > > > > > > Olshan
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey Jun,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks for
> > > > taking
> > > > > a
> > > > > > > look
> > > > > > > > >>> at
> > > > > > > > >>> > the
> > > > > > > > >>> > > > KIP
> > > > > > > > >>> > > > > > > again.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100. For
> the
> > > > epoch
> > > > > > > > >>> overflow
> > > > > > > > >>> > > case,
> > > > > > > > >>> > > > > only
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > marker
> > > > > > > > >>> > > > > > > > >> > > > > will
> > > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > > >>> > > > > > > > >> > > > > > > > > max
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps the
> > > > behavior
> > > > > > of
> > > > > > > > the
> > > > > > > > >>> rest
> > > > > > > > >>> > > of
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> markers
> > > > > > > > >>> > > > > > > > >> > > where
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > last
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch of
> the
> > > > > > > transaction
> > > > > > > > >>> > > records +
> > > > > > > > >>> > > > > 1.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101. You
> are
> > > > > correct
> > > > > > > > that
> > > > > > > > >>> we
> > > > > > > > >>> > > don't
> > > > > > > > >>> > > > > > need
> > > > > > > > >>> > > > > > > to
> > > > > > > > >>> > > > > > > > >> > write
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the
> same.
> > > > > > However,
> > > > > > > I
> > > > > > > > >>> was
> > > > > > > > >>> > > > writing
> > > > > > > > >>> > > > > it
> > > > > > > > >>> > > > > > > so
> > > > > > > > >>> > > > > > > > >> that
> > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > >>> > > > > > > > >> > > > can
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old
> clients
> > > > where
> > > > > we
> > > > > > > > don't
> > > > > > > > >>> > have
> > > > > > > > >>> > > > the
> > > > > > > > >>> > > > > > > > ability
> > > > > > > > >>> > > > > > > > >> do
> > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > >>> > > > > > > > >> > > > > > > > operation
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > clients
> that
> > > > can.
> > > > > > (Old
> > > > > > > > >>> clients
> > > > > > > > >>> > > > don't
> > > > > > > > >>> > > > > > > bump
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say for
> sure
> > > the
> > > > > > write
> > > > > > > > >>> belongs
> > > > > > > > >>> > > to
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > given
> > > > > > > > >>> > > > > > > > >> > > > > > transaction).
> > > > > > > > >>> > > > > > > > >> > > > > > > > If
> > > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn
> > request
> > > > > from
> > > > > > a
> > > > > > > > new
> > > > > > > > >>> > > client,
> > > > > > > > >>> > > > we
> > > > > > > > >>> > > > > > > will
> > > > > > > > >>> > > > > > > > >> fill
> > > > > > > > >>> > > > > > > > >> > > this
> > > > > > > > >>> > > > > > > > >> > > > > > > field.
> > > > > > > > >>> > > > > > > > >> > > > > > > > We
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > guarantee
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that any
> > > EndTxn
> > > > > > > requests
> > > > > > > > >>> with
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > same
> > > > > > > > >>> > > > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > are
> > > > > > > > >>> > > > > > > > >> > > > from
> > > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > same
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the same
> > > > > > transaction.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102. In
> > > prepare
> > > > > > phase,
> > > > > > > > we
> > > > > > > > >>> have
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > same
> > > > > > > > >>> > > > > > > > >> > producer
> > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had. It is
> > the
> > > > > > > producer
> > > > > > > > >>> ID and
> > > > > > > > >>> > > > epoch
> > > > > > > > >>> > > > > > > that
> > > > > > > > >>> > > > > > > > >> are
> > > > > > > > >>> > > > > > > > >> > on
> > > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > In
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase, we
> > stay
> > > > the
> > > > > > > same
> > > > > > > > >>> unless
> > > > > > > > >>> > > it
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > > > the
> > > > > > > > >>> > > > > > > > >> > overflow
> > > > > > > > >>> > > > > > > > >> > > > > case.
> > > > > > > > >>> > > > > > > > >> > > > > > > In
> > > > > > > > >>> > > > > > > > >> > > > > > > > > that
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set the
> > > producer
> > > > > ID
> > > > > > to
> > > > > > > > >>> the new
> > > > > > > > >>> > > one
> > > > > > > > >>> > > > > we
> > > > > > > > >>> > > > > > > > >> generated
> > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > complete.
> > This
> > > > is
> > > > > > for
> > > > > > > > >>> > downgrade
> > > > > > > > >>> > > > > > > > >> compatibility.
> > > > > > > > >>> > > > > > > > >> > > The
> > > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety
> > guards
> > > > for
> > > > > > > > retries
> > > > > > > > >>> and
> > > > > > > > >>> > > > > > failovers.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In prepare
> > > phase
> > > > > for
> > > > > > > > epoch
> > > > > > > > >>> > > > overflow
> > > > > > > > >>> > > > > > case
> > > > > > > > >>> > > > > > > > >> only
> > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > >>> > > > > > > > >> > > > > store
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID. This
> is
> > > for
> > > > > the
> > > > > > > case
> > > > > > > > >>> where
> > > > > > > > >>> > > we
> > > > > > > > >>> > > > > > reload
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > transaction
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > prepare
> > state.
> > > > > Once
> > > > > > > the
> > > > > > > > >>> > > > transaction
> > > > > > > > >>> > > > > is
> > > > > > > > >>> > > > > > > > >> > committed,
> > > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > can
> > > > > > > > >>> > > > > > > > >> > > > > > > > use
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the
> > client
> > > > > > already
> > > > > > > is
> > > > > > > > >>> > using.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In commit
> > > phase,
> > > > > we
> > > > > > > > store
> > > > > > > > >>> the
> > > > > > > > >>> > > > > previous
> > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > >>> > > > > > > > >> > > > > > > > of
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I think it
> > is
> > > > > easier
> > > > > > > to
> > > > > > > > >>> think
> > > > > > > > >>> > of
> > > > > > > > >>> > > > it
> > > > > > > > >>> > > > > as
> > > > > > > > >>> > > > > > > > just
> > > > > > > > >>> > > > > > > > >> how
> > > > > > > > >>> > > > > > > > >> > > we
> > > > > > > > >>> > > > > > > > >> > > > > were
> > > > > > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and epoch
> > > > before,
> > > > > > with
> > > > > > > > >>> some
> > > > > > > > >>> > > extra
> > > > > > > > >>> > > > > > > > bookeeping
> > > > > > > > >>> > > > > > > > >> > and
> > > > > > > > >>> > > > > > > > >> > > > edge
> > > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged
> > fields.
> > > > We
> > > > > > have
> > > > > > > > to
> > > > > > > > >>> do
> > > > > > > > >>> > it
> > > > > > > > >>> > > > this
> > > > > > > > >>> > > > > > way
> > > > > > > > >>> > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > > > > > >>> > > > > > > > >> > > > > > > > > with
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > downgrades.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103. Next
> > > > producer
> > > > > > ID
> > > > > > > is
> > > > > > > > >>> for
> > > > > > > > >>> > > > prepare
> > > > > > > > >>> > > > > > > > status
> > > > > > > > >>> > > > > > > > >> and
> > > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after
> > > complete.
> > > > > The
> > > > > > > > >>> reason why
> > > > > > > > >>> > > we
> > > > > > > > >>> > > > > need
> > > > > > > > >>> > > > > > > two
> > > > > > > > >>> > > > > > > > >> > > separate
> > > > > > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > backwards
> > > > > > > compatibility.
> > > > > > > > >>> We
> > > > > > > > >>> > need
> > > > > > > > >>> > > > to
> > > > > > > > >>> > > > > > keep
> > > > > > > > >>> > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > same
> > > > > > > > >>> > > > > > > > >> > > > > > > semantics
> > > > > > > > >>> > > > > > > > >> > > > > > > > > for
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged
> > > field
> > > > > in
> > > > > > > case
> > > > > > > > >>> we
> > > > > > > > >>> > > > > downgrade.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104. We
> set
> > > the
> > > > > > fields
> > > > > > > > as
> > > > > > > > >>> we
> > > > > > > > >>> > do
> > > > > > > > >>> > > in
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > transactional
> > > > > > > > >>> > > > > > > > >> > > > > > > state
> > > > > > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do this
> for
> > > > > > > > compatibility
> > > > > > > > >>> --
> > > > > > > > >>> > if
> > > > > > > > >>> > > we
> > > > > > > > >>> > > > > > > > >> downgrade,
> > > > > > > > >>> > > > > > > > >> > we
> > > > > > > > >>> > > > > > > > >> > > > will
> > > > > > > > >>> > > > > > > > >> > > > > > > only
> > > > > > > > >>> > > > > > > > >> > > > > > > > > have
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged
> > > > fields)
> > > > > > It
> > > > > > > > >>> will be
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > old
> > > > > > > > >>> > > > > > > > >> producer
> > > > > > > > >>> > > > > > > > >> > ID
> > > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > max
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope this
> > > helps.
> > > > > Let
> > > > > > > me
> > > > > > > > >>> know
> > > > > > > > >>> > if
> > > > > > > > >>> > > > you
> > > > > > > > >>> > > > > > have
> > > > > > > > >>> > > > > > > > >> > further
> > > > > > > > >>> > > > > > > > >> > > > > > > questions.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Justine
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On Wed,
> Dec
> > > 20,
> > > > > 2023
> > > > > > > at
> > > > > > > > >>> > 3:33 PM
> > > > > > > > >>> > > > Jun
> > > > > > > > >>> > > > > > Rao
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > <jun@confluent.io.invalid
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi,
> > Justine,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It seems
> > > that
> > > > > you
> > > > > > > have
> > > > > > > > >>> made
> > > > > > > > >>> > > some
> > > > > > > > >>> > > > > > > changes
> > > > > > > > >>> > > > > > > > >> to
> > > > > > > > >>> > > > > > > > >> > > > KIP-890
> > > > > > > > >>> > > > > > > > >> > > > > > > since
> > > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > particular,
> > > we
> > > > > are
> > > > > > > > >>> changing
> > > > > > > > >>> > > the
> > > > > > > > >>> > > > > > format
> > > > > > > > >>> > > > > > > > of
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionLogValue.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > comments
> > > > related
> > > > > > to
> > > > > > > > >>> that.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100.
> Just
> > to
> > > > be
> > > > > > > clear.
> > > > > > > > >>> The
> > > > > > > > >>> > > > > overflow
> > > > > > > > >>> > > > > > > case
> > > > > > > > >>> > > > > > > > >> > (i.e.
> > > > > > > > >>> > > > > > > > >> > > > > when a
> > > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> generated)
> > > is
> > > > > when
> > > > > > > the
> > > > > > > > >>> > current
> > > > > > > > >>> > > > > epoch
> > > > > > > > >>> > > > > > > > >> equals
> > > > > > > > >>> > > > > > > > >> > to
> > > > > > > > >>> > > > > > > > >> > > > max
> > > > > > > > >>> > > > > > > > >> > > > > -
> > > > > > > > >>> > > > > > > > >> > > > > > 1
> > > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101. For
> > the
> > > > > "not
> > > > > > > > epoch
> > > > > > > > >>> > > > overflow"
> > > > > > > > >>> > > > > > > case,
> > > > > > > > >>> > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > write
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > tagged
> > field
> > > > in
> > > > > > the
> > > > > > > > >>> complete
> > > > > > > > >>> > > > > phase.
> > > > > > > > >>> > > > > > Do
> > > > > > > > >>> > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > need
> > > > > > > > >>> > > > > > > > >> > > to
> > > > > > > > >>> > > > > > > > >> > > > > do
> > > > > > > > >>> > > > > > > > >> > > > > > > that
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > doesn't
> > > change
> > > > > in
> > > > > > > this
> > > > > > > > >>> case?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102. It
> > > seems
> > > > > that
> > > > > > > the
> > > > > > > > >>> > meaning
> > > > > > > > >>> > > > for
> > > > > > > > >>> > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> ProducerId/ProducerEpoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > TransactionLogValue
> > > > > > > > >>> changes
> > > > > > > > >>> > > > > > depending
> > > > > > > > >>> > > > > > > on
> > > > > > > > >>> > > > > > > > >> the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > > TransactionStatus
> > > > > > > > is
> > > > > > > > >>> > > > ongoing,
> > > > > > > > >>> > > > > > they
> > > > > > > > >>> > > > > > > > >> > > represent
> > > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > current
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > current
> > > > > > > > >>> ProducerEpoch.
> > > > > > > > >>> > > When
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > > > > > >>> > > > > > > > >> > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> PrepareCommit/PrepareAbort,
> > > > > > > > >>> > > they
> > > > > > > > >>> > > > > > > > represent
> > > > > > > > >>> > > > > > > > >> > the
> > > > > > > > >>> > > > > > > > >> > > > > > current
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > > > > > ProducerEpoch.
> > > > > > > > >>> When the
> > > > > > > > >>> > > > > > > > >> > TransactionStatus
> > > > > > > > >>> > > > > > > > >> > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > further
> > > depend
> > > > > on
> > > > > > > > >>> whether
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > epoch
> > > > > > > > >>> > > > > > > > >> overflows
> > > > > > > > >>> > > > > > > > >> > > or
> > > > > > > > >>> > > > > > > > >> > > > > not.
> > > > > > > > >>> > > > > > > > >> > > > > > > If
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> overflow,
> > > they
> > > > > > > > represent
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > > current
> > > > > > > > >>> > > > > > > > >> > > ProducerId
> > > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > ProducerEpoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > (max).
> > > > > Otherwise,
> > > > > > > they
> > > > > > > > >>> > > represent
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > > > > newly
> > > > > > > > >>> > > > > > > > >> > > > > generated
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > ProducerEpoch
> > > > of
> > > > > > 0.
> > > > > > > Is
> > > > > > > > >>> that
> > > > > > > > >>> > > > right?
> > > > > > > > >>> > > > > > > This
> > > > > > > > >>> > > > > > > > >> seems
> > > > > > > > >>> > > > > > > > >> > > not
> > > > > > > > >>> > > > > > > > >> > > > > > easy
> > > > > > > > >>> > > > > > > > >> > > > > > > to
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we
> provide
> > > > some
> > > > > > > > examples
> > > > > > > > >>> > like
> > > > > > > > >>> > > > what
> > > > > > > > >>> > > > > > > Artem
> > > > > > > > >>> > > > > > > > >> has
> > > > > > > > >>> > > > > > > > >> > > done
> > > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > considered a
> > > > > > simpler
> > > > > > > > >>> design
> > > > > > > > >>> > > > where
> > > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId/ProducerEpoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > represent
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the same
> > > value
> > > > > > (e.g.
> > > > > > > > >>> for the
> > > > > > > > >>> > > > > current
> > > > > > > > >>> > > > > > > > >> > > transaction)
> > > > > > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > TransactionStatus
> > > > > > > and
> > > > > > > > >>> epoch
> > > > > > > > >>> > > > > > overflow?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103.
> It's
> > > not
> > > > > > clear
> > > > > > > to
> > > > > > > > >>> me
> > > > > > > > >>> > why
> > > > > > > > >>> > > we
> > > > > > > > >>> > > > > > need
> > > > > > > > >>> > > > > > > 3
> > > > > > > > >>> > > > > > > > >> > fields:
> > > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > PrevProducerId,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > NextProducerId.
> > > > > > > Could
> > > > > > > > we
> > > > > > > > >>> > just
> > > > > > > > >>> > > > have
> > > > > > > > >>> > > > > > > > >> ProducerId
> > > > > > > > >>> > > > > > > > >> > > and
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104. For
> > > > > > > > >>> > > WriteTxnMarkerRequests,
> > > > > > > > >>> > > > > if
> > > > > > > > >>> > > > > > > the
> > > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we set
> the
> > > > > > > producerId
> > > > > > > > >>> and
> > > > > > > > >>> > the
> > > > > > > > >>> > > > > > > > >> producerEpoch?
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > > >>> > > > > > > > >> > > >
> > > > > > > > >>> > > > > > > > >> > >
> > > > > > > > >>> > > > > > > > >> >
> > > > > > > > >>> > > > > > > > >>
> > > > > > > > >>> > > > > > > > >
> > > > > > > > >>> > > > > > > >
> > > > > > > > >>> > > > > > >
> > > > > > > > >>> > > > > >
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > >
> > > > > > > > >>> > >
> > > > > > > > >>> >
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Calvin Liu <ca...@confluent.io.INVALID>]

Hey team,
In order to address the performance issue in the
https://issues.apache.org/jira/browse/KAFKA-18654. We are making a change
to retry the AddPartitionToTxn requests on the server side when hitting the
CONCURRENT_TRANSACTION error. This error is common after the transaction is
committed but before the markers are fully propagated to all the involved
partitions. With the server side retry, the client can avoid the frequent
and expensive retry for the produce request. In order to better control the
retry behavior, we proposed to add 2 following configs.

   - *add.partitions.to.txn.max.timeout.ms
   <http://add.partitions.to.txn.max.timeout.ms>* defines the maximum retry
   timeout when the server attempts to add the partition to the transaction.
   - *add.partitions.to.txn.retry.backoff.ms
   <http://add.partitions.to.txn.retry.backoff.ms>* defines how frequently
   the server will retry the AddPartitionToTxn.

Thanks!

On Thu, Nov 14, 2024 at 1:52 PM Justine Olshan <jo...@confluent.io.invalid>
wrote:

> Hey folks, just wanted to share another quick update with KIP-890.
>
> Making the change to bump epoch after every transaction means that we no
> longer need to call InitProducerId during the life of the producer to fence
> requests from previous transactions. With KIP-890 part 2, the new client
> will no longer call InitProducerId except on startup to fence a previous
> instance.
>
> With this change, some other calls to InitProducerId were inspected
> including the call after receiving an InvalidPidMappingException. This
> exception was changed to abortable as part of KIP-360: Improve reliability
> of idempotent/transactional producer
> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820
> >.
> However, this change means that we can violate EOS guarantees. As an
> example:
>
> Consider an application that is copying data from one partition to another
>
>    - Application instance A processes to offset 4
>    - Application instance B comes up and fences application instance A
>    - Application instance B processes to offset 5
>    - Application instances A and B are idle for
> transaction.id.expiration.ms,
>    transaction id expires on server
>    - Application instance A attempts to process offset 5 (since in its
>    view, that is next) -- if we recover from invalid pid mapping, we can
>    duplicate this processing
>
> Thus, INVALID_PID_MAPPING should be fatal to the producer and as part of
> KIP-890 the new client in 4.0 will treat the exception as fatal.
>
> This is consistent with KIP-1050: Consistent error handling for
> Transactions
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions
> >
> where
> errors that are fatal to the producer are in the "application recoverable"
> category. This is a grouping that indicates to the client that the producer
> needs to restart and recovery on the application side is necessary.
> KIP-1050 is approved so we are consistent with that decision.
>
> Let me know if there are any questions. I've updated the KIP to reflect
> this change.
>
> Justine
>
> On Wed, Jul 10, 2024 at 3:34 PM Jun Rao <ju...@confluent.io.invalid> wrote:
>
> > Sounds good, Justine. It would be useful to document that in the KIP.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Jul 10, 2024 at 2:59 PM Justine Olshan
> > <jo...@confluent.io.invalid>
> > wrote:
> >
> > > The client will send the newest EndTxn request version if and only if
> > both
> > > the client and the server support kip-890 part 2.
> > > We set the value in the record based on the EndTxn version.
> > >
> > > Justine
> > >
> > > On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <ju...@confluent.io.invalid>
> > wrote:
> > >
> > > > Hi, Justine,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > 120. If the broker sends TV Y for the finalized version in
> > > > ApiVersionResponse, but the client doesn't support Y, how does the
> > broker
> > > > know the TV that the client supports?
> > > >
> > > > Jun
> > > >
> > > > On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> > > > <jo...@confluent.io.invalid>
> > > > wrote:
> > > >
> > > > > Hey Jun,
> > > > >
> > > > > No worries. Work on this KIP has been blocked for a bit anyways --
> > > > catching
> > > > > up and rereading what I wrote :)
> > > > >
> > > > > 120. ClientTransactionProtocolVersion is the transaction version as
> > > > defined
> > > > > by the highest transaction version (feature version value)
> supported
> > by
> > > > the
> > > > > client and the server. This works by the broker sending an
> > > > > ApiVersionsRequest to the client with the finalized version.
> Assuming
> > > > > kip-890 part 2 is enabled by transaction version Y, if this request
> > > > > contains finalized version Y and the client has the logic to set
> this
> > > > > field, it will set Y. If the server has Y - 1 (kip 890 part 2 not
> > > enable)
> > > > > the client will send Y - 1, even though the client has the ability
> to
> > > > > support kip-890 part 2.
> > > > >
> > > > > 121. You are correct that this is not needed. However, currently
> that
> > > > field
> > > > > is already being set in memory -- just not written to disk. I think
> > it
> > > is
> > > > > ok to write it to disk though. Let me know if you think otherwise.
> > > > >
> > > > > Justine
> > > > >
> > > > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao <ju...@confluent.io.invalid>
> > > > wrote:
> > > > >
> > > > > > Hi, Justine,
> > > > > >
> > > > > > Thanks for the update and sorry for the late reply.
> > > > > >
> > > > > > 120. I am wondering what value is used for
> > > > > > ClientTransactionProtocolVersion. Is it the version of the
> > > > EndTxnRequest?
> > > > > >
> > > > > > 121. Earlier, you made the change to set lastProducerId in
> PREPARE
> > to
> > > > > > indicate that the marker is written for the new client. With the
> > new
> > > > > > ClientTransactionProtocolVersion field, it seems this is no
> longer
> > > > > > necessary.
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > > > > <jo...@confluent.io.invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi there -- another update!
> > > > > > >
> > > > > > > When looking into the implementation for the safe epoch bumps I
> > > > > realized
> > > > > > > that we are already populating previousProducerID in memory as
> > part
> > > > of
> > > > > > > KIP-360.
> > > > > > > If we are to start using flexible fields, it is better to
> always
> > > use
> > > > > this
> > > > > > > information and have an explicit (tagged) field to indicate
> > whether
> > > > the
> > > > > > > client supports KIP-890 part 2.
> > > > > > >
> > > > > > > I've included the extra field and how it is set in the KIP.
> I've
> > > also
> > > > > > > updated the KIP to explain that we will be setting the tagged
> > > fields
> > > > > when
> > > > > > > they are available for all transitions.
> > > > > > >
> > > > > > > Finally, I added clearer text about the transaction protocol
> > > versions
> > > > > > > included with this KIP. 1 for flexible transaction state
> records
> > > and
> > > > 2
> > > > > > for
> > > > > > > KIP-890 part 2 enablement.
> > > > > > >
> > > > > > > Justine
> > > > > > >
> > > > > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <
> > > jolshan@confluent.io
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey there -- small update to the KIP,
> > > > > > > >
> > > > > > > > The KIP mentioned introducing ABORTABLE_ERROR and bumping
> > > > > > TxnOffsetCommit
> > > > > > > > and Produce requests. I've changed the name in the KIP to
> > > > > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > > > > AbortableTransactionException to match the pattern we had for
> > > other
> > > > > > > errors.
> > > > > > > > I also mentioned bumping all 6 transactional APIs so we can
> > > future
> > > > > > > > proof/support the error on the client going forward. If a
> > future
> > > > > change
> > > > > > > > wants to have an error scenario that requires us to abort the
> > > > > > > transaction,
> > > > > > > > we can rely on the 3.8+ clients to support it. We ran into
> > issues
> > > > > > finding
> > > > > > > > good/generic error codes that older clients could support
> while
> > > > > working
> > > > > > > on
> > > > > > > > this KIP, so this should help in the future.
> > > > > > > >
> > > > > > > > The features discussion is still ongoing in KIP-1022. Will
> > update
> > > > > again
> > > > > > > > here when that concludes.
> > > > > > > >
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> > > > jolshan@confluent.io>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> I don't think AddPartitions is a good example since we
> > currenly
> > > > > don't
> > > > > > > >> gate the version on TV or MV. (We only set a different flag
> > > > > depending
> > > > > > on
> > > > > > > >> the TV)
> > > > > > > >>
> > > > > > > >> Even if we did want to gate it on TV, I think the idea is to
> > > move
> > > > > away
> > > > > > > >> from MV gating inter broker protocols. Ideally we can get
> to a
> > > > state
> > > > > > > where
> > > > > > > >> MV is just used for metadata changes.
> > > > > > > >>
> > > > > > > >> I think some of this discussion might fit more with the
> > feature
> > > > > > version
> > > > > > > >> KIP, so I can try to open that up soon. Until we settle
> that,
> > > some
> > > > > of
> > > > > > > the
> > > > > > > >> work in KIP-890 is blocked.
> > > > > > > >>
> > > > > > > >> Justine
> > > > > > > >>
> > > > > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao
> > <jun@confluent.io.invalid
> > > >
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >>> Hi, Justine,
> > > > > > > >>>
> > > > > > > >>> Thanks for the reply.
> > > > > > > >>>
> > > > > > > >>> Since AddPartitions is an inter broker request, will its
> > > version
> > > > be
> > > > > > > gated
> > > > > > > >>> only by TV or other features like MV too? For example, if
> we
> > > need
> > > > > to
> > > > > > > >>> change
> > > > > > > >>> the protocol for AddPartitions for reasons other than txn
> > > > > > verification
> > > > > > > in
> > > > > > > >>> the future, will the new version be gated by a new MV? If
> so,
> > > > does
> > > > > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > > > > >>>
> > > > > > > >>> Jun
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > > >>> > One TV gates the flexible feature version (no rpcs
> > involved,
> > > > only
> > > > > > the
> > > > > > > >>> > transactional records that should only be gated by TV)
> > > > > > > >>> > Another TV gates the ability to turn on kip-890 part 2.
> > This
> > > > > would
> > > > > > > >>> gate the
> > > > > > > >>> > version of Produce and EndTxn (likely only used by
> > > > transactions),
> > > > > > and
> > > > > > > >>> > specifies a flag in AddPartitionsToTxn though the version
> > is
> > > > > > already
> > > > > > > >>> used
> > > > > > > >>> > without TV.
> > > > > > > >>> >
> > > > > > > >>> > I think the only concern is the Produce request and we
> > could
> > > > > > consider
> > > > > > > >>> work
> > > > > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > > > > >>> >
> > > > > > > >>> > Justine
> > > > > > > >>> >
> > > > > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> > > > <jun@confluent.io.invalid
> > > > > >
> > > > > > > >>> wrote:
> > > > > > > >>> >
> > > > > > > >>> > > Hi, Justine,
> > > > > > > >>> > >
> > > > > > > >>> > > Which PRC/record protocols will TV guard? Going
> forward,
> > > will
> > > > > > those
> > > > > > > >>> > > PRC/record protocols only be guarded by TV and not by
> > other
> > > > > > > features
> > > > > > > >>> like
> > > > > > > >>> > > MV?
> > > > > > > >>> > >
> > > > > > > >>> > > Thanks,
> > > > > > > >>> > >
> > > > > > > >>> > > Jun
> > > > > > > >>> > >
> > > > > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > > > > >>> > <jolshan@confluent.io.invalid
> > > > > > > >>> > > >
> > > > > > > >>> > > wrote:
> > > > > > > >>> > >
> > > > > > > >>> > > > Hi Jun,
> > > > > > > >>> > > >
> > > > > > > >>> > > > Sorry I think I misunderstood your question or
> answered
> > > > > > > >>> incorrectly.
> > > > > > > >>> > The
> > > > > > > >>> > > TV
> > > > > > > >>> > > > version should ideally be fully independent from MV.
> > > > > > > >>> > > > At least for the changes I proposed, TV should not
> > affect
> > > > MV
> > > > > > and
> > > > > > > MV
> > > > > > > >>> > > should
> > > > > > > >>> > > > not affect TV/
> > > > > > > >>> > > >
> > > > > > > >>> > > > I think if we downgrade TV, only that feature should
> > > > > downgrade.
> > > > > > > >>> > Likewise
> > > > > > > >>> > > > the same with MV. The finalizedFeatures should just
> > > reflect
> > > > > the
> > > > > > > >>> feature
> > > > > > > >>> > > > downgrade we made.
> > > > > > > >>> > > >
> > > > > > > >>> > > > I also plan to write a new KIP for managing the disk
> > > format
> > > > > and
> > > > > > > >>> upgrade
> > > > > > > >>> > > > tool as we will need new flags to support these
> > features.
> > > > > That
> > > > > > > >>> should
> > > > > > > >>> > > help
> > > > > > > >>> > > > clarify some things.
> > > > > > > >>> > > >
> > > > > > > >>> > > > Justine
> > > > > > > >>> > > >
> > > > > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > > > > <jun@confluent.io.invalid
> > > > > > > >
> > > > > > > >>> > > wrote:
> > > > > > > >>> > > >
> > > > > > > >>> > > > > Hi, Justine,
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > Thanks for the reply.
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > So, if we downgrade TV, we could implicitly
> downgrade
> > > > > another
> > > > > > > >>> feature
> > > > > > > >>> > > > (say
> > > > > > > >>> > > > > MV) that has dependency (e.g. RPC). What would we
> > > return
> > > > > for
> > > > > > > >>> > > > > FinalizedFeatures for MV in ApiVersionsResponse in
> > that
> > > > > case?
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > Thanks,
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > Jun
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > > > > >>> > > > <jolshan@confluent.io.invalid
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > > wrote:
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > > Hey Jun,
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > > > Yes, the idea is that if we downgrade TV
> > (transaction
> > > > > > > version)
> > > > > > > >>> we
> > > > > > > >>> > > will
> > > > > > > >>> > > > > stop
> > > > > > > >>> > > > > > using the add partitions to txn optimization and
> > stop
> > > > > > writing
> > > > > > > >>> the
> > > > > > > >>> > > > > flexible
> > > > > > > >>> > > > > > feature version of the log.
> > > > > > > >>> > > > > > In the compatibility section I included some
> > > > explanations
> > > > > > on
> > > > > > > >>> how
> > > > > > > >>> > this
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > done.
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > > > Thanks,
> > > > > > > >>> > > > > > Justine
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > > > > >>> <ju...@confluent.io.invalid>
> > > > > > > >>> > > > > wrote:
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > > > > Hi, Justine,
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > > > Thanks for the update.
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > > > If we ever downgrade the transaction feature,
> any
> > > > > feature
> > > > > > > >>> > depending
> > > > > > > >>> > > > on
> > > > > > > >>> > > > > > > changes on top of those RPC/record
> > > > > > > >>> > > > > > > (AddPartitionsToTxnRequest/TransactionLogValue)
> > > > changes
> > > > > > > made
> > > > > > > >>> in
> > > > > > > >>> > > > KIP-890
> > > > > > > >>> > > > > > > will be automatically downgraded too?
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > > > Jun
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine Olshan
> > > > > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > > > > >>> > > > > > > wrote:
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > > > > Hey Jun,
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > > > I wanted to get back to you about your
> > questions
> > > > > about
> > > > > > > >>> MV/IBP.
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > > > Looking at the options, I think it makes the
> > most
> > > > > sense
> > > > > > > to
> > > > > > > >>> > > create a
> > > > > > > >>> > > > > > > > separate feature for transactions and use
> that
> > to
> > > > > > version
> > > > > > > >>> gate
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > features
> > > > > > > >>> > > > > > > > we need to version gate (flexible
> transactional
> > > > state
> > > > > > > >>> records
> > > > > > > >>> > and
> > > > > > > >>> > > > > using
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > new protocol)
> > > > > > > >>> > > > > > > > I've updated the KIP to include this change.
> > > > > Hopefully
> > > > > > > >>> that's
> > > > > > > >>> > > > > > everything
> > > > > > > >>> > > > > > > we
> > > > > > > >>> > > > > > > > need for this KIP :)
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > > > Justine
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine
> Olshan
> > <
> > > > > > > >>> > > > jolshan@confluent.io
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > > > > > wrote:
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > > > > Thanks Jun,
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > > I will update the KIP with the prev field
> for
> > > > > prepare
> > > > > > > as
> > > > > > > >>> > well.
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > > PREPARE
> > > > > > > >>> > > > > > > > > producerId: x
> > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty or z
> if
> > y
> > > > will
> > > > > > > >>> overflow
> > > > > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > > COMPLETE
> > > > > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we overflowed
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > > Thanks again,
> > > > > > > >>> > > > > > > > > Justine
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun Rao
> > > > > > > >>> > > <jun@confluent.io.invalid
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > >> Hi, Justine,
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > > > > >>> > > > > > > > >> (1) My point was that the coordinator
> could
> > > fail
> > > > > > right
> > > > > > > >>> after
> > > > > > > >>> > > > > writing
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> prepare marker. When the new txn
> coordinator
> > > > > > generates
> > > > > > > >>> the
> > > > > > > >>> > > > > complete
> > > > > > > >>> > > > > > > > marker
> > > > > > > >>> > > > > > > > >> after the failover, it needs some field
> from
> > > the
> > > > > > > prepare
> > > > > > > >>> > > marker
> > > > > > > >>> > > > to
> > > > > > > >>> > > > > > > > >> determine whether it's written by the new
> > > > client.
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >> (2) The changing of the behavior sounds
> good
> > > to
> > > > > me.
> > > > > > We
> > > > > > > >>> only
> > > > > > > >>> > > want
> > > > > > > >>> > > > > to
> > > > > > > >>> > > > > > > > return
> > > > > > > >>> > > > > > > > >> success if the prepare state is written by
> > the
> > > > new
> > > > > > > >>> client.
> > > > > > > >>> > So,
> > > > > > > >>> > > > in
> > > > > > > >>> > > > > > the
> > > > > > > >>> > > > > > > > >> non-overflow case, it seems that we also
> > need
> > > > sth
> > > > > in
> > > > > > > the
> > > > > > > >>> > > prepare
> > > > > > > >>> > > > > > > marker
> > > > > > > >>> > > > > > > > to
> > > > > > > >>> > > > > > > > >> tell us whether it's written by the new
> > > client.
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >> 112. Thanks for the explanation. That
> sounds
> > > > good
> > > > > to
> > > > > > > me.
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >> Jun
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM Justine
> > > Olshan
> > > > > > > >>> > > > > > > > >> <jo...@confluent.io.invalid> wrote:
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >> > 101.3 I realized that I actually have
> two
> > > > > > questions.
> > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need
> to
> > > > write
> > > > > > the
> > > > > > > >>> > > previous
> > > > > > > >>> > > > > > > produce
> > > > > > > >>> > > > > > > > Id
> > > > > > > >>> > > > > > > > >> > tagged field in the end maker so that we
> > > know
> > > > if
> > > > > > the
> > > > > > > >>> > marker
> > > > > > > >>> > > is
> > > > > > > >>> > > > > > from
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> new
> > > > > > > >>> > > > > > > > >> > client. Since the end maker is derived
> > from
> > > > the
> > > > > > > >>> prepare
> > > > > > > >>> > > > marker,
> > > > > > > >>> > > > > > > should
> > > > > > > >>> > > > > > > > >> we
> > > > > > > >>> > > > > > > > >> > write the previous produce Id in the
> > prepare
> > > > > > marker
> > > > > > > >>> field
> > > > > > > >>> > > too?
> > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > >>> > > > > > > > >> > we will lose this information when
> > deriving
> > > > the
> > > > > > end
> > > > > > > >>> > marker.
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > The "previous" producer ID is in the
> > normal
> > > > > > producer
> > > > > > > >>> ID
> > > > > > > >>> > > field.
> > > > > > > >>> > > > > So
> > > > > > > >>> > > > > > > yes,
> > > > > > > >>> > > > > > > > >> we
> > > > > > > >>> > > > > > > > >> > need it in prepare and that was always
> the
> > > > plan.
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I will
> > > enumerate
> > > > > the
> > > > > > > >>> fields
> > > > > > > >>> > and
> > > > > > > >>> > > > add
> > > > > > > >>> > > > > > > them
> > > > > > > >>> > > > > > > > to
> > > > > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > > > > >>> > > > > > > > >> > Say we have producer ID x and epoch y.
> > When
> > > we
> > > > > > > >>> overflow
> > > > > > > >>> > > epoch
> > > > > > > >>> > > > y
> > > > > > > >>> > > > > we
> > > > > > > >>> > > > > > > get
> > > > > > > >>> > > > > > > > >> > producer ID Z.
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > PREPARE
> > > > > > > >>> > > > > > > > >> > producerId: x
> > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged field):
> > > empty
> > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty or
> z
> > > if y
> > > > > > will
> > > > > > > >>> > overflow
> > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > COMPLETE
> > > > > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged field):
> x
> > > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > > > > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we
> overflowed
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > (2) In the prepare phase, if we retry
> and
> > > see
> > > > > > epoch
> > > > > > > -
> > > > > > > >>> 1 +
> > > > > > > >>> > ID
> > > > > > > >>> > > > in
> > > > > > > >>> > > > > > last
> > > > > > > >>> > > > > > > > >> seen
> > > > > > > >>> > > > > > > > >> > fields and are issuing the same command
> > (ie
> > > > > commit
> > > > > > > not
> > > > > > > >>> > > abort),
> > > > > > > >>> > > > > we
> > > > > > > >>> > > > > > > > return
> > > > > > > >>> > > > > > > > >> > success. The logic before KIP-890 seems
> to
> > > > > return
> > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > >>> > > > > > > > >> > in this case. Are we intentionally
> making
> > > this
> > > > > > > change?
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > Hmm -- we would fence the producer if
> the
> > > > epoch
> > > > > is
> > > > > > > >>> bumped
> > > > > > > >>> > > and
> > > > > > > >>> > > > we
> > > > > > > >>> > > > > > > get a
> > > > > > > >>> > > > > > > > >> > lower epoch. Yes -- we are intentionally
> > > > adding
> > > > > > this
> > > > > > > >>> to
> > > > > > > >>> > > > prevent
> > > > > > > >>> > > > > > > > fencing.
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > 112. We already merged the code that
> adds
> > > the
> > > > > > > >>> VerifyOnly
> > > > > > > >>> > > field
> > > > > > > >>> > > > > in
> > > > > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which is an
> > inter
> > > > > > broker
> > > > > > > >>> > request.
> > > > > > > >>> > > > It
> > > > > > > >>> > > > > > > seems
> > > > > > > >>> > > > > > > > >> that
> > > > > > > >>> > > > > > > > >> > we didn't bump up the IBP for that. Do
> you
> > > > know
> > > > > > why?
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > We no longer need IBP for all
> interbroker
> > > > > requests
> > > > > > > as
> > > > > > > >>> > > > > ApiVersions
> > > > > > > >>> > > > > > > > should
> > > > > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > > > > >>> > > > > > > > >> > We also handle unsupported version
> errors
> > > > > > correctly
> > > > > > > >>> if we
> > > > > > > >>> > > > > receive
> > > > > > > >>> > > > > > > them
> > > > > > > >>> > > > > > > > >> in
> > > > > > > >>> > > > > > > > >> > edge cases like upgrades/downgrades.
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > Justine
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM Jun Rao
> > > > > > > >>> > > > > <jun@confluent.io.invalid
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > > > > >> wrote:
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> > > 101.3 I realized that I actually have
> > two
> > > > > > > questions.
> > > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need
> to
> > > > write
> > > > > > the
> > > > > > > >>> > > previous
> > > > > > > >>> > > > > > > produce
> > > > > > > >>> > > > > > > > Id
> > > > > > > >>> > > > > > > > >> > > tagged field in the end maker so that
> we
> > > > know
> > > > > if
> > > > > > > the
> > > > > > > >>> > > marker
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > > from
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > new
> > > > > > > >>> > > > > > > > >> > > client. Since the end maker is derived
> > > from
> > > > > the
> > > > > > > >>> prepare
> > > > > > > >>> > > > > marker,
> > > > > > > >>> > > > > > > > >> should we
> > > > > > > >>> > > > > > > > >> > > write the previous produce Id in the
> > > prepare
> > > > > > > marker
> > > > > > > >>> > field
> > > > > > > >>> > > > too?
> > > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > > >>> > > > > > > > >> > > we will lose this information when
> > > deriving
> > > > > the
> > > > > > > end
> > > > > > > >>> > > marker.
> > > > > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we retry
> > and
> > > > see
> > > > > > > epoch
> > > > > > > >>> - 1
> > > > > > > >>> > +
> > > > > > > >>> > > ID
> > > > > > > >>> > > > > in
> > > > > > > >>> > > > > > > last
> > > > > > > >>> > > > > > > > >> seen
> > > > > > > >>> > > > > > > > >> > > fields and are issuing the same
> command
> > > (ie
> > > > > > commit
> > > > > > > >>> not
> > > > > > > >>> > > > abort),
> > > > > > > >>> > > > > > we
> > > > > > > >>> > > > > > > > >> return
> > > > > > > >>> > > > > > > > >> > > success. The logic before KIP-890
> seems
> > to
> > > > > > return
> > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > >>> > > > > > > > >> > > in this case. Are we intentionally
> > making
> > > > this
> > > > > > > >>> change?
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> > > 112. We already merged the code that
> > adds
> > > > the
> > > > > > > >>> VerifyOnly
> > > > > > > >>> > > > field
> > > > > > > >>> > > > > > in
> > > > > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which is an
> > > inter
> > > > > > > broker
> > > > > > > >>> > > request.
> > > > > > > >>> > > > > It
> > > > > > > >>> > > > > > > > seems
> > > > > > > >>> > > > > > > > >> > that
> > > > > > > >>> > > > > > > > >> > > we didn't bump up the IBP for that. Do
> > you
> > > > > know
> > > > > > > why?
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> > > Jun
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM
> Justine
> > > > Olshan
> > > > > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > > > > >>> > > > > > > > >> > > >
> > > > > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen" to
> > > "current
> > > > > > > >>> producer id
> > > > > > > >>> > > and
> > > > > > > >>> > > > > > > epoch"
> > > > > > > >>> > > > > > > > if
> > > > > > > >>> > > > > > > > >> > that
> > > > > > > >>> > > > > > > > >> > > > was the part that was confusing
> > > > > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > > > > >>> > > > > > > > >> > > > 112 We still need it. But I am still
> > > > > > finalizing
> > > > > > > >>> the
> > > > > > > >>> > > > design.
> > > > > > > >>> > > > > I
> > > > > > > >>> > > > > > > will
> > > > > > > >>> > > > > > > > >> > update
> > > > > > > >>> > > > > > > > >> > > > the KIP once I get the information
> > > > > finalized.
> > > > > > > >>> Sorry
> > > > > > > >>> > for
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > > delays.
> > > > > > > >>> > > > > > > > >> > > >
> > > > > > > >>> > > > > > > > >> > > > Justine
> > > > > > > >>> > > > > > > > >> > > >
> > > > > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at 10:50 AM Jun
> > Rao
> > > > > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > > >>> > > > > > > > >> > > >
> > > > > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow case,
> the
> > > > > previous
> > > > > > > ID
> > > > > > > >>> is
> > > > > > > >>> > the
> > > > > > > >>> > > > > same
> > > > > > > >>> > > > > > as
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > produce
> > > > > > > >>> > > > > > > > >> > > > > ID for the complete marker too,
> but
> > we
> > > > set
> > > > > > the
> > > > > > > >>> > > previous
> > > > > > > >>> > > > ID
> > > > > > > >>> > > > > > in
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > >>> > > > > > > > >> > > > > marker. Earlier you mentioned that
> > > this
> > > > is
> > > > > > to
> > > > > > > >>> know
> > > > > > > >>> > > that
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> marker is
> > > > > > > >>> > > > > > > > >> > > > > written by the new client so that
> we
> > > > could
> > > > > > > >>> return
> > > > > > > >>> > > > success
> > > > > > > >>> > > > > on
> > > > > > > >>> > > > > > > > >> retried
> > > > > > > >>> > > > > > > > >> > > > > endMarker requests. I was trying
> to
> > > > > > understand
> > > > > > > >>> why
> > > > > > > >>> > > this
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > not
> > > > > > > >>> > > > > > > > >> needed
> > > > > > > >>> > > > > > > > >> > > for
> > > > > > > >>> > > > > > > > >> > > > > the prepare marker since retry can
> > > > happen
> > > > > in
> > > > > > > the
> > > > > > > >>> > > prepare
> > > > > > > >>> > > > > > state
> > > > > > > >>> > > > > > > > >> too.
> > > > > > > >>> > > > > > > > >> > Is
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > reason that in the prepare state,
> we
> > > > > return
> > > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > > >>> > > > > > > > >> > > > instead
> > > > > > > >>> > > > > > > > >> > > > > of success on retried endMaker
> > > requests?
> > > > > If
> > > > > > > so,
> > > > > > > >>> > should
> > > > > > > >>> > > > we
> > > > > > > >>> > > > > > > change
> > > > > > > >>> > > > > > > > >> "If
> > > > > > > >>> > > > > > > > >> > we
> > > > > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID in
> last
> > > > seen
> > > > > > > >>> fields and
> > > > > > > >>> > > are
> > > > > > > >>> > > > > > > issuing
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > same
> > > > > > > >>> > > > > > > > >> > > > > command (ie commit not abort) we
> can
> > > > > return
> > > > > > > >>> (with
> > > > > > > >>> > the
> > > > > > > >>> > > > new
> > > > > > > >>> > > > > > > > epoch)"
> > > > > > > >>> > > > > > > > >> > > > > accordingly?
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP, a
> > delayed
> > > > > > endMaker
> > > > > > > >>> > request
> > > > > > > >>> > > > > > carries
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > same
> > > > > > > >>> > > > > > > > >> > > > > epoch and won't be fenced. This
> can
> > > > > > > >>> commit/abort a
> > > > > > > >>> > > > future
> > > > > > > >>> > > > > > > > >> transaction
> > > > > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure if we
> > have
> > > > > seen
> > > > > > > >>> this in
> > > > > > > >>> > > > > practice
> > > > > > > >>> > > > > > > > >> though.
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would be
> useful
> > > to
> > > > > make
> > > > > > > it
> > > > > > > >>> > clear
> > > > > > > >>> > > > that
> > > > > > > >>> > > > > > we
> > > > > > > >>> > > > > > > > can
> > > > > > > >>> > > > > > > > >> now
> > > > > > > >>> > > > > > > > >> > > > > populate the lastSeen field from
> the
> > > log
> > > > > > > >>> reliably.
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > >>> > > > > since
> > > > > > > >>> > > > > > > > it's
> > > > > > > >>> > > > > > > > >> > > called
> > > > > > > >>> > > > > > > > >> > > > > across brokers and we are changing
> > its
> > > > > > schema.
> > > > > > > >>> Are
> > > > > > > >>> > you
> > > > > > > >>> > > > > > saying
> > > > > > > >>> > > > > > > we
> > > > > > > >>> > > > > > > > >> > don't
> > > > > > > >>> > > > > > > > >> > > > need
> > > > > > > >>> > > > > > > > >> > > > > it any more? I thought that we
> > already
> > > > > > > >>> implemented
> > > > > > > >>> > the
> > > > > > > >>> > > > > > server
> > > > > > > >>> > > > > > > > side
> > > > > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > > >>> > > > > across
> > > > > > > >>> > > > > > > > >> brokers.
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at 5:05 PM
> > > Justine
> > > > > > Olshan
> > > > > > > >>> > > > > > > > >> > > > > <jo...@confluent.io.invalid>
> > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the previous
> ID
> > > in
> > > > > the
> > > > > > > >>> Prepare
> > > > > > > >>> > > > field
> > > > > > > >>> > > > > > > since
> > > > > > > >>> > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > don't
> > > > > > > >>> > > > > > > > >> > > > > need
> > > > > > > >>> > > > > > > > >> > > > > > it. It is the same producer ID
> as
> > > the
> > > > > main
> > > > > > > >>> > producer
> > > > > > > >>> > > ID
> > > > > > > >>> > > > > > > field.
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to
> reread
> > > your
> > > > > > > message
> > > > > > > >>> > about
> > > > > > > >>> > > > > > delayed
> > > > > > > >>> > > > > > > > >> > markers.
> > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn marker
> > > after
> > > > > the
> > > > > > > >>> > > transaction
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > > > already
> > > > > > > >>> > > > > > > > >> > > > > complete?
> > > > > > > >>> > > > > > > > >> > > > > > So we will commit the next
> > > transaction
> > > > > > early
> > > > > > > >>> > without
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > fixes
> > > > > > > >>> > > > > > > > >> in
> > > > > > > >>> > > > > > > > >> > > part
> > > > > > > >>> > > > > > > > >> > > > 2?
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > 111 Yes -- this terminology was
> > used
> > > > in
> > > > > a
> > > > > > > >>> previous
> > > > > > > >>> > > KIP
> > > > > > > >>> > > > > and
> > > > > > > >>> > > > > > > > never
> > > > > > > >>> > > > > > > > >> > > > > > implemented it in the log --
> only
> > in
> > > > > > memory
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which interbroker
> > > protocol
> > > > > are
> > > > > > > you
> > > > > > > >>> > > > referring
> > > > > > > >>> > > > > > to?
> > > > > > > >>> > > > > > > I
> > > > > > > >>> > > > > > > > am
> > > > > > > >>> > > > > > > > >> > > > working
> > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > >>> > > > > > > > >> > > > > > the design for the work to
> remove
> > > the
> > > > > > extra
> > > > > > > >>> add
> > > > > > > >>> > > > > partitions
> > > > > > > >>> > > > > > > > call
> > > > > > > >>> > > > > > > > >> > and I
> > > > > > > >>> > > > > > > > >> > > > > right
> > > > > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I have
> > yet
> > > to
> > > > > > > update
> > > > > > > >>> that
> > > > > > > >>> > > > > section
> > > > > > > >>> > > > > > > as
> > > > > > > >>> > > > > > > > I
> > > > > > > >>> > > > > > > > >> > > > finalize
> > > > > > > >>> > > > > > > > >> > > > > > the design so please stay tuned.
> > Was
> > > > > there
> > > > > > > >>> > anything
> > > > > > > >>> > > > else
> > > > > > > >>> > > > > > you
> > > > > > > >>> > > > > > > > >> > thought
> > > > > > > >>> > > > > > > > >> > > > > needed
> > > > > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > Justine
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at 3:07 PM
> > Jun
> > > > Rao
> > > > > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > I don't see this create any
> > issue.
> > > > It
> > > > > > just
> > > > > > > >>> makes
> > > > > > > >>> > > it
> > > > > > > >>> > > > a
> > > > > > > >>> > > > > > bit
> > > > > > > >>> > > > > > > > >> hard to
> > > > > > > >>> > > > > > > > >> > > > > explain
> > > > > > > >>> > > > > > > > >> > > > > > > what this non-tagged produce
> id
> > > > field
> > > > > > > >>> means. We
> > > > > > > >>> > > are
> > > > > > > >>> > > > > > > > >> essentially
> > > > > > > >>> > > > > > > > >> > > > trying
> > > > > > > >>> > > > > > > > >> > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > combine two actions
> (completing
> > a
> > > > txn
> > > > > > and
> > > > > > > >>> init a
> > > > > > > >>> > > new
> > > > > > > >>> > > > > > > produce
> > > > > > > >>> > > > > > > > >> Id)
> > > > > > > >>> > > > > > > > >> > > in a
> > > > > > > >>> > > > > > > > >> > > > > > > single record. But, this may
> be
> > > fine
> > > > > > too.
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > A few other follow up
> comments.
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason that
> we
> > > > only
> > > > > > set
> > > > > > > >>> the
> > > > > > > >>> > > > previous
> > > > > > > >>> > > > > > > > >> produce id
> > > > > > > >>> > > > > > > > >> > > > > tagged
> > > > > > > >>> > > > > > > > >> > > > > > > field in the complete marker,
> > but
> > > > not
> > > > > in
> > > > > > > the
> > > > > > > >>> > > prepare
> > > > > > > >>> > > > > > > marker,
> > > > > > > >>> > > > > > > > >> is
> > > > > > > >>> > > > > > > > >> > > that
> > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > prepare state, we always
> return
> > > > > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > > > > >>> > > > > > on
> > > > > > > >>> > > > > > > > >> > retried
> > > > > > > >>> > > > > > > > >> > > > > > endMaker
> > > > > > > >>> > > > > > > > >> > > > > > > requests?
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > 110. "I believe your second
> > point
> > > is
> > > > > > > >>> mentioned
> > > > > > > >>> > in
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > KIP. I
> > > > > > > >>> > > > > > > > >> can
> > > > > > > >>> > > > > > > > >> > > add
> > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > >>> > > > > > > > >> > > > > > > text on
> > > > > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > > > > >>> > > > > > > > >> > > > > > > > The delayed message case can
> > > also
> > > > > > > violate
> > > > > > > >>> EOS
> > > > > > > >>> > if
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > > delayed
> > > > > > > >>> > > > > > > > >> > > > message
> > > > > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > > > > addPartitionsToTxn
> > > > > > > >>> > request
> > > > > > > >>> > > > > comes
> > > > > > > >>> > > > > > > in.
> > > > > > > >>> > > > > > > > >> > > > > Effectively
> > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > may see a message from a
> > previous
> > > > > > > (aborted)
> > > > > > > >>> > > > > transaction
> > > > > > > >>> > > > > > > > become
> > > > > > > >>> > > > > > > > >> > part
> > > > > > > >>> > > > > > > > >> > > > of
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > The above is the case when a
> > > delayed
> > > > > > > >>> message is
> > > > > > > >>> > > > > appended
> > > > > > > >>> > > > > > > to
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > data
> > > > > > > >>> > > > > > > > >> > > > > > > partition. What I mentioned
> is a
> > > > > > slightly
> > > > > > > >>> > > different
> > > > > > > >>> > > > > case
> > > > > > > >>> > > > > > > > when
> > > > > > > >>> > > > > > > > >> a
> > > > > > > >>> > > > > > > > >> > > > delayed
> > > > > > > >>> > > > > > > > >> > > > > > > marker is appended to the
> > > > transaction
> > > > > > log
> > > > > > > >>> > > partition.
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once we
> move
> > > past
> > > > > the
> > > > > > > >>> Prepare
> > > > > > > >>> > > and
> > > > > > > >>> > > > > > > > Complete
> > > > > > > >>> > > > > > > > >> > > states,
> > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > don’t need to worry about
> > lastSeen
> > > > > > fields
> > > > > > > >>> and
> > > > > > > >>> > > clear
> > > > > > > >>> > > > > > them,
> > > > > > > >>> > > > > > > > just
> > > > > > > >>> > > > > > > > >> > > handle
> > > > > > > >>> > > > > > > > >> > > > > > state
> > > > > > > >>> > > > > > > > >> > > > > > > transitions as normal.". Is
> the
> > > > > lastSeen
> > > > > > > >>> field
> > > > > > > >>> > the
> > > > > > > >>> > > > > same
> > > > > > > >>> > > > > > as
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field in
> > > > > > > >>> TransactionLogValue?
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > 112. Since the kip changes the
> > > > > > > inter-broker
> > > > > > > >>> > > > protocol,
> > > > > > > >>> > > > > > > should
> > > > > > > >>> > > > > > > > >> we
> > > > > > > >>> > > > > > > > >> > > bump
> > > > > > > >>> > > > > > > > >> > > > up
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this
> feature
> > > only
> > > > > for
> > > > > > > the
> > > > > > > >>> > KRaft
> > > > > > > >>> > > > > mode?
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at
> 11:13 AM
> > > > > Justine
> > > > > > > >>> Olshan
> > > > > > > >>> > > > > > > > >> > > > > > > <jolshan@confluent.io.invalid
> >
> > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > I'm glad we are getting to
> > > > > convergence
> > > > > > > on
> > > > > > > >>> the
> > > > > > > >>> > > > > design.
> > > > > > > >>> > > > > > :)
> > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > While I understand it seems
> a
> > > > little
> > > > > > > >>> "weird".
> > > > > > > >>> > > I'm
> > > > > > > >>> > > > > not
> > > > > > > >>> > > > > > > sure
> > > > > > > >>> > > > > > > > >> what
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > > benefit
> > > > > > > >>> > > > > > > > >> > > > > > > > of writing an extra record
> to
> > > the
> > > > > log.
> > > > > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool to
> > > describe
> > > > > > > >>> transactions
> > > > > > > >>> > > > won't
> > > > > > > >>> > > > > > > work
> > > > > > > >>> > > > > > > > >> (ie,
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > > complete
> > > > > > > >>> > > > > > > > >> > > > > > > > state is needed to calculate
> > the
> > > > > time
> > > > > > > >>> since
> > > > > > > >>> > the
> > > > > > > >>> > > > > > > > transaction
> > > > > > > >>> > > > > > > > >> > > > > completed?)
> > > > > > > >>> > > > > > > > >> > > > > > > > If we have a reason like
> this,
> > > it
> > > > is
> > > > > > > >>> enough to
> > > > > > > >>> > > > > > convince
> > > > > > > >>> > > > > > > me
> > > > > > > >>> > > > > > > > >> we
> > > > > > > >>> > > > > > > > >> > > need
> > > > > > > >>> > > > > > > > >> > > > > such
> > > > > > > >>> > > > > > > > >> > > > > > > an
> > > > > > > >>> > > > > > > > >> > > > > > > > extra record. It seems like
> it
> > > > would
> > > > > > be
> > > > > > > >>> > > replacing
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > record
> > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is this
> > correct?
> > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024 at
> > 5:14 PM
> > > > Jun
> > > > > > Rao
> > > > > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > > > > >>> > > > > > > > >> > > >
> > > > > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks for the
> explanation.
> > I
> > > > > > > >>> understand the
> > > > > > > >>> > > > > > intention
> > > > > > > >>> > > > > > > > >> now.
> > > > > > > >>> > > > > > > > >> > In
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > > > > >>> > > > > > > > >> > > > > > > > > case, we set the
> non-tagged
> > > > field
> > > > > to
> > > > > > > >>> the old
> > > > > > > >>> > > pid
> > > > > > > >>> > > > > > (and
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> max
> > > > > > > >>> > > > > > > > >> > > > > epoch)
> > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > prepare marker so that we
> > > could
> > > > > > > >>> correctly
> > > > > > > >>> > > write
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> marker to
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > data
> > > > > > > >>> > > > > > > > >> > > > > > > > > partition if the broker
> > > > > downgrades.
> > > > > > > When
> > > > > > > >>> > > writing
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> complete
> > > > > > > >>> > > > > > > > >> > > > > marker,
> > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > know the marker has
> already
> > > been
> > > > > > > >>> written to
> > > > > > > >>> > > the
> > > > > > > >>> > > > > data
> > > > > > > >>> > > > > > > > >> > partition.
> > > > > > > >>> > > > > > > > >> > > > We
> > > > > > > >>> > > > > > > > >> > > > > > set
> > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to the
> new
> > > pid
> > > > to
> > > > > > > avoid
> > > > > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > client if the broker
> > > downgrades.
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > The above seems to work.
> > It's
> > > > > just a
> > > > > > > bit
> > > > > > > >>> > > > > > inconsistent
> > > > > > > >>> > > > > > > > for
> > > > > > > >>> > > > > > > > >> a
> > > > > > > >>> > > > > > > > >> > > > prepare
> > > > > > > >>> > > > > > > > >> > > > > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > > > and a complete marker to
> use
> > > > > > different
> > > > > > > >>> pids
> > > > > > > >>> > in
> > > > > > > >>> > > > > this
> > > > > > > >>> > > > > > > > >> special
> > > > > > > >>> > > > > > > > >> > > case.
> > > > > > > >>> > > > > > > > >> > > > > If
> > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > downgrade with the
> complete
> > > > > marker,
> > > > > > it
> > > > > > > >>> seems
> > > > > > > >>> > > > that
> > > > > > > >>> > > > > we
> > > > > > > >>> > > > > > > > will
> > > > > > > >>> > > > > > > > >> > never
> > > > > > > >>> > > > > > > > >> > > > be
> > > > > > > >>> > > > > > > > >> > > > > > able
> > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > write the complete marker
> > with
> > > > the
> > > > > > old
> > > > > > > >>> pid.
> > > > > > > >>> > > Not
> > > > > > > >>> > > > > sure
> > > > > > > >>> > > > > > > if
> > > > > > > >>> > > > > > > > it
> > > > > > > >>> > > > > > > > >> > > causes
> > > > > > > >>> > > > > > > > >> > > > > any
> > > > > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems a bit
> > > weird.
> > > > > > > >>> Instead of
> > > > > > > >>> > > > > writing
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > complete
> > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > > > with the new pid, could we
> > > write
> > > > > two
> > > > > > > >>> > records:
> > > > > > > >>> > > a
> > > > > > > >>> > > > > > > complete
> > > > > > > >>> > > > > > > > >> > marker
> > > > > > > >>> > > > > > > > >> > > > > with
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > old pid followed by a
> > > > > > > >>> TransactionLogValue
> > > > > > > >>> > with
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > new
> > > > > > > >>> > > > > > > > pid
> > > > > > > >>> > > > > > > > >> > and
> > > > > > > >>> > > > > > > > >> > > an
> > > > > > > >>> > > > > > > > >> > > > > > empty
> > > > > > > >>> > > > > > > > >> > > > > > > > > state? We could make the
> two
> > > > > records
> > > > > > > in
> > > > > > > >>> the
> > > > > > > >>> > > same
> > > > > > > >>> > > > > > batch
> > > > > > > >>> > > > > > > > so
> > > > > > > >>> > > > > > > > >> > that
> > > > > > > >>> > > > > > > > >> > > > they
> > > > > > > >>> > > > > > > > >> > > > > > > will
> > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > >>> > > > > > > > >> > > > > > > > > added to the log
> atomically.
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024 at
> > > 5:40 PM
> > > > > > > Justine
> > > > > > > >>> > Olshan
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > <jolshan@confluent.io.invalid
> > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare marker
> is
> > > > > written,
> > > > > > > >>> but the
> > > > > > > >>> > > > > endTxn
> > > > > > > >>> > > > > > > > >> response
> > > > > > > >>> > > > > > > > >> > is
> > > > > > > >>> > > > > > > > >> > > > not
> > > > > > > >>> > > > > > > > >> > > > > > > > > received
> > > > > > > >>> > > > > > > > >> > > > > > > > > > by the client when the
> > > server
> > > > > > > >>> downgrades
> > > > > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare marker
> is
> > > > > > written,
> > > > > > > >>> the
> > > > > > > >>> > > endTxn
> > > > > > > >>> > > > > > > > response
> > > > > > > >>> > > > > > > > >> is
> > > > > > > >>> > > > > > > > >> > > > > received
> > > > > > > >>> > > > > > > > >> > > > > > > by
> > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > client when the server
> > > > > downgrades.
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > I think I am still a
> > little
> > > > > > > confused.
> > > > > > > >>> In
> > > > > > > >>> > > both
> > > > > > > >>> > > > of
> > > > > > > >>> > > > > > > these
> > > > > > > >>> > > > > > > > >> > cases,
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction log has the
> > old
> > > > > > producer
> > > > > > > >>> ID.
> > > > > > > >>> > We
> > > > > > > >>> > > > > don't
> > > > > > > >>> > > > > > > > write
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > in the prepare marker's
> > non
> > > > > tagged
> > > > > > > >>> fields.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > If the server downgrades
> > > now,
> > > > it
> > > > > > > would
> > > > > > > >>> > read
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > > records
> > > > > > > >>> > > > > > > > >> not
> > > > > > > >>> > > > > > > > >> > > in
> > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > >>> > > > > > > > >> > > > > > > > > > fields and the complete
> > > marker
> > > > > > will
> > > > > > > >>> also
> > > > > > > >>> > > have
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > old
> > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > >>> > > > > > > > >> > > > > ID.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > (If we had used the new
> > > > producer
> > > > > > ID,
> > > > > > > >>> we
> > > > > > > >>> > > would
> > > > > > > >>> > > > > not
> > > > > > > >>> > > > > > > have
> > > > > > > >>> > > > > > > > >> > > > > > transactional
> > > > > > > >>> > > > > > > > >> > > > > > > > > > correctness since the
> > > producer
> > > > > id
> > > > > > > >>> doesn't
> > > > > > > >>> > > > match
> > > > > > > >>> > > > > > the
> > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > state would not be
> correct
> > > on
> > > > > the
> > > > > > > data
> > > > > > > >>> > > > > partition.)
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > In the overflow case,
> I'd
> > > > expect
> > > > > > the
> > > > > > > >>> > > following
> > > > > > > >>> > > > > to
> > > > > > > >>> > > > > > > > >> happen on
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > > client
> > > > > > > >>> > > > > > > > >> > > > > > > > > side
> > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we retry
> EndTxn
> > > --
> > > > it
> > > > > > is
> > > > > > > >>> the
> > > > > > > >>> > same
> > > > > > > >>> > > > > > > producer
> > > > > > > >>> > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > and
> > > > > > > >>> > > > > > > > >> > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > -
> > > > > > > >>> > > > > > > > >> > > > > > > 1
> > > > > > > >>> > > > > > > > >> > > > > > > > > this
> > > > > > > >>> > > > > > > > >> > > > > > > > > > would fence the producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't retry
> > > > EndTxn
> > > > > > and
> > > > > > > >>> use
> > > > > > > >>> > the
> > > > > > > >>> > > > new
> > > > > > > >>> > > > > > > > >> producer id
> > > > > > > >>> > > > > > > > >> > > > which
> > > > > > > >>> > > > > > > > >> > > > > > > would
> > > > > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > > > > InvalidPidMappingException
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have
> special
> > > > > handling
> > > > > > > for
> > > > > > > >>> > when
> > > > > > > >>> > > a
> > > > > > > >>> > > > > > server
> > > > > > > >>> > > > > > > > >> > > > downgrades.
> > > > > > > >>> > > > > > > > >> > > > > > When
> > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > >>> > > > > > > > >> > > > > > > > > > reconnects we could get
> an
> > > API
> > > > > > > version
> > > > > > > >>> > > request
> > > > > > > >>> > > > > > > showing
> > > > > > > >>> > > > > > > > >> > > KIP-890
> > > > > > > >>> > > > > > > > >> > > > > > part 2
> > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > not supported. In that
> > case,
> > > > we
> > > > > > can
> > > > > > > >>> call
> > > > > > > >>> > > > > > > > initProducerId
> > > > > > > >>> > > > > > > > >> to
> > > > > > > >>> > > > > > > > >> > > > abort
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In the
> > > overflow
> > > > > > case,
> > > > > > > >>> this
> > > > > > > >>> > > > > correctly
> > > > > > > >>> > > > > > > > gives
> > > > > > > >>> > > > > > > > >> > us a
> > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > I guess the
> corresponding
> > > case
> > > > > > would
> > > > > > > >>> be
> > > > > > > >>> > > where
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> *complete
> > > > > > > >>> > > > > > > > >> > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > *is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > written but the endTxn
> is
> > > not
> > > > > > > >>> received by
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > client
> > > > > > > >>> > > > > > > > and
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > > server
> > > > > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This would
> > > result
> > > > in
> > > > > > the
> > > > > > > >>> > > > transaction
> > > > > > > >>> > > > > > > > >> > coordinator
> > > > > > > >>> > > > > > > > >> > > > > having
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old one.
> > If
> > > > the
> > > > > > > client
> > > > > > > >>> > > > retries,
> > > > > > > >>> > > > > it
> > > > > > > >>> > > > > > > > will
> > > > > > > >>> > > > > > > > >> > > receive
> > > > > > > >>> > > > > > > > >> > > > > an
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > InvalidPidMappingException.
> > > > The
> > > > > > > >>> > > InitProducerId
> > > > > > > >>> > > > > > > > scenario
> > > > > > > >>> > > > > > > > >> > above
> > > > > > > >>> > > > > > > > >> > > > > would
> > > > > > > >>> > > > > > > > >> > > > > > > > help
> > > > > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my
> > > compatibility
> > > > > > story
> > > > > > > is
> > > > > > > >>> > meant
> > > > > > > >>> > > > to
> > > > > > > >>> > > > > > > > support
> > > > > > > >>> > > > > > > > >> > > > > downgrades
> > > > > > > >>> > > > > > > > >> > > > > > > > server
> > > > > > > >>> > > > > > > > >> > > > > > > > > > side in keeping the
> > > > > transactional
> > > > > > > >>> > > correctness.
> > > > > > > >>> > > > > > > Keeping
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > > client
> > > > > > > >>> > > > > > > > >> > > > > > > from
> > > > > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is not
> the
> > > > > > priority.
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I can
> > also
> > > > add
> > > > > > text
> > > > > > > >>> in
> > > > > > > >>> > the
> > > > > > > >>> > > > KIP
> > > > > > > >>> > > > > > > about
> > > > > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > > > > >>> > > > > > > > >> > > > > > > if
> > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > think that fixes some
> edge
> > > > > cases.
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12, 2024 at
> > > > 4:10 PM
> > > > > > Jun
> > > > > > > >>> Rao
> > > > > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the reply.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we don't
> > need
> > > > to
> > > > > > > >>> optimize
> > > > > > > >>> > for
> > > > > > > >>> > > > > > fencing
> > > > > > > >>> > > > > > > > >> during
> > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > Regarding consistency,
> > > there
> > > > > are
> > > > > > > two
> > > > > > > >>> > > > possible
> > > > > > > >>> > > > > > > cases:
> > > > > > > >>> > > > > > > > >> (1)
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > > prepare
> > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > is written, but the
> > endTxn
> > > > > > > response
> > > > > > > >>> is
> > > > > > > >>> > not
> > > > > > > >>> > > > > > > received
> > > > > > > >>> > > > > > > > by
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > > client
> > > > > > > >>> > > > > > > > >> > > > > > > > when
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > server downgrades; (2)
> > > the
> > > > > > > prepare
> > > > > > > >>> > marker
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > > > written,
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > > endTxn
> > > > > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > is received by the
> > client
> > > > when
> > > > > > the
> > > > > > > >>> > server
> > > > > > > >>> > > > > > > > downgrades.
> > > > > > > >>> > > > > > > > >> In
> > > > > > > >>> > > > > > > > >> > > (1),
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > client
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > will have the old
> > produce
> > > Id
> > > > > and
> > > > > > > in
> > > > > > > >>> (2),
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > client
> > > > > > > >>> > > > > > > > >> will
> > > > > > > >>> > > > > > > > >> > > have
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If we
> > > downgrade
> > > > > > right
> > > > > > > >>> after
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > prepare
> > > > > > > >>> > > > > > > > >> > marker,
> > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > can't
> > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > consistent to both (1)
> > and
> > > > (2)
> > > > > > > >>> since we
> > > > > > > >>> > > can
> > > > > > > >>> > > > > only
> > > > > > > >>> > > > > > > put
> > > > > > > >>> > > > > > > > >> one
> > > > > > > >>> > > > > > > > >> > > > value
> > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > existing produce Id
> > field.
> > > > > It's
> > > > > > > >>> also not
> > > > > > > >>> > > > clear
> > > > > > > >>> > > > > > > which
> > > > > > > >>> > > > > > > > >> case
> > > > > > > >>> > > > > > > > >> > > is
> > > > > > > >>> > > > > > > > >> > > > > more
> > > > > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > So we could probably
> be
> > > > > > consistent
> > > > > > > >>> with
> > > > > > > >>> > > > either
> > > > > > > >>> > > > > > > case.
> > > > > > > >>> > > > > > > > >> By
> > > > > > > >>> > > > > > > > >> > > > putting
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in the
> > prepare
> > > > > > marker,
> > > > > > > >>> we
> > > > > > > >>> > are
> > > > > > > >>> > > > > > > consistent
> > > > > > > >>> > > > > > > > >> with
> > > > > > > >>> > > > > > > > >> > > > case
> > > > > > > >>> > > > > > > > >> > > > > > (2)
> > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > it
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > also has the slight
> > > benefit
> > > > > that
> > > > > > > the
> > > > > > > >>> > > produce
> > > > > > > >>> > > > > > field
> > > > > > > >>> > > > > > > > in
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > > > prepare
> > > > > > > >>> > > > > > > > >> > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > complete marker are
> > > > consistent
> > > > > > in
> > > > > > > >>> the
> > > > > > > >>> > > > overflow
> > > > > > > >>> > > > > > > case.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12, 2024
> at
> > > > > 3:11 PM
> > > > > > > >>> Justine
> > > > > > > >>> > > > Olshan
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > <jolshan@confluent.io.invalid
> > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the case you
> > > describe,
> > > > we
> > > > > > > would
> > > > > > > >>> > need
> > > > > > > >>> > > to
> > > > > > > >>> > > > > > have
> > > > > > > >>> > > > > > > a
> > > > > > > >>> > > > > > > > >> > delayed
> > > > > > > >>> > > > > > > > >> > > > > > > request,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > successful EndTxn,
> > and a
> > > > > > > >>> successful
> > > > > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > > > > >>> > > > > > > > >> > > and
> > > > > > > >>> > > > > > > > >> > > > > then
> > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn
> request
> > > go
> > > > > > > through
> > > > > > > >>> for
> > > > > > > >>> > a
> > > > > > > >>> > > > > given
> > > > > > > >>> > > > > > > > >> producer.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to figure
> > out
> > > > if
> > > > > it
> > > > > > > is
> > > > > > > >>> > > possible
> > > > > > > >>> > > > > for
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> > client
> > > > > > > >>> > > > > > > > >> > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > a previous request
> is
> > > > > delayed
> > > > > > > >>> > somewhere.
> > > > > > > >>> > > > But
> > > > > > > >>> > > > > > > yes,
> > > > > > > >>> > > > > > > > in
> > > > > > > >>> > > > > > > > >> > this
> > > > > > > >>> > > > > > > > >> > > > > case
> > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > >>> > > > > > > > >> > > > > > > > > think
> > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > would fence the
> > client.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the overflow
> > > case.
> > > > > In
> > > > > > > the
> > > > > > > >>> > > overflow
> > > > > > > >>> > > > > > case,
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are different
> on
> > > the
> > > > > > > marker
> > > > > > > >>> and
> > > > > > > >>> > on
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > new
> > > > > > > >>> > > > > > > > >> > > > transaction.
> > > > > > > >>> > > > > > > > >> > > > > > So
> > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to use
> the
> > > max
> > > > > > epoch
> > > > > > > >>> but
> > > > > > > >>> > the
> > > > > > > >>> > > > new
> > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > >>> > > > > > > > >> > > > > > > > start
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and
> epoch 0
> > > in
> > > > > the
> > > > > > > >>> > > > transactional
> > > > > > > >>> > > > > > > state.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the server
> > downgrade
> > > > > case,
> > > > > > we
> > > > > > > >>> want
> > > > > > > >>> > to
> > > > > > > >>> > > > see
> > > > > > > >>> > > > > > the
> > > > > > > >>> > > > > > > > >> > producer
> > > > > > > >>> > > > > > > > >> > > > ID
> > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > >>> > > > > > > > >> > > > > > > > that
> > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > what the client will
> > > have.
> > > > > If
> > > > > > we
> > > > > > > >>> > > complete
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> commit,
> > > > > > > >>> > > > > > > > >> > and
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > state is reloaded,
> we
> > > need
> > > > > the
> > > > > > > new
> > > > > > > >>> > > > producer
> > > > > > > >>> > > > > ID
> > > > > > > >>> > > > > > > in
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > state
> > > > > > > >>> > > > > > > > >> > > > > so
> > > > > > > >>> > > > > > > > >> > > > > > > > there
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid producer
> ID
> > > > > > mapping.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > The server downgrade
> > > cases
> > > > > are
> > > > > > > >>> > > considering
> > > > > > > >>> > > > > > > > >> > transactional
> > > > > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing from
> > > > previous
> > > > > > > >>> behavior
> > > > > > > >>> > --
> > > > > > > >>> > > > and
> > > > > > > >>> > > > > > are
> > > > > > > >>> > > > > > > > not
> > > > > > > >>> > > > > > > > >> > > > > concerned
> > > > > > > >>> > > > > > > > >> > > > > > > > about
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the
> safety
> > > from
> > > > > > > fencing
> > > > > > > >>> > > retries
> > > > > > > >>> > > > > (as
> > > > > > > >>> > > > > > > we
> > > > > > > >>> > > > > > > > >> have
> > > > > > > >>> > > > > > > > >> > > > > > downgraded
> > > > > > > >>> > > > > > > > >> > > > > > > > so
> > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to
> > support).
> > > > > > Perhaps
> > > > > > > >>> this
> > > > > > > >>> > is
> > > > > > > >>> > > a
> > > > > > > >>> > > > > > trade
> > > > > > > >>> > > > > > > > off,
> > > > > > > >>> > > > > > > > >> > but
> > > > > > > >>> > > > > > > > >> > > I
> > > > > > > >>> > > > > > > > >> > > > > > think
> > > > > > > >>> > > > > > > > >> > > > > > > it
> > > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client
> > > downgrades,
> > > > > it
> > > > > > > will
> > > > > > > >>> > have
> > > > > > > >>> > > > > > > restarted
> > > > > > > >>> > > > > > > > >> and
> > > > > > > >>> > > > > > > > >> > it
> > > > > > > >>> > > > > > > > >> > > is
> > > > > > > >>> > > > > > > > >> > > > > ok
> > > > > > > >>> > > > > > > > >> > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > have a new producer
> ID
> > > > too).
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan 12, 2024
> > at
> > > > > > 11:42 AM
> > > > > > > >>> Jun
> > > > > > > >>> > Rao
> > > > > > > >>> > > > > > > > >> > > > > > > <jun@confluent.io.invalid
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for the
> > reply.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If the
> marker
> > > is
> > > > > > > written
> > > > > > > >>> by
> > > > > > > >>> > the
> > > > > > > >>> > > > new
> > > > > > > >>> > > > > > > > >> client, we
> > > > > > > >>> > > > > > > > >> > > can
> > > > > > > >>> > > > > > > > >> > > > > as
> > > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the last email
> > > guarantee
> > > > > > that
> > > > > > > >>> any
> > > > > > > >>> > > EndTxn
> > > > > > > >>> > > > > > > > requests
> > > > > > > >>> > > > > > > > >> > with
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from the same
> > producer
> > > > and
> > > > > > the
> > > > > > > >>> same
> > > > > > > >>> > > > > > > transaction.
> > > > > > > >>> > > > > > > > >> Then
> > > > > > > >>> > > > > > > > >> > > we
> > > > > > > >>> > > > > > > > >> > > > > > don't
> > > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > return a fenced
> > error
> > > > but
> > > > > > can
> > > > > > > >>> handle
> > > > > > > >>> > > > > > > gracefully
> > > > > > > >>> > > > > > > > as
> > > > > > > >>> > > > > > > > >> > > > > described
> > > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When a delayed
> > EndTnx
> > > > > > request
> > > > > > > is
> > > > > > > >>> > > > > processed,
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> txn
> > > > > > > >>> > > > > > > > >> > > state
> > > > > > > >>> > > > > > > > >> > > > > > could
> > > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next txn.
> I
> > > > guess
> > > > > in
> > > > > > > >>> this
> > > > > > > >>> > case
> > > > > > > >>> > > > we
> > > > > > > >>> > > > > > > still
> > > > > > > >>> > > > > > > > >> > return
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > fenced
> > > > > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed
> request?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry, my
> > > question
> > > > > was
> > > > > > > >>> > > inaccurate.
> > > > > > > >>> > > > > What
> > > > > > > >>> > > > > > > you
> > > > > > > >>> > > > > > > > >> > > > described
> > > > > > > >>> > > > > > > > >> > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > "The downgrade
> > > > > > compatibility I
> > > > > > > >>> > mention
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > that
> > > > > > > >>> > > > > > > > we
> > > > > > > >>> > > > > > > > >> > keep
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in the
> > main
> > > > > > > >>> (non-tagged)
> > > > > > > >>> > > > fields
> > > > > > > >>> > > > > as
> > > > > > > >>> > > > > > > we
> > > > > > > >>> > > > > > > > >> did
> > > > > > > >>> > > > > > > > >> > > > before
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > code
> > > > > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > server side." If
> we
> > > want
> > > > > to
> > > > > > do
> > > > > > > >>> this,
> > > > > > > >>> > > it
> > > > > > > >>> > > > > > seems
> > > > > > > >>> > > > > > > > >> that we
> > > > > > > >>> > > > > > > > >> > > > > should
> > > > > > > >>> > > > > > > > >> > > > > > > use
> > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > current produce Id
> > and
> > > > max
> > > > > > > >>> epoch in
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > existing
> > > > > > > >>> > > > > > > > >> > > > producerId
> > > > > > > >>> > > > > > > > >> > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerEpoch
> fields
> > > for
> > > > > > both
> > > > > > > >>> the
> > > > > > > >>> > > > prepare
> > > > > > > >>> > > > > > and
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > complete
> > > > > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > The downgrade can
> > > happen
> > > > > > after
> > > > > > > >>> the
> > > > > > > >>> > > > > complete
> > > > > > > >>> > > > > > > > >> marker is
> > > > > > > >>> > > > > > > > >> > > > > > written.
> > > > > > > >>> > > > > > > > >> > > > > > > > With
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > you described, the
> > > > > > downgraded
> > > > > > > >>> > > > coordinator
> > > > > > > >>> > > > > > will
> > > > > > > >>> > > > > > > > see
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of the old
> > > one.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan 12,
> 2024
> > > at
> > > > > > > 10:44 AM
> > > > > > > >>> > > Justine
> > > > > > > >>> > > > > > > Olshan
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > <jolshan@confluent.io.invalid
> > > > > > > >
> > > > > > > >>> > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can update the
> > > > > > > description.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe your
> > > second
> > > > > > point
> > > > > > > is
> > > > > > > >>> > > > mentioned
> > > > > > > >>> > > > > > in
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > KIP.
> > > > > > > >>> > > > > > > > >> > > I
> > > > > > > >>> > > > > > > > >> > > > > can
> > > > > > > >>> > > > > > > > >> > > > > > > add
> > > > > > > >>> > > > > > > > >> > > > > > > > > more
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it is
> > > helpful.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The delayed
> > > message
> > > > > case
> > > > > > > can
> > > > > > > >>> > also
> > > > > > > >>> > > > > > violate
> > > > > > > >>> > > > > > > > EOS
> > > > > > > >>> > > > > > > > >> if
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > > delayed
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in after
> the
> > > > next
> > > > > > > >>> > > > > addPartitionsToTxn
> > > > > > > >>> > > > > > > > >> request
> > > > > > > >>> > > > > > > > >> > > > comes
> > > > > > > >>> > > > > > > > >> > > > > > in.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a
> message
> > > > from a
> > > > > > > >>> previous
> > > > > > > >>> > > > > > (aborted)
> > > > > > > >>> > > > > > > > >> > > transaction
> > > > > > > >>> > > > > > > > >> > > > > > > become
> > > > > > > >>> > > > > > > > >> > > > > > > > > part
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > next
> transaction.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the marker is
> > > > written
> > > > > > by
> > > > > > > >>> the
> > > > > > > >>> > new
> > > > > > > >>> > > > > > client,
> > > > > > > >>> > > > > > > we
> > > > > > > >>> > > > > > > > >> can
> > > > > > > >>> > > > > > > > >> > > as I
> > > > > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email
> > guarantee
> > > > > that
> > > > > > > any
> > > > > > > >>> > EndTxn
> > > > > > > >>> > > > > > > requests
> > > > > > > >>> > > > > > > > >> with
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > same
> > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same
> producer
> > > and
> > > > > the
> > > > > > > same
> > > > > > > >>> > > > > > transaction.
> > > > > > > >>> > > > > > > > >> Then we
> > > > > > > >>> > > > > > > > >> > > > don't
> > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced error but
> > can
> > > > > > handle
> > > > > > > >>> > > gracefully
> > > > > > > >>> > > > > as
> > > > > > > >>> > > > > > > > >> described
> > > > > > > >>> > > > > > > > >> > > in
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't think a
> > > > boolean
> > > > > is
> > > > > > > >>> useful
> > > > > > > >>> > > > since
> > > > > > > >>> > > > > it
> > > > > > > >>> > > > > > > is
> > > > > > > >>> > > > > > > > >> > > directly
> > > > > > > >>> > > > > > > > >> > > > > > > encoded
> > > > > > > >>> > > > > > > > >> > > > > > > > by
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence or
> lack
> > of
> > > > the
> > > > > > > >>> tagged
> > > > > > > >>> > > field
> > > > > > > >>> > > > > > being
> > > > > > > >>> > > > > > > > >> > written.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the prepare
> > > marker
> > > > we
> > > > > > > will
> > > > > > > >>> have
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > same
> > > > > > > >>> > > > > > > > >> > producer
> > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In the
> > > Complete
> > > > > > state
> > > > > > > >>> we
> > > > > > > >>> > may
> > > > > > > >>> > > > not.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure why
> > the
> > > > > > ongoing
> > > > > > > >>> state
> > > > > > > >>> > > > > matters
> > > > > > > >>> > > > > > > for
> > > > > > > >>> > > > > > > > >> this
> > > > > > > >>> > > > > > > > >> > > > KIP.
> > > > > > > >>> > > > > > > > >> > > > > It
> > > > > > > >>> > > > > > > > >> > > > > > > > does
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure
> what
> > > you
> > > > > are
> > > > > > > >>> > referring
> > > > > > > >>> > > to
> > > > > > > >>> > > > > > about
> > > > > > > >>> > > > > > > > >> > writing
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > previous
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the
> prepare
> > > > > marker.
> > > > > > > >>> This is
> > > > > > > >>> > > not
> > > > > > > >>> > > > in
> > > > > > > >>> > > > > > the
> > > > > > > >>> > > > > > > > >> KIP.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the overflow
> > > case,
> > > > we
> > > > > > > >>> write the
> > > > > > > >>> > > > > > > > >> nextProducerId
> > > > > > > >>> > > > > > > > >> > in
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so we
> know
> > > > what
> > > > > we
> > > > > > > >>> > assigned
> > > > > > > >>> > > > when
> > > > > > > >>> > > > > > we
> > > > > > > >>> > > > > > > > >> reload
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we
> complete,
> > we
> > > > > > > >>> transition
> > > > > > > >>> > this
> > > > > > > >>> > > > ID
> > > > > > > >>> > > > > to
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> main
> > > > > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the
> previous
> > > > > producer
> > > > > > > ID
> > > > > > > >>> > field
> > > > > > > >>> > > > > filled
> > > > > > > >>> > > > > > > in.
> > > > > > > >>> > > > > > > > >> This
> > > > > > > >>> > > > > > > > >> > > is
> > > > > > > >>> > > > > > > > >> > > > so
> > > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > can
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry case
> > the
> > > > > > > operation
> > > > > > > >>> > > > completed
> > > > > > > >>> > > > > > > > >> > successfully
> > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > don't
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer. The
> > > > downgrade
> > > > > > > >>> > > compatibility
> > > > > > > >>> > > > I
> > > > > > > >>> > > > > > > > mention
> > > > > > > >>> > > > > > > > >> is
> > > > > > > >>> > > > > > > > >> > > that
> > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > keep
> > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer ID and
> > > epoch
> > > > in
> > > > > > the
> > > > > > > >>> main
> > > > > > > >>> > > > > > > (non-tagged)
> > > > > > > >>> > > > > > > > >> > fields
> > > > > > > >>> > > > > > > > >> > > > as
> > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > did
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on the
> server
> > > > side.
> > > > > > If
> > > > > > > >>> the
> > > > > > > >>> > > server
> > > > > > > >>> > > > > > > > >> downgrades,
> > > > > > > >>> > > > > > > > >> > we
> > > > > > > >>> > > > > > > > >> > > > are
> > > > > > > >>> > > > > > > > >> > > > > > > still
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This addresses
> > both
> > > > the
> > > > > > > >>> prepare
> > > > > > > >>> > and
> > > > > > > >>> > > > > > complete
> > > > > > > >>> > > > > > > > >> state
> > > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri, Jan 12,
> > 2024
> > > > at
> > > > > > > >>> 10:21 AM
> > > > > > > >>> > Jun
> > > > > > > >>> > > > Rao
> > > > > > > >>> > > > > > > > >> > > > > > > > > <jun@confluent.io.invalid
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi, Justine,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks for the
> > > > reply.
> > > > > > > Sorry
> > > > > > > >>> for
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > delay. I
> > > > > > > >>> > > > > > > > >> > have a
> > > > > > > >>> > > > > > > > >> > > > few
> > > > > > > >>> > > > > > > > >> > > > > > > more
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I think
> the
> > > > > > > motivation
> > > > > > > >>> > > section
> > > > > > > >>> > > > > > could
> > > > > > > >>> > > > > > > be
> > > > > > > >>> > > > > > > > >> > > > improved.
> > > > > > > >>> > > > > > > > >> > > > > > One
> > > > > > > >>> > > > > > > > >> > > > > > > of
> > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > motivations
> > listed
> > > > by
> > > > > > the
> > > > > > > >>> KIP is
> > > > > > > >>> > > > "This
> > > > > > > >>> > > > > > can
> > > > > > > >>> > > > > > > > >> happen
> > > > > > > >>> > > > > > > > >> > > > when
> > > > > > > >>> > > > > > > > >> > > > > a
> > > > > > > >>> > > > > > > > >> > > > > > > > > message
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or delayed due
> > to
> > > > > > > networking
> > > > > > > >>> > > issues
> > > > > > > >>> > > > > or a
> > > > > > > >>> > > > > > > > >> network
> > > > > > > >>> > > > > > > > >> > > > > > partition,
> > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > transaction
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts, and
> then
> > > the
> > > > > > > delayed
> > > > > > > >>> > > message
> > > > > > > >>> > > > > > > finally
> > > > > > > >>> > > > > > > > >> > comes
> > > > > > > >>> > > > > > > > >> > > > > in.".
> > > > > > > >>> > > > > > > > >> > > > > > > This
> > > > > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very accurate.
> > > > Without
> > > > > > > >>> KIP-890,
> > > > > > > >>> > > > > > currently,
> > > > > > > >>> > > > > > > > if
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts an
> > ongoing
> > > > > > > >>> transaction,
> > > > > > > >>> > it
> > > > > > > >>> > > > > > already
> > > > > > > >>> > > > > > > > >> bumps
> > > > > > > >>> > > > > > > > >> > up
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which prevents
> > the
> > > > > > delayed
> > > > > > > >>> > produce
> > > > > > > >>> > > > > > message
> > > > > > > >>> > > > > > > > >> from
> > > > > > > >>> > > > > > > > >> > > being
> > > > > > > >>> > > > > > > > >> > > > > > added
> > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > partition.
> What
> > > can
> > > > > > cause
> > > > > > > a
> > > > > > > >>> > > hanging
> > > > > > > >>> > > > > > > > >> transaction
> > > > > > > >>> > > > > > > > >> > is
> > > > > > > >>> > > > > > > > >> > > > that
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completes
> > (either
> > > > > aborts
> > > > > > > or
> > > > > > > >>> > > > commits) a
> > > > > > > >>> > > > > > > > >> > transaction
> > > > > > > >>> > > > > > > > >> > > > > before
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > successful ack
> > on
> > > > > > messages
> > > > > > > >>> > > published
> > > > > > > >>> > > > > in
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> same
> > > > > > > >>> > > > > > > > >> > > txn.
> > > > > > > >>> > > > > > > > >> > > > > In
> > > > > > > >>> > > > > > > > >> > > > > > > this
> > > > > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible for
> the
> > > > > delayed
> > > > > > > >>> message
> > > > > > > >>> > > to
> > > > > > > >>> > > > be
> > > > > > > >>> > > > > > > > >> appended
> > > > > > > >>> > > > > > > > >> > to
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker,
> causing
> > a
> > > > > > > >>> transaction to
> > > > > > > >>> > > > hang.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A similar
> issue
> > > (not
> > > > > > > >>> mentioned
> > > > > > > >>> > in
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > > >> motivation)
> > > > > > > >>> > > > > > > > >> > > > could
> > > > > > > >>> > > > > > > > >> > > > > > > > happen
> > > > > > > >>> > > > > > > > >> > > > > > > > > on
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker in the
> > > > > > > coordinator's
> > > > > > > >>> log.
> > > > > > > >>> > > For
> > > > > > > >>> > > > > > > > example,
> > > > > > > >>> > > > > > > > >> > it's
> > > > > > > >>> > > > > > > > >> > > > > > possible
> > > > > > > >>> > > > > > > > >> > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest
> to
> > > be
> > > > > > > delayed
> > > > > > > >>> on
> > > > > > > >>> > the
> > > > > > > >>> > > > > > > > >> coordinator.
> > > > > > > >>> > > > > > > > >> > By
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > time
> > > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest
> is
> > > > > > > processed,
> > > > > > > >>> it's
> > > > > > > >>> > > > > > possible
> > > > > > > >>> > > > > > > > that
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > >>> > > > > > > > >> > > > > > > > txn
> > > > > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completed and
> a
> > > new
> > > > > txn
> > > > > > > has
> > > > > > > >>> > > started.
> > > > > > > >>> > > > > > > > >> Currently,
> > > > > > > >>> > > > > > > > >> > > since
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped on
> every
> > > txn,
> > > > > the
> > > > > > > >>> delayed
> > > > > > > >>> > > > > > > > EndTxnRequest
> > > > > > > >>> > > > > > > > >> > will
> > > > > > > >>> > > > > > > > >> > > > add
> > > > > > > >>> > > > > > > > >> > > > > > an
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare marker
> > > (and
> > > > > > > >>> eventually a
> > > > > > > >>> > > > > > complete
> > > > > > > >>> > > > > > > > >> marker)
> > > > > > > >>> > > > > > > > >> > > to
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't cause
> the
> > > > > > > transaction
> > > > > > > >>> to
> > > > > > > >>> > > hang,
> > > > > > > >>> > > > > but
> > > > > > > >>> > > > > > > it
> > > > > > > >>> > > > > > > > >> will
> > > > > > > >>> > > > > > > > >> > > > break
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The proposal
> in
> > > this
> > > > > KIP
> > > > > > > >>> will
> > > > > > > >>> > > > address
> > > > > > > >>> > > > > > this
> > > > > > > >>> > > > > > > > >> issue
> > > > > > > >>> > > > > > > > >> > > too.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.
> "However, I
> > > was
> > > > > > > >>> writing it
> > > > > > > >>> > so
> > > > > > > >>> > > > > that
> > > > > > > >>> > > > > > we
> > > > > > > >>> > > > > > > > can
> > > > > > > >>> > > > > > > > >> > > > > > distinguish
> > > > > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old clients
> > where
> > > we
> > > > > > don't
> > > > > > > >>> have
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > ability
> > > > > > > >>> > > > > > > > do
> > > > > > > >>> > > > > > > > >> > this
> > > > > > > >>> > > > > > > > >> > > > > > > operation
> > > > > > > >>> > > > > > > > >> > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients that
> > can.
> > > > (Old
> > > > > > > >>> clients
> > > > > > > >>> > > don't
> > > > > > > >>> > > > > > bump
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > epoch
> > > > > > > >>> > > > > > > > >> > > > on
> > > > > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > > > > >>> > > > > > > > >> > > > > > > > > so
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for sure
> the
> > > > write
> > > > > > > >>> belongs
> > > > > > > >>> > to
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > given
> > > > > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I am
> > > wondering
> > > > > why
> > > > > > > we
> > > > > > > >>> need
> > > > > > > >>> > > to
> > > > > > > >>> > > > > > > > >> distinguish
> > > > > > > >>> > > > > > > > >> > > > whether
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written by the
> > old
> > > > and
> > > > > > the
> > > > > > > >>> new
> > > > > > > >>> > > > client.
> > > > > > > >>> > > > > > > Could
> > > > > > > >>> > > > > > > > >> you
> > > > > > > >>> > > > > > > > >> > > > > describe
> > > > > > > >>> > > > > > > > >> > > > > > > > what
> > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > differently if
> > we
> > > > know
> > > > > > the
> > > > > > > >>> > marker
> > > > > > > >>> > > is
> > > > > > > >>> > > > > > > written
> > > > > > > >>> > > > > > > > >> by
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > client?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If we do
> > > need
> > > > a
> > > > > > way
> > > > > > > to
> > > > > > > >>> > > > > distinguish
> > > > > > > >>> > > > > > > > >> whether
> > > > > > > >>> > > > > > > > >> > > the
> > > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old and
> the
> > > new
> > > > > > > client.
> > > > > > > >>> > Would
> > > > > > > >>> > > it
> > > > > > > >>> > > > > be
> > > > > > > >>> > > > > > > > >> simpler
> > > > > > > >>> > > > > > > > >> > to
> > > > > > > >>> > > > > > > > >> > > > just
> > > > > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field instead
> of
> > > > > > > indirectly
> > > > > > > >>> > > through
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> previous
> > > > > > > >>> > > > > > > > >> > > > > produce
> > > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3 It's not
> > > clear
> > > > > to
> > > > > > me
> > > > > > > >>> why
> > > > > > > >>> > we
> > > > > > > >>> > > > only
> > > > > > > >>> > > > > > add
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > previous
> > > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the complete
> > > marker,
> > > > > but
> > > > > > > >>> not in
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > > prepare
> > > > > > > >>> > > > > > > > >> > marker.
> > > > > > > >>> > > > > > > > >> > > > If
> > > > > > > >>> > > > > > > > >> > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > want
> > > > > > > >>> > > > > > > > >> > > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether a
> marker
> > > is
> > > > > > > written
> > > > > > > >>> by
> > > > > > > >>> > the
> > > > > > > >>> > > > new
> > > > > > > >>> > > > > > > > client
> > > > > > > >>> > > > > > > > >> or
> > > > > > > >>> > > > > > > > >> > > not,
> > > > > > > >>> > > > > > > > >> > > > > it
> > > > > > > >>> > > > > > > > >> > > > > > > > seems
> > > > > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do this
> > > > > consistently
> > > > > > > for
> > > > > > > >>> all
> > > > > > > >>> > > > > markers.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4 What
> about
> > > the
> > > > > > > >>> > > > > TransactionLogValue
> > > > > > > >>> > > > > > > > >> record
> > > > > > > >>> > > > > > > > >> > > > > > > representing
> > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state? Should
> we
> > > > also
> > > > > > > >>> > distinguish
> > > > > > > >>> > > > > > whether
> > > > > > > >>> > > > > > > > it's
> > > > > > > >>> > > > > > > > >> > > > written
> > > > > > > >>> > > > > > > > >> > > > > by
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > old
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new client?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In the
> > > overflow
> > > > > > case,
> > > > > > > >>> it's
> > > > > > > >>> > > > still
> > > > > > > >>> > > > > > not
> > > > > > > >>> > > > > > > > >> clear
> > > > > > > >>> > > > > > > > >> > to
> > > > > > > >>> > > > > > > > >> > > me
> > > > > > > >>> > > > > > > > >> > > > > why
> > > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous
> produce
> > > Id
> > > > in
> > > > > > the
> > > > > > > >>> > prepare
> > > > > > > >>> > > > > > marker
> > > > > > > >>> > > > > > > > >> while
> > > > > > > >>> > > > > > > > >> > > > writing
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the
> complete
> > > > > marker.
> > > > > > > You
> > > > > > > >>> > > > mentioned
> > > > > > > >>> > > > > > that
> > > > > > > >>> > > > > > > > >> it's
> > > > > > > >>> > > > > > > > >> > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could
> > downgrade
> > > > > with
> > > > > > > >>> either
> > > > > > > >>> > the
> > > > > > > >>> > > > > > prepare
> > > > > > > >>> > > > > > > > >> marker
> > > > > > > >>> > > > > > > > >> > > or
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In either
> case,
> > > the
> > > > > > > >>> downgraded
> > > > > > > >>> > > > > > coordinator
> > > > > > > >>> > > > > > > > >> should
> > > > > > > >>> > > > > > > > >> > > see
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > same
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > (probably the
> > > > previous
> > > > > > > >>> produce
> > > > > > > >>> > > Id),
> > > > > > > >>> > > > > > right?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed, Dec
> 20,
> > > 2023
> > > > > at
> > > > > > > >>> 6:00 PM
> > > > > > > >>> > > > > Justine
> > > > > > > >>> > > > > > > > Olshan
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey Jun,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks for
> > > taking
> > > > a
> > > > > > look
> > > > > > > >>> at
> > > > > > > >>> > the
> > > > > > > >>> > > > KIP
> > > > > > > >>> > > > > > > again.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100. For the
> > > epoch
> > > > > > > >>> overflow
> > > > > > > >>> > > case,
> > > > > > > >>> > > > > only
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> > marker
> > > > > > > >>> > > > > > > > >> > > > > will
> > > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > > >>> > > > > > > > >> > > > > > > > > max
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps the
> > > behavior
> > > > > of
> > > > > > > the
> > > > > > > >>> rest
> > > > > > > >>> > > of
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> markers
> > > > > > > >>> > > > > > > > >> > > where
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > last
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch of the
> > > > > > transaction
> > > > > > > >>> > > records +
> > > > > > > >>> > > > > 1.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101. You are
> > > > correct
> > > > > > > that
> > > > > > > >>> we
> > > > > > > >>> > > don't
> > > > > > > >>> > > > > > need
> > > > > > > >>> > > > > > > to
> > > > > > > >>> > > > > > > > >> > write
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the same.
> > > > > However,
> > > > > > I
> > > > > > > >>> was
> > > > > > > >>> > > > writing
> > > > > > > >>> > > > > it
> > > > > > > >>> > > > > > > so
> > > > > > > >>> > > > > > > > >> that
> > > > > > > >>> > > > > > > > >> > we
> > > > > > > >>> > > > > > > > >> > > > can
> > > > > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old clients
> > > where
> > > > we
> > > > > > > don't
> > > > > > > >>> > have
> > > > > > > >>> > > > the
> > > > > > > >>> > > > > > > > ability
> > > > > > > >>> > > > > > > > >> do
> > > > > > > >>> > > > > > > > >> > > this
> > > > > > > >>> > > > > > > > >> > > > > > > > operation
> > > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > clients that
> > > can.
> > > > > (Old
> > > > > > > >>> clients
> > > > > > > >>> > > > don't
> > > > > > > >>> > > > > > > bump
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > epoch
> > > > > > > >>> > > > > > > > >> > > > > on
> > > > > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say for sure
> > the
> > > > > write
> > > > > > > >>> belongs
> > > > > > > >>> > > to
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > given
> > > > > > > >>> > > > > > > > >> > > > > > transaction).
> > > > > > > >>> > > > > > > > >> > > > > > > > If
> > > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn
> request
> > > > from
> > > > > a
> > > > > > > new
> > > > > > > >>> > > client,
> > > > > > > >>> > > > we
> > > > > > > >>> > > > > > > will
> > > > > > > >>> > > > > > > > >> fill
> > > > > > > >>> > > > > > > > >> > > this
> > > > > > > >>> > > > > > > > >> > > > > > > field.
> > > > > > > >>> > > > > > > > >> > > > > > > > We
> > > > > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > guarantee
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that any
> > EndTxn
> > > > > > requests
> > > > > > > >>> with
> > > > > > > >>> > > the
> > > > > > > >>> > > > > same
> > > > > > > >>> > > > > > > > epoch
> > > > > > > >>> > > > > > > > >> > are
> > > > > > > >>> > > > > > > > >> > > > from
> > > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > same
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the same
> > > > > transaction.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102. In
> > prepare
> > > > > phase,
> > > > > > > we
> > > > > > > >>> have
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > same
> > > > > > > >>> > > > > > > > >> > producer
> > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had. It is
> the
> > > > > > producer
> > > > > > > >>> ID and
> > > > > > > >>> > > > epoch
> > > > > > > >>> > > > > > > that
> > > > > > > >>> > > > > > > > >> are
> > > > > > > >>> > > > > > > > >> > on
> > > > > > > >>> > > > > > > > >> > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > > > > >>> > > > > > > > >> > > > > > > > > In
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase, we
> stay
> > > the
> > > > > > same
> > > > > > > >>> unless
> > > > > > > >>> > > it
> > > > > > > >>> > > > is
> > > > > > > >>> > > > > > the
> > > > > > > >>> > > > > > > > >> > overflow
> > > > > > > >>> > > > > > > > >> > > > > case.
> > > > > > > >>> > > > > > > > >> > > > > > > In
> > > > > > > >>> > > > > > > > >> > > > > > > > > that
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set the
> > producer
> > > > ID
> > > > > to
> > > > > > > >>> the new
> > > > > > > >>> > > one
> > > > > > > >>> > > > > we
> > > > > > > >>> > > > > > > > >> generated
> > > > > > > >>> > > > > > > > >> > > and
> > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > complete.
> This
> > > is
> > > > > for
> > > > > > > >>> > downgrade
> > > > > > > >>> > > > > > > > >> compatibility.
> > > > > > > >>> > > > > > > > >> > > The
> > > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety
> guards
> > > for
> > > > > > > retries
> > > > > > > >>> and
> > > > > > > >>> > > > > > failovers.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In prepare
> > phase
> > > > for
> > > > > > > epoch
> > > > > > > >>> > > > overflow
> > > > > > > >>> > > > > > case
> > > > > > > >>> > > > > > > > >> only
> > > > > > > >>> > > > > > > > >> > we
> > > > > > > >>> > > > > > > > >> > > > > store
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID. This is
> > for
> > > > the
> > > > > > case
> > > > > > > >>> where
> > > > > > > >>> > > we
> > > > > > > >>> > > > > > reload
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > transaction
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > prepare
> state.
> > > > Once
> > > > > > the
> > > > > > > >>> > > > transaction
> > > > > > > >>> > > > > is
> > > > > > > >>> > > > > > > > >> > committed,
> > > > > > > >>> > > > > > > > >> > > > we
> > > > > > > >>> > > > > > > > >> > > > > > can
> > > > > > > >>> > > > > > > > >> > > > > > > > use
> > > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the
> client
> > > > > already
> > > > > > is
> > > > > > > >>> > using.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In commit
> > phase,
> > > > we
> > > > > > > store
> > > > > > > >>> the
> > > > > > > >>> > > > > previous
> > > > > > > >>> > > > > > > > >> producer
> > > > > > > >>> > > > > > > > >> > > ID
> > > > > > > >>> > > > > > > > >> > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > >>> > > > > > > > >> > > > > > > > of
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I think it
> is
> > > > easier
> > > > > > to
> > > > > > > >>> think
> > > > > > > >>> > of
> > > > > > > >>> > > > it
> > > > > > > >>> > > > > as
> > > > > > > >>> > > > > > > > just
> > > > > > > >>> > > > > > > > >> how
> > > > > > > >>> > > > > > > > >> > > we
> > > > > > > >>> > > > > > > > >> > > > > were
> > > > > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and epoch
> > > before,
> > > > > with
> > > > > > > >>> some
> > > > > > > >>> > > extra
> > > > > > > >>> > > > > > > > bookeeping
> > > > > > > >>> > > > > > > > >> > and
> > > > > > > >>> > > > > > > > >> > > > edge
> > > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged
> fields.
> > > We
> > > > > have
> > > > > > > to
> > > > > > > >>> do
> > > > > > > >>> > it
> > > > > > > >>> > > > this
> > > > > > > >>> > > > > > way
> > > > > > > >>> > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > > > > >>> > > > > > > > >> > > > > > > > > with
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > downgrades.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103. Next
> > > producer
> > > > > ID
> > > > > > is
> > > > > > > >>> for
> > > > > > > >>> > > > prepare
> > > > > > > >>> > > > > > > > status
> > > > > > > >>> > > > > > > > >> and
> > > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after
> > complete.
> > > > The
> > > > > > > >>> reason why
> > > > > > > >>> > > we
> > > > > > > >>> > > > > need
> > > > > > > >>> > > > > > > two
> > > > > > > >>> > > > > > > > >> > > separate
> > > > > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > backwards
> > > > > > compatibility.
> > > > > > > >>> We
> > > > > > > >>> > need
> > > > > > > >>> > > > to
> > > > > > > >>> > > > > > keep
> > > > > > > >>> > > > > > > > the
> > > > > > > >>> > > > > > > > >> > same
> > > > > > > >>> > > > > > > > >> > > > > > > semantics
> > > > > > > >>> > > > > > > > >> > > > > > > > > for
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged
> > field
> > > > in
> > > > > > case
> > > > > > > >>> we
> > > > > > > >>> > > > > downgrade.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104. We set
> > the
> > > > > fields
> > > > > > > as
> > > > > > > >>> we
> > > > > > > >>> > do
> > > > > > > >>> > > in
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> > > > transactional
> > > > > > > >>> > > > > > > > >> > > > > > > state
> > > > > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do this for
> > > > > > > compatibility
> > > > > > > >>> --
> > > > > > > >>> > if
> > > > > > > >>> > > we
> > > > > > > >>> > > > > > > > >> downgrade,
> > > > > > > >>> > > > > > > > >> > we
> > > > > > > >>> > > > > > > > >> > > > will
> > > > > > > >>> > > > > > > > >> > > > > > > only
> > > > > > > >>> > > > > > > > >> > > > > > > > > have
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged
> > > fields)
> > > > > It
> > > > > > > >>> will be
> > > > > > > >>> > > the
> > > > > > > >>> > > > > old
> > > > > > > >>> > > > > > > > >> producer
> > > > > > > >>> > > > > > > > >> > ID
> > > > > > > >>> > > > > > > > >> > > > and
> > > > > > > >>> > > > > > > > >> > > > > > max
> > > > > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope this
> > helps.
> > > > Let
> > > > > > me
> > > > > > > >>> know
> > > > > > > >>> > if
> > > > > > > >>> > > > you
> > > > > > > >>> > > > > > have
> > > > > > > >>> > > > > > > > >> > further
> > > > > > > >>> > > > > > > > >> > > > > > > questions.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Justine
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On Wed, Dec
> > 20,
> > > > 2023
> > > > > > at
> > > > > > > >>> > 3:33 PM
> > > > > > > >>> > > > Jun
> > > > > > > >>> > > > > > Rao
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> <jun@confluent.io.invalid
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi,
> Justine,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It seems
> > that
> > > > you
> > > > > > have
> > > > > > > >>> made
> > > > > > > >>> > > some
> > > > > > > >>> > > > > > > changes
> > > > > > > >>> > > > > > > > >> to
> > > > > > > >>> > > > > > > > >> > > > KIP-890
> > > > > > > >>> > > > > > > > >> > > > > > > since
> > > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> particular,
> > we
> > > > are
> > > > > > > >>> changing
> > > > > > > >>> > > the
> > > > > > > >>> > > > > > format
> > > > > > > >>> > > > > > > > of
> > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionLogValue.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > comments
> > > related
> > > > > to
> > > > > > > >>> that.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100. Just
> to
> > > be
> > > > > > clear.
> > > > > > > >>> The
> > > > > > > >>> > > > > overflow
> > > > > > > >>> > > > > > > case
> > > > > > > >>> > > > > > > > >> > (i.e.
> > > > > > > >>> > > > > > > > >> > > > > when a
> > > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > generated)
> > is
> > > > when
> > > > > > the
> > > > > > > >>> > current
> > > > > > > >>> > > > > epoch
> > > > > > > >>> > > > > > > > >> equals
> > > > > > > >>> > > > > > > > >> > to
> > > > > > > >>> > > > > > > > >> > > > max
> > > > > > > >>> > > > > > > > >> > > > > -
> > > > > > > >>> > > > > > > > >> > > > > > 1
> > > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101. For
> the
> > > > "not
> > > > > > > epoch
> > > > > > > >>> > > > overflow"
> > > > > > > >>> > > > > > > case,
> > > > > > > >>> > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > write
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > tagged
> field
> > > in
> > > > > the
> > > > > > > >>> complete
> > > > > > > >>> > > > > phase.
> > > > > > > >>> > > > > > Do
> > > > > > > >>> > > > > > > > we
> > > > > > > >>> > > > > > > > >> > need
> > > > > > > >>> > > > > > > > >> > > to
> > > > > > > >>> > > > > > > > >> > > > > do
> > > > > > > >>> > > > > > > > >> > > > > > > that
> > > > > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > doesn't
> > change
> > > > in
> > > > > > this
> > > > > > > >>> case?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102. It
> > seems
> > > > that
> > > > > > the
> > > > > > > >>> > meaning
> > > > > > > >>> > > > for
> > > > > > > >>> > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId/ProducerEpoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > TransactionLogValue
> > > > > > > >>> changes
> > > > > > > >>> > > > > > depending
> > > > > > > >>> > > > > > > on
> > > > > > > >>> > > > > > > > >> the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > > TransactionStatus
> > > > > > > is
> > > > > > > >>> > > > ongoing,
> > > > > > > >>> > > > > > they
> > > > > > > >>> > > > > > > > >> > > represent
> > > > > > > >>> > > > > > > > >> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > current
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> current
> > > > > > > >>> ProducerEpoch.
> > > > > > > >>> > > When
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > > > > >>> > > > > > > > >> > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> PrepareCommit/PrepareAbort,
> > > > > > > >>> > > they
> > > > > > > >>> > > > > > > > represent
> > > > > > > >>> > > > > > > > >> > the
> > > > > > > >>> > > > > > > > >> > > > > > current
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > > > > ProducerEpoch.
> > > > > > > >>> When the
> > > > > > > >>> > > > > > > > >> > TransactionStatus
> > > > > > > >>> > > > > > > > >> > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > further
> > depend
> > > > on
> > > > > > > >>> whether
> > > > > > > >>> > the
> > > > > > > >>> > > > > epoch
> > > > > > > >>> > > > > > > > >> overflows
> > > > > > > >>> > > > > > > > >> > > or
> > > > > > > >>> > > > > > > > >> > > > > not.
> > > > > > > >>> > > > > > > > >> > > > > > > If
> > > > > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > overflow,
> > they
> > > > > > > represent
> > > > > > > >>> > the
> > > > > > > >>> > > > > > current
> > > > > > > >>> > > > > > > > >> > > ProducerId
> > > > > > > >>> > > > > > > > >> > > > > and
> > > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> ProducerEpoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > (max).
> > > > Otherwise,
> > > > > > they
> > > > > > > >>> > > represent
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > > > > newly
> > > > > > > >>> > > > > > > > >> > > > > generated
> > > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > ProducerEpoch
> > > of
> > > > > 0.
> > > > > > Is
> > > > > > > >>> that
> > > > > > > >>> > > > right?
> > > > > > > >>> > > > > > > This
> > > > > > > >>> > > > > > > > >> seems
> > > > > > > >>> > > > > > > > >> > > not
> > > > > > > >>> > > > > > > > >> > > > > > easy
> > > > > > > >>> > > > > > > > >> > > > > > > to
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we provide
> > > some
> > > > > > > examples
> > > > > > > >>> > like
> > > > > > > >>> > > > what
> > > > > > > >>> > > > > > > Artem
> > > > > > > >>> > > > > > > > >> has
> > > > > > > >>> > > > > > > > >> > > done
> > > > > > > >>> > > > > > > > >> > > > > in
> > > > > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> considered a
> > > > > simpler
> > > > > > > >>> design
> > > > > > > >>> > > > where
> > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId/ProducerEpoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > represent
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the same
> > value
> > > > > (e.g.
> > > > > > > >>> for the
> > > > > > > >>> > > > > current
> > > > > > > >>> > > > > > > > >> > > transaction)
> > > > > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > TransactionStatus
> > > > > > and
> > > > > > > >>> epoch
> > > > > > > >>> > > > > > overflow?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103. It's
> > not
> > > > > clear
> > > > > > to
> > > > > > > >>> me
> > > > > > > >>> > why
> > > > > > > >>> > > we
> > > > > > > >>> > > > > > need
> > > > > > > >>> > > > > > > 3
> > > > > > > >>> > > > > > > > >> > fields:
> > > > > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > PrevProducerId,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > NextProducerId.
> > > > > > Could
> > > > > > > we
> > > > > > > >>> > just
> > > > > > > >>> > > > have
> > > > > > > >>> > > > > > > > >> ProducerId
> > > > > > > >>> > > > > > > > >> > > and
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104. For
> > > > > > > >>> > > WriteTxnMarkerRequests,
> > > > > > > >>> > > > > if
> > > > > > > >>> > > > > > > the
> > > > > > > >>> > > > > > > > >> > > producer
> > > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we set the
> > > > > > producerId
> > > > > > > >>> and
> > > > > > > >>> > the
> > > > > > > >>> > > > > > > > >> producerEpoch?
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > > >>> > > > > > > > >> > > > > >
> > > > > > > >>> > > > > > > > >> > > > >
> > > > > > > >>> > > > > > > > >> > > >
> > > > > > > >>> > > > > > > > >> > >
> > > > > > > >>> > > > > > > > >> >
> > > > > > > >>> > > > > > > > >>
> > > > > > > >>> > > > > > > > >
> > > > > > > >>> > > > > > > >
> > > > > > > >>> > > > > > >
> > > > > > > >>> > > > > >
> > > > > > > >>> > > > >
> > > > > > > >>> > > >
> > > > > > > >>> > >
> > > > > > > >>> >
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Justine Olshan <jo...@confluent.io.INVALID>]

Hey folks, just wanted to share another quick update with KIP-890.

Making the change to bump epoch after every transaction means that we no
longer need to call InitProducerId during the life of the producer to fence
requests from previous transactions. With KIP-890 part 2, the new client
will no longer call InitProducerId except on startup to fence a previous
instance.

With this change, some other calls to InitProducerId were inspected
including the call after receiving an InvalidPidMappingException. This
exception was changed to abortable as part of KIP-360: Improve reliability
of idempotent/transactional producer
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820>.
However, this change means that we can violate EOS guarantees. As an
example:

Consider an application that is copying data from one partition to another

   - Application instance A processes to offset 4
   - Application instance B comes up and fences application instance A
   - Application instance B processes to offset 5
   - Application instances A and B are idle for transaction.id.expiration.ms,
   transaction id expires on server
   - Application instance A attempts to process offset 5 (since in its
   view, that is next) -- if we recover from invalid pid mapping, we can
   duplicate this processing

Thus, INVALID_PID_MAPPING should be fatal to the producer and as part of
KIP-890 the new client in 4.0 will treat the exception as fatal.

This is consistent with KIP-1050: Consistent error handling for Transactions
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions>
where
errors that are fatal to the producer are in the "application recoverable"
category. This is a grouping that indicates to the client that the producer
needs to restart and recovery on the application side is necessary.
KIP-1050 is approved so we are consistent with that decision.

Let me know if there are any questions. I've updated the KIP to reflect
this change.

Justine

On Wed, Jul 10, 2024 at 3:34 PM Jun Rao <ju...@confluent.io.invalid> wrote:

> Sounds good, Justine. It would be useful to document that in the KIP.
>
> Thanks,
>
> Jun
>
> On Wed, Jul 10, 2024 at 2:59 PM Justine Olshan
> <jo...@confluent.io.invalid>
> wrote:
>
> > The client will send the newest EndTxn request version if and only if
> both
> > the client and the server support kip-890 part 2.
> > We set the value in the record based on the EndTxn version.
> >
> > Justine
> >
> > On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <ju...@confluent.io.invalid>
> wrote:
> >
> > > Hi, Justine,
> > >
> > > Thanks for the reply.
> > >
> > > 120. If the broker sends TV Y for the finalized version in
> > > ApiVersionResponse, but the client doesn't support Y, how does the
> broker
> > > know the TV that the client supports?
> > >
> > > Jun
> > >
> > > On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> > > <jo...@confluent.io.invalid>
> > > wrote:
> > >
> > > > Hey Jun,
> > > >
> > > > No worries. Work on this KIP has been blocked for a bit anyways --
> > > catching
> > > > up and rereading what I wrote :)
> > > >
> > > > 120. ClientTransactionProtocolVersion is the transaction version as
> > > defined
> > > > by the highest transaction version (feature version value) supported
> by
> > > the
> > > > client and the server. This works by the broker sending an
> > > > ApiVersionsRequest to the client with the finalized version. Assuming
> > > > kip-890 part 2 is enabled by transaction version Y, if this request
> > > > contains finalized version Y and the client has the logic to set this
> > > > field, it will set Y. If the server has Y - 1 (kip 890 part 2 not
> > enable)
> > > > the client will send Y - 1, even though the client has the ability to
> > > > support kip-890 part 2.
> > > >
> > > > 121. You are correct that this is not needed. However, currently that
> > > field
> > > > is already being set in memory -- just not written to disk. I think
> it
> > is
> > > > ok to write it to disk though. Let me know if you think otherwise.
> > > >
> > > > Justine
> > > >
> > > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao <ju...@confluent.io.invalid>
> > > wrote:
> > > >
> > > > > Hi, Justine,
> > > > >
> > > > > Thanks for the update and sorry for the late reply.
> > > > >
> > > > > 120. I am wondering what value is used for
> > > > > ClientTransactionProtocolVersion. Is it the version of the
> > > EndTxnRequest?
> > > > >
> > > > > 121. Earlier, you made the change to set lastProducerId in PREPARE
> to
> > > > > indicate that the marker is written for the new client. With the
> new
> > > > > ClientTransactionProtocolVersion field, it seems this is no longer
> > > > > necessary.
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > > > <jo...@confluent.io.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hi there -- another update!
> > > > > >
> > > > > > When looking into the implementation for the safe epoch bumps I
> > > > realized
> > > > > > that we are already populating previousProducerID in memory as
> part
> > > of
> > > > > > KIP-360.
> > > > > > If we are to start using flexible fields, it is better to always
> > use
> > > > this
> > > > > > information and have an explicit (tagged) field to indicate
> whether
> > > the
> > > > > > client supports KIP-890 part 2.
> > > > > >
> > > > > > I've included the extra field and how it is set in the KIP. I've
> > also
> > > > > > updated the KIP to explain that we will be setting the tagged
> > fields
> > > > when
> > > > > > they are available for all transitions.
> > > > > >
> > > > > > Finally, I added clearer text about the transaction protocol
> > versions
> > > > > > included with this KIP. 1 for flexible transaction state records
> > and
> > > 2
> > > > > for
> > > > > > KIP-890 part 2 enablement.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <
> > jolshan@confluent.io
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hey there -- small update to the KIP,
> > > > > > >
> > > > > > > The KIP mentioned introducing ABORTABLE_ERROR and bumping
> > > > > TxnOffsetCommit
> > > > > > > and Produce requests. I've changed the name in the KIP to
> > > > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > > > AbortableTransactionException to match the pattern we had for
> > other
> > > > > > errors.
> > > > > > > I also mentioned bumping all 6 transactional APIs so we can
> > future
> > > > > > > proof/support the error on the client going forward. If a
> future
> > > > change
> > > > > > > wants to have an error scenario that requires us to abort the
> > > > > > transaction,
> > > > > > > we can rely on the 3.8+ clients to support it. We ran into
> issues
> > > > > finding
> > > > > > > good/generic error codes that older clients could support while
> > > > working
> > > > > > on
> > > > > > > this KIP, so this should help in the future.
> > > > > > >
> > > > > > > The features discussion is still ongoing in KIP-1022. Will
> update
> > > > again
> > > > > > > here when that concludes.
> > > > > > >
> > > > > > > Justine
> > > > > > >
> > > > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> > > jolshan@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> I don't think AddPartitions is a good example since we
> currenly
> > > > don't
> > > > > > >> gate the version on TV or MV. (We only set a different flag
> > > > depending
> > > > > on
> > > > > > >> the TV)
> > > > > > >>
> > > > > > >> Even if we did want to gate it on TV, I think the idea is to
> > move
> > > > away
> > > > > > >> from MV gating inter broker protocols. Ideally we can get to a
> > > state
> > > > > > where
> > > > > > >> MV is just used for metadata changes.
> > > > > > >>
> > > > > > >> I think some of this discussion might fit more with the
> feature
> > > > > version
> > > > > > >> KIP, so I can try to open that up soon. Until we settle that,
> > some
> > > > of
> > > > > > the
> > > > > > >> work in KIP-890 is blocked.
> > > > > > >>
> > > > > > >> Justine
> > > > > > >>
> > > > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao
> <jun@confluent.io.invalid
> > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >>> Hi, Justine,
> > > > > > >>>
> > > > > > >>> Thanks for the reply.
> > > > > > >>>
> > > > > > >>> Since AddPartitions is an inter broker request, will its
> > version
> > > be
> > > > > > gated
> > > > > > >>> only by TV or other features like MV too? For example, if we
> > need
> > > > to
> > > > > > >>> change
> > > > > > >>> the protocol for AddPartitions for reasons other than txn
> > > > > verification
> > > > > > in
> > > > > > >>> the future, will the new version be gated by a new MV? If so,
> > > does
> > > > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > > > >>>
> > > > > > >>> Jun
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>> > One TV gates the flexible feature version (no rpcs
> involved,
> > > only
> > > > > the
> > > > > > >>> > transactional records that should only be gated by TV)
> > > > > > >>> > Another TV gates the ability to turn on kip-890 part 2.
> This
> > > > would
> > > > > > >>> gate the
> > > > > > >>> > version of Produce and EndTxn (likely only used by
> > > transactions),
> > > > > and
> > > > > > >>> > specifies a flag in AddPartitionsToTxn though the version
> is
> > > > > already
> > > > > > >>> used
> > > > > > >>> > without TV.
> > > > > > >>> >
> > > > > > >>> > I think the only concern is the Produce request and we
> could
> > > > > consider
> > > > > > >>> work
> > > > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > > > >>> >
> > > > > > >>> > Justine
> > > > > > >>> >
> > > > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> > > <jun@confluent.io.invalid
> > > > >
> > > > > > >>> wrote:
> > > > > > >>> >
> > > > > > >>> > > Hi, Justine,
> > > > > > >>> > >
> > > > > > >>> > > Which PRC/record protocols will TV guard? Going forward,
> > will
> > > > > those
> > > > > > >>> > > PRC/record protocols only be guarded by TV and not by
> other
> > > > > > features
> > > > > > >>> like
> > > > > > >>> > > MV?
> > > > > > >>> > >
> > > > > > >>> > > Thanks,
> > > > > > >>> > >
> > > > > > >>> > > Jun
> > > > > > >>> > >
> > > > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > > > >>> > <jolshan@confluent.io.invalid
> > > > > > >>> > > >
> > > > > > >>> > > wrote:
> > > > > > >>> > >
> > > > > > >>> > > > Hi Jun,
> > > > > > >>> > > >
> > > > > > >>> > > > Sorry I think I misunderstood your question or answered
> > > > > > >>> incorrectly.
> > > > > > >>> > The
> > > > > > >>> > > TV
> > > > > > >>> > > > version should ideally be fully independent from MV.
> > > > > > >>> > > > At least for the changes I proposed, TV should not
> affect
> > > MV
> > > > > and
> > > > > > MV
> > > > > > >>> > > should
> > > > > > >>> > > > not affect TV/
> > > > > > >>> > > >
> > > > > > >>> > > > I think if we downgrade TV, only that feature should
> > > > downgrade.
> > > > > > >>> > Likewise
> > > > > > >>> > > > the same with MV. The finalizedFeatures should just
> > reflect
> > > > the
> > > > > > >>> feature
> > > > > > >>> > > > downgrade we made.
> > > > > > >>> > > >
> > > > > > >>> > > > I also plan to write a new KIP for managing the disk
> > format
> > > > and
> > > > > > >>> upgrade
> > > > > > >>> > > > tool as we will need new flags to support these
> features.
> > > > That
> > > > > > >>> should
> > > > > > >>> > > help
> > > > > > >>> > > > clarify some things.
> > > > > > >>> > > >
> > > > > > >>> > > > Justine
> > > > > > >>> > > >
> > > > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > > > <jun@confluent.io.invalid
> > > > > > >
> > > > > > >>> > > wrote:
> > > > > > >>> > > >
> > > > > > >>> > > > > Hi, Justine,
> > > > > > >>> > > > >
> > > > > > >>> > > > > Thanks for the reply.
> > > > > > >>> > > > >
> > > > > > >>> > > > > So, if we downgrade TV, we could implicitly downgrade
> > > > another
> > > > > > >>> feature
> > > > > > >>> > > > (say
> > > > > > >>> > > > > MV) that has dependency (e.g. RPC). What would we
> > return
> > > > for
> > > > > > >>> > > > > FinalizedFeatures for MV in ApiVersionsResponse in
> that
> > > > case?
> > > > > > >>> > > > >
> > > > > > >>> > > > > Thanks,
> > > > > > >>> > > > >
> > > > > > >>> > > > > Jun
> > > > > > >>> > > > >
> > > > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > > > >>> > > > <jolshan@confluent.io.invalid
> > > > > > >>> > > > > >
> > > > > > >>> > > > > wrote:
> > > > > > >>> > > > >
> > > > > > >>> > > > > > Hey Jun,
> > > > > > >>> > > > > >
> > > > > > >>> > > > > > Yes, the idea is that if we downgrade TV
> (transaction
> > > > > > version)
> > > > > > >>> we
> > > > > > >>> > > will
> > > > > > >>> > > > > stop
> > > > > > >>> > > > > > using the add partitions to txn optimization and
> stop
> > > > > writing
> > > > > > >>> the
> > > > > > >>> > > > > flexible
> > > > > > >>> > > > > > feature version of the log.
> > > > > > >>> > > > > > In the compatibility section I included some
> > > explanations
> > > > > on
> > > > > > >>> how
> > > > > > >>> > this
> > > > > > >>> > > > is
> > > > > > >>> > > > > > done.
> > > > > > >>> > > > > >
> > > > > > >>> > > > > > Thanks,
> > > > > > >>> > > > > > Justine
> > > > > > >>> > > > > >
> > > > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > > > >>> <ju...@confluent.io.invalid>
> > > > > > >>> > > > > wrote:
> > > > > > >>> > > > > >
> > > > > > >>> > > > > > > Hi, Justine,
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > > > Thanks for the update.
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > > > If we ever downgrade the transaction feature, any
> > > > feature
> > > > > > >>> > depending
> > > > > > >>> > > > on
> > > > > > >>> > > > > > > changes on top of those RPC/record
> > > > > > >>> > > > > > > (AddPartitionsToTxnRequest/TransactionLogValue)
> > > changes
> > > > > > made
> > > > > > >>> in
> > > > > > >>> > > > KIP-890
> > > > > > >>> > > > > > > will be automatically downgraded too?
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > > > Jun
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine Olshan
> > > > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > > > >>> > > > > > > wrote:
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > > > > Hey Jun,
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > > > I wanted to get back to you about your
> questions
> > > > about
> > > > > > >>> MV/IBP.
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > > > Looking at the options, I think it makes the
> most
> > > > sense
> > > > > > to
> > > > > > >>> > > create a
> > > > > > >>> > > > > > > > separate feature for transactions and use that
> to
> > > > > version
> > > > > > >>> gate
> > > > > > >>> > > the
> > > > > > >>> > > > > > > features
> > > > > > >>> > > > > > > > we need to version gate (flexible transactional
> > > state
> > > > > > >>> records
> > > > > > >>> > and
> > > > > > >>> > > > > using
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > new protocol)
> > > > > > >>> > > > > > > > I've updated the KIP to include this change.
> > > > Hopefully
> > > > > > >>> that's
> > > > > > >>> > > > > > everything
> > > > > > >>> > > > > > > we
> > > > > > >>> > > > > > > > need for this KIP :)
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > > > Justine
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine Olshan
> <
> > > > > > >>> > > > jolshan@confluent.io
> > > > > > >>> > > > > >
> > > > > > >>> > > > > > > > wrote:
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > > > > Thanks Jun,
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > > I will update the KIP with the prev field for
> > > > prepare
> > > > > > as
> > > > > > >>> > well.
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > > PREPARE
> > > > > > >>> > > > > > > > > producerId: x
> > > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty or z if
> y
> > > will
> > > > > > >>> overflow
> > > > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > > COMPLETE
> > > > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we overflowed
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > > Thanks again,
> > > > > > >>> > > > > > > > > Justine
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun Rao
> > > > > > >>> > > <jun@confluent.io.invalid
> > > > > > >>> > > > >
> > > > > > >>> > > > > > > > wrote:
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > >> Hi, Justine,
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > > > >>> > > > > > > > >> (1) My point was that the coordinator could
> > fail
> > > > > right
> > > > > > >>> after
> > > > > > >>> > > > > writing
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> prepare marker. When the new txn coordinator
> > > > > generates
> > > > > > >>> the
> > > > > > >>> > > > > complete
> > > > > > >>> > > > > > > > marker
> > > > > > >>> > > > > > > > >> after the failover, it needs some field from
> > the
> > > > > > prepare
> > > > > > >>> > > marker
> > > > > > >>> > > > to
> > > > > > >>> > > > > > > > >> determine whether it's written by the new
> > > client.
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >> (2) The changing of the behavior sounds good
> > to
> > > > me.
> > > > > We
> > > > > > >>> only
> > > > > > >>> > > want
> > > > > > >>> > > > > to
> > > > > > >>> > > > > > > > return
> > > > > > >>> > > > > > > > >> success if the prepare state is written by
> the
> > > new
> > > > > > >>> client.
> > > > > > >>> > So,
> > > > > > >>> > > > in
> > > > > > >>> > > > > > the
> > > > > > >>> > > > > > > > >> non-overflow case, it seems that we also
> need
> > > sth
> > > > in
> > > > > > the
> > > > > > >>> > > prepare
> > > > > > >>> > > > > > > marker
> > > > > > >>> > > > > > > > to
> > > > > > >>> > > > > > > > >> tell us whether it's written by the new
> > client.
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >> 112. Thanks for the explanation. That sounds
> > > good
> > > > to
> > > > > > me.
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >> Jun
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM Justine
> > Olshan
> > > > > > >>> > > > > > > > >> <jo...@confluent.io.invalid> wrote:
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >> > 101.3 I realized that I actually have two
> > > > > questions.
> > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need to
> > > write
> > > > > the
> > > > > > >>> > > previous
> > > > > > >>> > > > > > > produce
> > > > > > >>> > > > > > > > Id
> > > > > > >>> > > > > > > > >> > tagged field in the end maker so that we
> > know
> > > if
> > > > > the
> > > > > > >>> > marker
> > > > > > >>> > > is
> > > > > > >>> > > > > > from
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> new
> > > > > > >>> > > > > > > > >> > client. Since the end maker is derived
> from
> > > the
> > > > > > >>> prepare
> > > > > > >>> > > > marker,
> > > > > > >>> > > > > > > should
> > > > > > >>> > > > > > > > >> we
> > > > > > >>> > > > > > > > >> > write the previous produce Id in the
> prepare
> > > > > marker
> > > > > > >>> field
> > > > > > >>> > > too?
> > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > >>> > > > > > > > >> > we will lose this information when
> deriving
> > > the
> > > > > end
> > > > > > >>> > marker.
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > The "previous" producer ID is in the
> normal
> > > > > producer
> > > > > > >>> ID
> > > > > > >>> > > field.
> > > > > > >>> > > > > So
> > > > > > >>> > > > > > > yes,
> > > > > > >>> > > > > > > > >> we
> > > > > > >>> > > > > > > > >> > need it in prepare and that was always the
> > > plan.
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I will
> > enumerate
> > > > the
> > > > > > >>> fields
> > > > > > >>> > and
> > > > > > >>> > > > add
> > > > > > >>> > > > > > > them
> > > > > > >>> > > > > > > > to
> > > > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > > > >>> > > > > > > > >> > Say we have producer ID x and epoch y.
> When
> > we
> > > > > > >>> overflow
> > > > > > >>> > > epoch
> > > > > > >>> > > > y
> > > > > > >>> > > > > we
> > > > > > >>> > > > > > > get
> > > > > > >>> > > > > > > > >> > producer ID Z.
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > PREPARE
> > > > > > >>> > > > > > > > >> > producerId: x
> > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged field):
> > empty
> > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty or z
> > if y
> > > > > will
> > > > > > >>> > overflow
> > > > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > COMPLETE
> > > > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged field): x
> > > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > > > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we overflowed
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > (2) In the prepare phase, if we retry and
> > see
> > > > > epoch
> > > > > > -
> > > > > > >>> 1 +
> > > > > > >>> > ID
> > > > > > >>> > > > in
> > > > > > >>> > > > > > last
> > > > > > >>> > > > > > > > >> seen
> > > > > > >>> > > > > > > > >> > fields and are issuing the same command
> (ie
> > > > commit
> > > > > > not
> > > > > > >>> > > abort),
> > > > > > >>> > > > > we
> > > > > > >>> > > > > > > > return
> > > > > > >>> > > > > > > > >> > success. The logic before KIP-890 seems to
> > > > return
> > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > >>> > > > > > > > >> > in this case. Are we intentionally making
> > this
> > > > > > change?
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > Hmm -- we would fence the producer if the
> > > epoch
> > > > is
> > > > > > >>> bumped
> > > > > > >>> > > and
> > > > > > >>> > > > we
> > > > > > >>> > > > > > > get a
> > > > > > >>> > > > > > > > >> > lower epoch. Yes -- we are intentionally
> > > adding
> > > > > this
> > > > > > >>> to
> > > > > > >>> > > > prevent
> > > > > > >>> > > > > > > > fencing.
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > 112. We already merged the code that adds
> > the
> > > > > > >>> VerifyOnly
> > > > > > >>> > > field
> > > > > > >>> > > > > in
> > > > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which is an
> inter
> > > > > broker
> > > > > > >>> > request.
> > > > > > >>> > > > It
> > > > > > >>> > > > > > > seems
> > > > > > >>> > > > > > > > >> that
> > > > > > >>> > > > > > > > >> > we didn't bump up the IBP for that. Do you
> > > know
> > > > > why?
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > We no longer need IBP for all interbroker
> > > > requests
> > > > > > as
> > > > > > >>> > > > > ApiVersions
> > > > > > >>> > > > > > > > should
> > > > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > > > >>> > > > > > > > >> > We also handle unsupported version errors
> > > > > correctly
> > > > > > >>> if we
> > > > > > >>> > > > > receive
> > > > > > >>> > > > > > > them
> > > > > > >>> > > > > > > > >> in
> > > > > > >>> > > > > > > > >> > edge cases like upgrades/downgrades.
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > Justine
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM Jun Rao
> > > > > > >>> > > > > <jun@confluent.io.invalid
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > > > > >> wrote:
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> > > 101.3 I realized that I actually have
> two
> > > > > > questions.
> > > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need to
> > > write
> > > > > the
> > > > > > >>> > > previous
> > > > > > >>> > > > > > > produce
> > > > > > >>> > > > > > > > Id
> > > > > > >>> > > > > > > > >> > > tagged field in the end maker so that we
> > > know
> > > > if
> > > > > > the
> > > > > > >>> > > marker
> > > > > > >>> > > > is
> > > > > > >>> > > > > > > from
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > new
> > > > > > >>> > > > > > > > >> > > client. Since the end maker is derived
> > from
> > > > the
> > > > > > >>> prepare
> > > > > > >>> > > > > marker,
> > > > > > >>> > > > > > > > >> should we
> > > > > > >>> > > > > > > > >> > > write the previous produce Id in the
> > prepare
> > > > > > marker
> > > > > > >>> > field
> > > > > > >>> > > > too?
> > > > > > >>> > > > > > > > >> Otherwise,
> > > > > > >>> > > > > > > > >> > > we will lose this information when
> > deriving
> > > > the
> > > > > > end
> > > > > > >>> > > marker.
> > > > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we retry
> and
> > > see
> > > > > > epoch
> > > > > > >>> - 1
> > > > > > >>> > +
> > > > > > >>> > > ID
> > > > > > >>> > > > > in
> > > > > > >>> > > > > > > last
> > > > > > >>> > > > > > > > >> seen
> > > > > > >>> > > > > > > > >> > > fields and are issuing the same command
> > (ie
> > > > > commit
> > > > > > >>> not
> > > > > > >>> > > > abort),
> > > > > > >>> > > > > > we
> > > > > > >>> > > > > > > > >> return
> > > > > > >>> > > > > > > > >> > > success. The logic before KIP-890 seems
> to
> > > > > return
> > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > >>> > > > > > > > >> > > in this case. Are we intentionally
> making
> > > this
> > > > > > >>> change?
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> > > 112. We already merged the code that
> adds
> > > the
> > > > > > >>> VerifyOnly
> > > > > > >>> > > > field
> > > > > > >>> > > > > > in
> > > > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which is an
> > inter
> > > > > > broker
> > > > > > >>> > > request.
> > > > > > >>> > > > > It
> > > > > > >>> > > > > > > > seems
> > > > > > >>> > > > > > > > >> > that
> > > > > > >>> > > > > > > > >> > > we didn't bump up the IBP for that. Do
> you
> > > > know
> > > > > > why?
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> > > Jun
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM Justine
> > > Olshan
> > > > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > > > >>> > > > > > > > >> > > >
> > > > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen" to
> > "current
> > > > > > >>> producer id
> > > > > > >>> > > and
> > > > > > >>> > > > > > > epoch"
> > > > > > >>> > > > > > > > if
> > > > > > >>> > > > > > > > >> > that
> > > > > > >>> > > > > > > > >> > > > was the part that was confusing
> > > > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > > > >>> > > > > > > > >> > > > 112 We still need it. But I am still
> > > > > finalizing
> > > > > > >>> the
> > > > > > >>> > > > design.
> > > > > > >>> > > > > I
> > > > > > >>> > > > > > > will
> > > > > > >>> > > > > > > > >> > update
> > > > > > >>> > > > > > > > >> > > > the KIP once I get the information
> > > > finalized.
> > > > > > >>> Sorry
> > > > > > >>> > for
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > > delays.
> > > > > > >>> > > > > > > > >> > > >
> > > > > > >>> > > > > > > > >> > > > Justine
> > > > > > >>> > > > > > > > >> > > >
> > > > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at 10:50 AM Jun
> Rao
> > > > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > wrote:
> > > > > > >>> > > > > > > > >> > > >
> > > > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow case, the
> > > > previous
> > > > > > ID
> > > > > > >>> is
> > > > > > >>> > the
> > > > > > >>> > > > > same
> > > > > > >>> > > > > > as
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > produce
> > > > > > >>> > > > > > > > >> > > > > ID for the complete marker too, but
> we
> > > set
> > > > > the
> > > > > > >>> > > previous
> > > > > > >>> > > > ID
> > > > > > >>> > > > > > in
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > complete
> > > > > > >>> > > > > > > > >> > > > > marker. Earlier you mentioned that
> > this
> > > is
> > > > > to
> > > > > > >>> know
> > > > > > >>> > > that
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> marker is
> > > > > > >>> > > > > > > > >> > > > > written by the new client so that we
> > > could
> > > > > > >>> return
> > > > > > >>> > > > success
> > > > > > >>> > > > > on
> > > > > > >>> > > > > > > > >> retried
> > > > > > >>> > > > > > > > >> > > > > endMarker requests. I was trying to
> > > > > understand
> > > > > > >>> why
> > > > > > >>> > > this
> > > > > > >>> > > > is
> > > > > > >>> > > > > > not
> > > > > > >>> > > > > > > > >> needed
> > > > > > >>> > > > > > > > >> > > for
> > > > > > >>> > > > > > > > >> > > > > the prepare marker since retry can
> > > happen
> > > > in
> > > > > > the
> > > > > > >>> > > prepare
> > > > > > >>> > > > > > state
> > > > > > >>> > > > > > > > >> too.
> > > > > > >>> > > > > > > > >> > Is
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > reason that in the prepare state, we
> > > > return
> > > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > > >>> > > > > > > > >> > > > instead
> > > > > > >>> > > > > > > > >> > > > > of success on retried endMaker
> > requests?
> > > > If
> > > > > > so,
> > > > > > >>> > should
> > > > > > >>> > > > we
> > > > > > >>> > > > > > > change
> > > > > > >>> > > > > > > > >> "If
> > > > > > >>> > > > > > > > >> > we
> > > > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID in last
> > > seen
> > > > > > >>> fields and
> > > > > > >>> > > are
> > > > > > >>> > > > > > > issuing
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > same
> > > > > > >>> > > > > > > > >> > > > > command (ie commit not abort) we can
> > > > return
> > > > > > >>> (with
> > > > > > >>> > the
> > > > > > >>> > > > new
> > > > > > >>> > > > > > > > epoch)"
> > > > > > >>> > > > > > > > >> > > > > accordingly?
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP, a
> delayed
> > > > > endMaker
> > > > > > >>> > request
> > > > > > >>> > > > > > carries
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > same
> > > > > > >>> > > > > > > > >> > > > > epoch and won't be fenced. This can
> > > > > > >>> commit/abort a
> > > > > > >>> > > > future
> > > > > > >>> > > > > > > > >> transaction
> > > > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure if we
> have
> > > > seen
> > > > > > >>> this in
> > > > > > >>> > > > > practice
> > > > > > >>> > > > > > > > >> though.
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would be useful
> > to
> > > > make
> > > > > > it
> > > > > > >>> > clear
> > > > > > >>> > > > that
> > > > > > >>> > > > > > we
> > > > > > >>> > > > > > > > can
> > > > > > >>> > > > > > > > >> now
> > > > > > >>> > > > > > > > >> > > > > populate the lastSeen field from the
> > log
> > > > > > >>> reliably.
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > >>> > > > > since
> > > > > > >>> > > > > > > > it's
> > > > > > >>> > > > > > > > >> > > called
> > > > > > >>> > > > > > > > >> > > > > across brokers and we are changing
> its
> > > > > schema.
> > > > > > >>> Are
> > > > > > >>> > you
> > > > > > >>> > > > > > saying
> > > > > > >>> > > > > > > we
> > > > > > >>> > > > > > > > >> > don't
> > > > > > >>> > > > > > > > >> > > > need
> > > > > > >>> > > > > > > > >> > > > > it any more? I thought that we
> already
> > > > > > >>> implemented
> > > > > > >>> > the
> > > > > > >>> > > > > > server
> > > > > > >>> > > > > > > > side
> > > > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > > > >>> > AddPartitionsToTxnRequest
> > > > > > >>> > > > > across
> > > > > > >>> > > > > > > > >> brokers.
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > Jun
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at 5:05 PM
> > Justine
> > > > > Olshan
> > > > > > >>> > > > > > > > >> > > > > <jo...@confluent.io.invalid>
> > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the previous ID
> > in
> > > > the
> > > > > > >>> Prepare
> > > > > > >>> > > > field
> > > > > > >>> > > > > > > since
> > > > > > >>> > > > > > > > we
> > > > > > >>> > > > > > > > >> > > don't
> > > > > > >>> > > > > > > > >> > > > > need
> > > > > > >>> > > > > > > > >> > > > > > it. It is the same producer ID as
> > the
> > > > main
> > > > > > >>> > producer
> > > > > > >>> > > ID
> > > > > > >>> > > > > > > field.
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to reread
> > your
> > > > > > message
> > > > > > >>> > about
> > > > > > >>> > > > > > delayed
> > > > > > >>> > > > > > > > >> > markers.
> > > > > > >>> > > > > > > > >> > > > If
> > > > > > >>> > > > > > > > >> > > > > we
> > > > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn marker
> > after
> > > > the
> > > > > > >>> > > transaction
> > > > > > >>> > > > is
> > > > > > >>> > > > > > > > already
> > > > > > >>> > > > > > > > >> > > > > complete?
> > > > > > >>> > > > > > > > >> > > > > > So we will commit the next
> > transaction
> > > > > early
> > > > > > >>> > without
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > fixes
> > > > > > >>> > > > > > > > >> in
> > > > > > >>> > > > > > > > >> > > part
> > > > > > >>> > > > > > > > >> > > > 2?
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > 111 Yes -- this terminology was
> used
> > > in
> > > > a
> > > > > > >>> previous
> > > > > > >>> > > KIP
> > > > > > >>> > > > > and
> > > > > > >>> > > > > > > > never
> > > > > > >>> > > > > > > > >> > > > > > implemented it in the log -- only
> in
> > > > > memory
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which interbroker
> > protocol
> > > > are
> > > > > > you
> > > > > > >>> > > > referring
> > > > > > >>> > > > > > to?
> > > > > > >>> > > > > > > I
> > > > > > >>> > > > > > > > am
> > > > > > >>> > > > > > > > >> > > > working
> > > > > > >>> > > > > > > > >> > > > > on
> > > > > > >>> > > > > > > > >> > > > > > the design for the work to remove
> > the
> > > > > extra
> > > > > > >>> add
> > > > > > >>> > > > > partitions
> > > > > > >>> > > > > > > > call
> > > > > > >>> > > > > > > > >> > and I
> > > > > > >>> > > > > > > > >> > > > > right
> > > > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I have
> yet
> > to
> > > > > > update
> > > > > > >>> that
> > > > > > >>> > > > > section
> > > > > > >>> > > > > > > as
> > > > > > >>> > > > > > > > I
> > > > > > >>> > > > > > > > >> > > > finalize
> > > > > > >>> > > > > > > > >> > > > > > the design so please stay tuned.
> Was
> > > > there
> > > > > > >>> > anything
> > > > > > >>> > > > else
> > > > > > >>> > > > > > you
> > > > > > >>> > > > > > > > >> > thought
> > > > > > >>> > > > > > > > >> > > > > needed
> > > > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > Justine
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at 3:07 PM
> Jun
> > > Rao
> > > > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > I don't see this create any
> issue.
> > > It
> > > > > just
> > > > > > >>> makes
> > > > > > >>> > > it
> > > > > > >>> > > > a
> > > > > > >>> > > > > > bit
> > > > > > >>> > > > > > > > >> hard to
> > > > > > >>> > > > > > > > >> > > > > explain
> > > > > > >>> > > > > > > > >> > > > > > > what this non-tagged produce id
> > > field
> > > > > > >>> means. We
> > > > > > >>> > > are
> > > > > > >>> > > > > > > > >> essentially
> > > > > > >>> > > > > > > > >> > > > trying
> > > > > > >>> > > > > > > > >> > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > combine two actions (completing
> a
> > > txn
> > > > > and
> > > > > > >>> init a
> > > > > > >>> > > new
> > > > > > >>> > > > > > > produce
> > > > > > >>> > > > > > > > >> Id)
> > > > > > >>> > > > > > > > >> > > in a
> > > > > > >>> > > > > > > > >> > > > > > > single record. But, this may be
> > fine
> > > > > too.
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > A few other follow up comments.
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason that we
> > > only
> > > > > set
> > > > > > >>> the
> > > > > > >>> > > > previous
> > > > > > >>> > > > > > > > >> produce id
> > > > > > >>> > > > > > > > >> > > > > tagged
> > > > > > >>> > > > > > > > >> > > > > > > field in the complete marker,
> but
> > > not
> > > > in
> > > > > > the
> > > > > > >>> > > prepare
> > > > > > >>> > > > > > > marker,
> > > > > > >>> > > > > > > > >> is
> > > > > > >>> > > > > > > > >> > > that
> > > > > > >>> > > > > > > > >> > > > in
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > prepare state, we always return
> > > > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > > > >>> > > > > > on
> > > > > > >>> > > > > > > > >> > retried
> > > > > > >>> > > > > > > > >> > > > > > endMaker
> > > > > > >>> > > > > > > > >> > > > > > > requests?
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > 110. "I believe your second
> point
> > is
> > > > > > >>> mentioned
> > > > > > >>> > in
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > KIP. I
> > > > > > >>> > > > > > > > >> can
> > > > > > >>> > > > > > > > >> > > add
> > > > > > >>> > > > > > > > >> > > > > more
> > > > > > >>> > > > > > > > >> > > > > > > text on
> > > > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > > > >>> > > > > > > > >> > > > > > > > The delayed message case can
> > also
> > > > > > violate
> > > > > > >>> EOS
> > > > > > >>> > if
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > > delayed
> > > > > > >>> > > > > > > > >> > > > message
> > > > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > > > addPartitionsToTxn
> > > > > > >>> > request
> > > > > > >>> > > > > comes
> > > > > > >>> > > > > > > in.
> > > > > > >>> > > > > > > > >> > > > > Effectively
> > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > may see a message from a
> previous
> > > > > > (aborted)
> > > > > > >>> > > > > transaction
> > > > > > >>> > > > > > > > become
> > > > > > >>> > > > > > > > >> > part
> > > > > > >>> > > > > > > > >> > > > of
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > The above is the case when a
> > delayed
> > > > > > >>> message is
> > > > > > >>> > > > > appended
> > > > > > >>> > > > > > > to
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > data
> > > > > > >>> > > > > > > > >> > > > > > > partition. What I mentioned is a
> > > > > slightly
> > > > > > >>> > > different
> > > > > > >>> > > > > case
> > > > > > >>> > > > > > > > when
> > > > > > >>> > > > > > > > >> a
> > > > > > >>> > > > > > > > >> > > > delayed
> > > > > > >>> > > > > > > > >> > > > > > > marker is appended to the
> > > transaction
> > > > > log
> > > > > > >>> > > partition.
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once we move
> > past
> > > > the
> > > > > > >>> Prepare
> > > > > > >>> > > and
> > > > > > >>> > > > > > > > Complete
> > > > > > >>> > > > > > > > >> > > states,
> > > > > > >>> > > > > > > > >> > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > don’t need to worry about
> lastSeen
> > > > > fields
> > > > > > >>> and
> > > > > > >>> > > clear
> > > > > > >>> > > > > > them,
> > > > > > >>> > > > > > > > just
> > > > > > >>> > > > > > > > >> > > handle
> > > > > > >>> > > > > > > > >> > > > > > state
> > > > > > >>> > > > > > > > >> > > > > > > transitions as normal.". Is the
> > > > lastSeen
> > > > > > >>> field
> > > > > > >>> > the
> > > > > > >>> > > > > same
> > > > > > >>> > > > > > as
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > previous
> > > > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field in
> > > > > > >>> TransactionLogValue?
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > 112. Since the kip changes the
> > > > > > inter-broker
> > > > > > >>> > > > protocol,
> > > > > > >>> > > > > > > should
> > > > > > >>> > > > > > > > >> we
> > > > > > >>> > > > > > > > >> > > bump
> > > > > > >>> > > > > > > > >> > > > up
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this feature
> > only
> > > > for
> > > > > > the
> > > > > > >>> > KRaft
> > > > > > >>> > > > > mode?
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > Jun
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at 11:13 AM
> > > > Justine
> > > > > > >>> Olshan
> > > > > > >>> > > > > > > > >> > > > > > > <jo...@confluent.io.invalid>
> > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > I'm glad we are getting to
> > > > convergence
> > > > > > on
> > > > > > >>> the
> > > > > > >>> > > > > design.
> > > > > > >>> > > > > > :)
> > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > While I understand it seems a
> > > little
> > > > > > >>> "weird".
> > > > > > >>> > > I'm
> > > > > > >>> > > > > not
> > > > > > >>> > > > > > > sure
> > > > > > >>> > > > > > > > >> what
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > > benefit
> > > > > > >>> > > > > > > > >> > > > > > > > of writing an extra record to
> > the
> > > > log.
> > > > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool to
> > describe
> > > > > > >>> transactions
> > > > > > >>> > > > won't
> > > > > > >>> > > > > > > work
> > > > > > >>> > > > > > > > >> (ie,
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > > complete
> > > > > > >>> > > > > > > > >> > > > > > > > state is needed to calculate
> the
> > > > time
> > > > > > >>> since
> > > > > > >>> > the
> > > > > > >>> > > > > > > > transaction
> > > > > > >>> > > > > > > > >> > > > > completed?)
> > > > > > >>> > > > > > > > >> > > > > > > > If we have a reason like this,
> > it
> > > is
> > > > > > >>> enough to
> > > > > > >>> > > > > > convince
> > > > > > >>> > > > > > > me
> > > > > > >>> > > > > > > > >> we
> > > > > > >>> > > > > > > > >> > > need
> > > > > > >>> > > > > > > > >> > > > > such
> > > > > > >>> > > > > > > > >> > > > > > > an
> > > > > > >>> > > > > > > > >> > > > > > > > extra record. It seems like it
> > > would
> > > > > be
> > > > > > >>> > > replacing
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > record
> > > > > > >>> > > > > > > > >> > > > written
> > > > > > >>> > > > > > > > >> > > > > on
> > > > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is this
> correct?
> > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024 at
> 5:14 PM
> > > Jun
> > > > > Rao
> > > > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > > > >>> > > > > > > > >> > > >
> > > > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > Thanks for the explanation.
> I
> > > > > > >>> understand the
> > > > > > >>> > > > > > intention
> > > > > > >>> > > > > > > > >> now.
> > > > > > >>> > > > > > > > >> > In
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > > > >>> > > > > > > > >> > > > > > > > > case, we set the non-tagged
> > > field
> > > > to
> > > > > > >>> the old
> > > > > > >>> > > pid
> > > > > > >>> > > > > > (and
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> max
> > > > > > >>> > > > > > > > >> > > > > epoch)
> > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > prepare marker so that we
> > could
> > > > > > >>> correctly
> > > > > > >>> > > write
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> marker to
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > data
> > > > > > >>> > > > > > > > >> > > > > > > > > partition if the broker
> > > > downgrades.
> > > > > > When
> > > > > > >>> > > writing
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> complete
> > > > > > >>> > > > > > > > >> > > > > marker,
> > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > know the marker has already
> > been
> > > > > > >>> written to
> > > > > > >>> > > the
> > > > > > >>> > > > > data
> > > > > > >>> > > > > > > > >> > partition.
> > > > > > >>> > > > > > > > >> > > > We
> > > > > > >>> > > > > > > > >> > > > > > set
> > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to the new
> > pid
> > > to
> > > > > > avoid
> > > > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > client if the broker
> > downgrades.
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > The above seems to work.
> It's
> > > > just a
> > > > > > bit
> > > > > > >>> > > > > > inconsistent
> > > > > > >>> > > > > > > > for
> > > > > > >>> > > > > > > > >> a
> > > > > > >>> > > > > > > > >> > > > prepare
> > > > > > >>> > > > > > > > >> > > > > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > > > and a complete marker to use
> > > > > different
> > > > > > >>> pids
> > > > > > >>> > in
> > > > > > >>> > > > > this
> > > > > > >>> > > > > > > > >> special
> > > > > > >>> > > > > > > > >> > > case.
> > > > > > >>> > > > > > > > >> > > > > If
> > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > downgrade with the complete
> > > > marker,
> > > > > it
> > > > > > >>> seems
> > > > > > >>> > > > that
> > > > > > >>> > > > > we
> > > > > > >>> > > > > > > > will
> > > > > > >>> > > > > > > > >> > never
> > > > > > >>> > > > > > > > >> > > > be
> > > > > > >>> > > > > > > > >> > > > > > able
> > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > write the complete marker
> with
> > > the
> > > > > old
> > > > > > >>> pid.
> > > > > > >>> > > Not
> > > > > > >>> > > > > sure
> > > > > > >>> > > > > > > if
> > > > > > >>> > > > > > > > it
> > > > > > >>> > > > > > > > >> > > causes
> > > > > > >>> > > > > > > > >> > > > > any
> > > > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems a bit
> > weird.
> > > > > > >>> Instead of
> > > > > > >>> > > > > writing
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> > > complete
> > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > > > with the new pid, could we
> > write
> > > > two
> > > > > > >>> > records:
> > > > > > >>> > > a
> > > > > > >>> > > > > > > complete
> > > > > > >>> > > > > > > > >> > marker
> > > > > > >>> > > > > > > > >> > > > > with
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > old pid followed by a
> > > > > > >>> TransactionLogValue
> > > > > > >>> > with
> > > > > > >>> > > > the
> > > > > > >>> > > > > > new
> > > > > > >>> > > > > > > > pid
> > > > > > >>> > > > > > > > >> > and
> > > > > > >>> > > > > > > > >> > > an
> > > > > > >>> > > > > > > > >> > > > > > empty
> > > > > > >>> > > > > > > > >> > > > > > > > > state? We could make the two
> > > > records
> > > > > > in
> > > > > > >>> the
> > > > > > >>> > > same
> > > > > > >>> > > > > > batch
> > > > > > >>> > > > > > > > so
> > > > > > >>> > > > > > > > >> > that
> > > > > > >>> > > > > > > > >> > > > they
> > > > > > >>> > > > > > > > >> > > > > > > will
> > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > >>> > > > > > > > >> > > > > > > > > added to the log atomically.
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024 at
> > 5:40 PM
> > > > > > Justine
> > > > > > >>> > Olshan
> > > > > > >>> > > > > > > > >> > > > > > > > >
> <jolshan@confluent.io.invalid
> > >
> > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare marker is
> > > > written,
> > > > > > >>> but the
> > > > > > >>> > > > > endTxn
> > > > > > >>> > > > > > > > >> response
> > > > > > >>> > > > > > > > >> > is
> > > > > > >>> > > > > > > > >> > > > not
> > > > > > >>> > > > > > > > >> > > > > > > > > received
> > > > > > >>> > > > > > > > >> > > > > > > > > > by the client when the
> > server
> > > > > > >>> downgrades
> > > > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare marker is
> > > > > written,
> > > > > > >>> the
> > > > > > >>> > > endTxn
> > > > > > >>> > > > > > > > response
> > > > > > >>> > > > > > > > >> is
> > > > > > >>> > > > > > > > >> > > > > received
> > > > > > >>> > > > > > > > >> > > > > > > by
> > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > client when the server
> > > > downgrades.
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > I think I am still a
> little
> > > > > > confused.
> > > > > > >>> In
> > > > > > >>> > > both
> > > > > > >>> > > > of
> > > > > > >>> > > > > > > these
> > > > > > >>> > > > > > > > >> > cases,
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > transaction log has the
> old
> > > > > producer
> > > > > > >>> ID.
> > > > > > >>> > We
> > > > > > >>> > > > > don't
> > > > > > >>> > > > > > > > write
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > new
> > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > in the prepare marker's
> non
> > > > tagged
> > > > > > >>> fields.
> > > > > > >>> > > > > > > > >> > > > > > > > > > If the server downgrades
> > now,
> > > it
> > > > > > would
> > > > > > >>> > read
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > > records
> > > > > > >>> > > > > > > > >> not
> > > > > > >>> > > > > > > > >> > > in
> > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > >>> > > > > > > > >> > > > > > > > > > fields and the complete
> > marker
> > > > > will
> > > > > > >>> also
> > > > > > >>> > > have
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > old
> > > > > > >>> > > > > > > > >> > > producer
> > > > > > >>> > > > > > > > >> > > > > ID.
> > > > > > >>> > > > > > > > >> > > > > > > > > > (If we had used the new
> > > producer
> > > > > ID,
> > > > > > >>> we
> > > > > > >>> > > would
> > > > > > >>> > > > > not
> > > > > > >>> > > > > > > have
> > > > > > >>> > > > > > > > >> > > > > > transactional
> > > > > > >>> > > > > > > > >> > > > > > > > > > correctness since the
> > producer
> > > > id
> > > > > > >>> doesn't
> > > > > > >>> > > > match
> > > > > > >>> > > > > > the
> > > > > > >>> > > > > > > > >> > > transaction
> > > > > > >>> > > > > > > > >> > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > state would not be correct
> > on
> > > > the
> > > > > > data
> > > > > > >>> > > > > partition.)
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > In the overflow case, I'd
> > > expect
> > > > > the
> > > > > > >>> > > following
> > > > > > >>> > > > > to
> > > > > > >>> > > > > > > > >> happen on
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > > client
> > > > > > >>> > > > > > > > >> > > > > > > > > side
> > > > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we retry EndTxn
> > --
> > > it
> > > > > is
> > > > > > >>> the
> > > > > > >>> > same
> > > > > > >>> > > > > > > producer
> > > > > > >>> > > > > > > > ID
> > > > > > >>> > > > > > > > >> > and
> > > > > > >>> > > > > > > > >> > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > -
> > > > > > >>> > > > > > > > >> > > > > > > 1
> > > > > > >>> > > > > > > > >> > > > > > > > > this
> > > > > > >>> > > > > > > > >> > > > > > > > > > would fence the producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't retry
> > > EndTxn
> > > > > and
> > > > > > >>> use
> > > > > > >>> > the
> > > > > > >>> > > > new
> > > > > > >>> > > > > > > > >> producer id
> > > > > > >>> > > > > > > > >> > > > which
> > > > > > >>> > > > > > > > >> > > > > > > would
> > > > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > > > InvalidPidMappingException
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have special
> > > > handling
> > > > > > for
> > > > > > >>> > when
> > > > > > >>> > > a
> > > > > > >>> > > > > > server
> > > > > > >>> > > > > > > > >> > > > downgrades.
> > > > > > >>> > > > > > > > >> > > > > > When
> > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > >>> > > > > > > > >> > > > > > > > > > reconnects we could get an
> > API
> > > > > > version
> > > > > > >>> > > request
> > > > > > >>> > > > > > > showing
> > > > > > >>> > > > > > > > >> > > KIP-890
> > > > > > >>> > > > > > > > >> > > > > > part 2
> > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > not supported. In that
> case,
> > > we
> > > > > can
> > > > > > >>> call
> > > > > > >>> > > > > > > > initProducerId
> > > > > > >>> > > > > > > > >> to
> > > > > > >>> > > > > > > > >> > > > abort
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In the
> > overflow
> > > > > case,
> > > > > > >>> this
> > > > > > >>> > > > > correctly
> > > > > > >>> > > > > > > > gives
> > > > > > >>> > > > > > > > >> > us a
> > > > > > >>> > > > > > > > >> > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > I guess the corresponding
> > case
> > > > > would
> > > > > > >>> be
> > > > > > >>> > > where
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> *complete
> > > > > > >>> > > > > > > > >> > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > *is
> > > > > > >>> > > > > > > > >> > > > > > > > > > written but the endTxn is
> > not
> > > > > > >>> received by
> > > > > > >>> > > the
> > > > > > >>> > > > > > client
> > > > > > >>> > > > > > > > and
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > > server
> > > > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This would
> > result
> > > in
> > > > > the
> > > > > > >>> > > > transaction
> > > > > > >>> > > > > > > > >> > coordinator
> > > > > > >>> > > > > > > > >> > > > > having
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old one.
> If
> > > the
> > > > > > client
> > > > > > >>> > > > retries,
> > > > > > >>> > > > > it
> > > > > > >>> > > > > > > > will
> > > > > > >>> > > > > > > > >> > > receive
> > > > > > >>> > > > > > > > >> > > > > an
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> InvalidPidMappingException.
> > > The
> > > > > > >>> > > InitProducerId
> > > > > > >>> > > > > > > > scenario
> > > > > > >>> > > > > > > > >> > above
> > > > > > >>> > > > > > > > >> > > > > would
> > > > > > >>> > > > > > > > >> > > > > > > > help
> > > > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my
> > compatibility
> > > > > story
> > > > > > is
> > > > > > >>> > meant
> > > > > > >>> > > > to
> > > > > > >>> > > > > > > > support
> > > > > > >>> > > > > > > > >> > > > > downgrades
> > > > > > >>> > > > > > > > >> > > > > > > > server
> > > > > > >>> > > > > > > > >> > > > > > > > > > side in keeping the
> > > > transactional
> > > > > > >>> > > correctness.
> > > > > > >>> > > > > > > Keeping
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > > client
> > > > > > >>> > > > > > > > >> > > > > > > from
> > > > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is not the
> > > > > priority.
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I can
> also
> > > add
> > > > > text
> > > > > > >>> in
> > > > > > >>> > the
> > > > > > >>> > > > KIP
> > > > > > >>> > > > > > > about
> > > > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > > > >>> > > > > > > > >> > > > > > > if
> > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > think that fixes some edge
> > > > cases.
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12, 2024 at
> > > 4:10 PM
> > > > > Jun
> > > > > > >>> Rao
> > > > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the reply.
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we don't
> need
> > > to
> > > > > > >>> optimize
> > > > > > >>> > for
> > > > > > >>> > > > > > fencing
> > > > > > >>> > > > > > > > >> during
> > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > Regarding consistency,
> > there
> > > > are
> > > > > > two
> > > > > > >>> > > > possible
> > > > > > >>> > > > > > > cases:
> > > > > > >>> > > > > > > > >> (1)
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > > prepare
> > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > > > > > is written, but the
> endTxn
> > > > > > response
> > > > > > >>> is
> > > > > > >>> > not
> > > > > > >>> > > > > > > received
> > > > > > >>> > > > > > > > by
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > > client
> > > > > > >>> > > > > > > > >> > > > > > > > when
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > server downgrades; (2)
> > the
> > > > > > prepare
> > > > > > >>> > marker
> > > > > > >>> > > > is
> > > > > > >>> > > > > > > > written,
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > > endTxn
> > > > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > > > >>> > > > > > > > >> > > > > > > > > > > is received by the
> client
> > > when
> > > > > the
> > > > > > >>> > server
> > > > > > >>> > > > > > > > downgrades.
> > > > > > >>> > > > > > > > >> In
> > > > > > >>> > > > > > > > >> > > (1),
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > client
> > > > > > >>> > > > > > > > >> > > > > > > > > > > will have the old
> produce
> > Id
> > > > and
> > > > > > in
> > > > > > >>> (2),
> > > > > > >>> > > the
> > > > > > >>> > > > > > > client
> > > > > > >>> > > > > > > > >> will
> > > > > > >>> > > > > > > > >> > > have
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If we
> > downgrade
> > > > > right
> > > > > > >>> after
> > > > > > >>> > > the
> > > > > > >>> > > > > > > prepare
> > > > > > >>> > > > > > > > >> > marker,
> > > > > > >>> > > > > > > > >> > > > we
> > > > > > >>> > > > > > > > >> > > > > > > can't
> > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > >>> > > > > > > > >> > > > > > > > > > > consistent to both (1)
> and
> > > (2)
> > > > > > >>> since we
> > > > > > >>> > > can
> > > > > > >>> > > > > only
> > > > > > >>> > > > > > > put
> > > > > > >>> > > > > > > > >> one
> > > > > > >>> > > > > > > > >> > > > value
> > > > > > >>> > > > > > > > >> > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > existing produce Id
> field.
> > > > It's
> > > > > > >>> also not
> > > > > > >>> > > > clear
> > > > > > >>> > > > > > > which
> > > > > > >>> > > > > > > > >> case
> > > > > > >>> > > > > > > > >> > > is
> > > > > > >>> > > > > > > > >> > > > > more
> > > > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > So we could probably be
> > > > > consistent
> > > > > > >>> with
> > > > > > >>> > > > either
> > > > > > >>> > > > > > > case.
> > > > > > >>> > > > > > > > >> By
> > > > > > >>> > > > > > > > >> > > > putting
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in the
> prepare
> > > > > marker,
> > > > > > >>> we
> > > > > > >>> > are
> > > > > > >>> > > > > > > consistent
> > > > > > >>> > > > > > > > >> with
> > > > > > >>> > > > > > > > >> > > > case
> > > > > > >>> > > > > > > > >> > > > > > (2)
> > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > it
> > > > > > >>> > > > > > > > >> > > > > > > > > > > also has the slight
> > benefit
> > > > that
> > > > > > the
> > > > > > >>> > > produce
> > > > > > >>> > > > > > field
> > > > > > >>> > > > > > > > in
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > > > prepare
> > > > > > >>> > > > > > > > >> > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > complete marker are
> > > consistent
> > > > > in
> > > > > > >>> the
> > > > > > >>> > > > overflow
> > > > > > >>> > > > > > > case.
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12, 2024 at
> > > > 3:11 PM
> > > > > > >>> Justine
> > > > > > >>> > > > Olshan
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > <jolshan@confluent.io.invalid
> > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the case you
> > describe,
> > > we
> > > > > > would
> > > > > > >>> > need
> > > > > > >>> > > to
> > > > > > >>> > > > > > have
> > > > > > >>> > > > > > > a
> > > > > > >>> > > > > > > > >> > delayed
> > > > > > >>> > > > > > > > >> > > > > > > request,
> > > > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > successful EndTxn,
> and a
> > > > > > >>> successful
> > > > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > > > >>> > > > > > > > >> > > and
> > > > > > >>> > > > > > > > >> > > > > then
> > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn request
> > go
> > > > > > through
> > > > > > >>> for
> > > > > > >>> > a
> > > > > > >>> > > > > given
> > > > > > >>> > > > > > > > >> producer.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to figure
> out
> > > if
> > > > it
> > > > > > is
> > > > > > >>> > > possible
> > > > > > >>> > > > > for
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> > client
> > > > > > >>> > > > > > > > >> > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > a previous request is
> > > > delayed
> > > > > > >>> > somewhere.
> > > > > > >>> > > > But
> > > > > > >>> > > > > > > yes,
> > > > > > >>> > > > > > > > in
> > > > > > >>> > > > > > > > >> > this
> > > > > > >>> > > > > > > > >> > > > > case
> > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > >>> > > > > > > > >> > > > > > > > > think
> > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > would fence the
> client.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the overflow
> > case.
> > > > In
> > > > > > the
> > > > > > >>> > > overflow
> > > > > > >>> > > > > > case,
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > producer
> > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are different on
> > the
> > > > > > marker
> > > > > > >>> and
> > > > > > >>> > on
> > > > > > >>> > > > the
> > > > > > >>> > > > > > new
> > > > > > >>> > > > > > > > >> > > > transaction.
> > > > > > >>> > > > > > > > >> > > > > > So
> > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to use the
> > max
> > > > > epoch
> > > > > > >>> but
> > > > > > >>> > the
> > > > > > >>> > > > new
> > > > > > >>> > > > > > > > >> > transaction
> > > > > > >>> > > > > > > > >> > > > > should
> > > > > > >>> > > > > > > > >> > > > > > > > start
> > > > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and epoch 0
> > in
> > > > the
> > > > > > >>> > > > transactional
> > > > > > >>> > > > > > > state.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > In the server
> downgrade
> > > > case,
> > > > > we
> > > > > > >>> want
> > > > > > >>> > to
> > > > > > >>> > > > see
> > > > > > >>> > > > > > the
> > > > > > >>> > > > > > > > >> > producer
> > > > > > >>> > > > > > > > >> > > > ID
> > > > > > >>> > > > > > > > >> > > > > as
> > > > > > >>> > > > > > > > >> > > > > > > > that
> > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > what the client will
> > have.
> > > > If
> > > > > we
> > > > > > >>> > > complete
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> commit,
> > > > > > >>> > > > > > > > >> > and
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > state is reloaded, we
> > need
> > > > the
> > > > > > new
> > > > > > >>> > > > producer
> > > > > > >>> > > > > ID
> > > > > > >>> > > > > > > in
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > state
> > > > > > >>> > > > > > > > >> > > > > so
> > > > > > >>> > > > > > > > >> > > > > > > > there
> > > > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid producer ID
> > > > > mapping.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > The server downgrade
> > cases
> > > > are
> > > > > > >>> > > considering
> > > > > > >>> > > > > > > > >> > transactional
> > > > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing from
> > > previous
> > > > > > >>> behavior
> > > > > > >>> > --
> > > > > > >>> > > > and
> > > > > > >>> > > > > > are
> > > > > > >>> > > > > > > > not
> > > > > > >>> > > > > > > > >> > > > > concerned
> > > > > > >>> > > > > > > > >> > > > > > > > about
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the safety
> > from
> > > > > > fencing
> > > > > > >>> > > retries
> > > > > > >>> > > > > (as
> > > > > > >>> > > > > > > we
> > > > > > >>> > > > > > > > >> have
> > > > > > >>> > > > > > > > >> > > > > > downgraded
> > > > > > >>> > > > > > > > >> > > > > > > > so
> > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to
> support).
> > > > > Perhaps
> > > > > > >>> this
> > > > > > >>> > is
> > > > > > >>> > > a
> > > > > > >>> > > > > > trade
> > > > > > >>> > > > > > > > off,
> > > > > > >>> > > > > > > > >> > but
> > > > > > >>> > > > > > > > >> > > I
> > > > > > >>> > > > > > > > >> > > > > > think
> > > > > > >>> > > > > > > > >> > > > > > > it
> > > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client
> > downgrades,
> > > > it
> > > > > > will
> > > > > > >>> > have
> > > > > > >>> > > > > > > restarted
> > > > > > >>> > > > > > > > >> and
> > > > > > >>> > > > > > > > >> > it
> > > > > > >>> > > > > > > > >> > > is
> > > > > > >>> > > > > > > > >> > > > > ok
> > > > > > >>> > > > > > > > >> > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > have a new producer ID
> > > too).
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan 12, 2024
> at
> > > > > 11:42 AM
> > > > > > >>> Jun
> > > > > > >>> > Rao
> > > > > > >>> > > > > > > > >> > > > > > > <jun@confluent.io.invalid
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for the
> reply.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If the marker
> > is
> > > > > > written
> > > > > > >>> by
> > > > > > >>> > the
> > > > > > >>> > > > new
> > > > > > >>> > > > > > > > >> client, we
> > > > > > >>> > > > > > > > >> > > can
> > > > > > >>> > > > > > > > >> > > > > as
> > > > > > >>> > > > > > > > >> > > > > > I
> > > > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the last email
> > guarantee
> > > > > that
> > > > > > >>> any
> > > > > > >>> > > EndTxn
> > > > > > >>> > > > > > > > requests
> > > > > > >>> > > > > > > > >> > with
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from the same
> producer
> > > and
> > > > > the
> > > > > > >>> same
> > > > > > >>> > > > > > > transaction.
> > > > > > >>> > > > > > > > >> Then
> > > > > > >>> > > > > > > > >> > > we
> > > > > > >>> > > > > > > > >> > > > > > don't
> > > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > return a fenced
> error
> > > but
> > > > > can
> > > > > > >>> handle
> > > > > > >>> > > > > > > gracefully
> > > > > > >>> > > > > > > > as
> > > > > > >>> > > > > > > > >> > > > > described
> > > > > > >>> > > > > > > > >> > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When a delayed
> EndTnx
> > > > > request
> > > > > > is
> > > > > > >>> > > > > processed,
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> txn
> > > > > > >>> > > > > > > > >> > > state
> > > > > > >>> > > > > > > > >> > > > > > could
> > > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next txn. I
> > > guess
> > > > in
> > > > > > >>> this
> > > > > > >>> > case
> > > > > > >>> > > > we
> > > > > > >>> > > > > > > still
> > > > > > >>> > > > > > > > >> > return
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > fenced
> > > > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed request?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry, my
> > question
> > > > was
> > > > > > >>> > > inaccurate.
> > > > > > >>> > > > > What
> > > > > > >>> > > > > > > you
> > > > > > >>> > > > > > > > >> > > > described
> > > > > > >>> > > > > > > > >> > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > "The downgrade
> > > > > compatibility I
> > > > > > >>> > mention
> > > > > > >>> > > > is
> > > > > > >>> > > > > > that
> > > > > > >>> > > > > > > > we
> > > > > > >>> > > > > > > > >> > keep
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > same
> > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in the
> main
> > > > > > >>> (non-tagged)
> > > > > > >>> > > > fields
> > > > > > >>> > > > > as
> > > > > > >>> > > > > > > we
> > > > > > >>> > > > > > > > >> did
> > > > > > >>> > > > > > > > >> > > > before
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > code
> > > > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > server side." If we
> > want
> > > > to
> > > > > do
> > > > > > >>> this,
> > > > > > >>> > > it
> > > > > > >>> > > > > > seems
> > > > > > >>> > > > > > > > >> that we
> > > > > > >>> > > > > > > > >> > > > > should
> > > > > > >>> > > > > > > > >> > > > > > > use
> > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > current produce Id
> and
> > > max
> > > > > > >>> epoch in
> > > > > > >>> > > the
> > > > > > >>> > > > > > > existing
> > > > > > >>> > > > > > > > >> > > > producerId
> > > > > > >>> > > > > > > > >> > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerEpoch fields
> > for
> > > > > both
> > > > > > >>> the
> > > > > > >>> > > > prepare
> > > > > > >>> > > > > > and
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > complete
> > > > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > The downgrade can
> > happen
> > > > > after
> > > > > > >>> the
> > > > > > >>> > > > > complete
> > > > > > >>> > > > > > > > >> marker is
> > > > > > >>> > > > > > > > >> > > > > > written.
> > > > > > >>> > > > > > > > >> > > > > > > > With
> > > > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > you described, the
> > > > > downgraded
> > > > > > >>> > > > coordinator
> > > > > > >>> > > > > > will
> > > > > > >>> > > > > > > > see
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of the old
> > one.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan 12, 2024
> > at
> > > > > > 10:44 AM
> > > > > > >>> > > Justine
> > > > > > >>> > > > > > > Olshan
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > <jolshan@confluent.io.invalid
> > > > > > >
> > > > > > >>> > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can update the
> > > > > > description.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe your
> > second
> > > > > point
> > > > > > is
> > > > > > >>> > > > mentioned
> > > > > > >>> > > > > > in
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > KIP.
> > > > > > >>> > > > > > > > >> > > I
> > > > > > >>> > > > > > > > >> > > > > can
> > > > > > >>> > > > > > > > >> > > > > > > add
> > > > > > >>> > > > > > > > >> > > > > > > > > more
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it is
> > helpful.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The delayed
> > message
> > > > case
> > > > > > can
> > > > > > >>> > also
> > > > > > >>> > > > > > violate
> > > > > > >>> > > > > > > > EOS
> > > > > > >>> > > > > > > > >> if
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > > delayed
> > > > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in after the
> > > next
> > > > > > >>> > > > > addPartitionsToTxn
> > > > > > >>> > > > > > > > >> request
> > > > > > >>> > > > > > > > >> > > > comes
> > > > > > >>> > > > > > > > >> > > > > > in.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a message
> > > from a
> > > > > > >>> previous
> > > > > > >>> > > > > > (aborted)
> > > > > > >>> > > > > > > > >> > > transaction
> > > > > > >>> > > > > > > > >> > > > > > > become
> > > > > > >>> > > > > > > > >> > > > > > > > > part
> > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > next transaction.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the marker is
> > > written
> > > > > by
> > > > > > >>> the
> > > > > > >>> > new
> > > > > > >>> > > > > > client,
> > > > > > >>> > > > > > > we
> > > > > > >>> > > > > > > > >> can
> > > > > > >>> > > > > > > > >> > > as I
> > > > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email
> guarantee
> > > > that
> > > > > > any
> > > > > > >>> > EndTxn
> > > > > > >>> > > > > > > requests
> > > > > > >>> > > > > > > > >> with
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > same
> > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same producer
> > and
> > > > the
> > > > > > same
> > > > > > >>> > > > > > transaction.
> > > > > > >>> > > > > > > > >> Then we
> > > > > > >>> > > > > > > > >> > > > don't
> > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced error but
> can
> > > > > handle
> > > > > > >>> > > gracefully
> > > > > > >>> > > > > as
> > > > > > >>> > > > > > > > >> described
> > > > > > >>> > > > > > > > >> > > in
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't think a
> > > boolean
> > > > is
> > > > > > >>> useful
> > > > > > >>> > > > since
> > > > > > >>> > > > > it
> > > > > > >>> > > > > > > is
> > > > > > >>> > > > > > > > >> > > directly
> > > > > > >>> > > > > > > > >> > > > > > > encoded
> > > > > > >>> > > > > > > > >> > > > > > > > by
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence or lack
> of
> > > the
> > > > > > >>> tagged
> > > > > > >>> > > field
> > > > > > >>> > > > > > being
> > > > > > >>> > > > > > > > >> > written.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the prepare
> > marker
> > > we
> > > > > > will
> > > > > > >>> have
> > > > > > >>> > > the
> > > > > > >>> > > > > > same
> > > > > > >>> > > > > > > > >> > producer
> > > > > > >>> > > > > > > > >> > > ID
> > > > > > >>> > > > > > > > >> > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In the
> > Complete
> > > > > state
> > > > > > >>> we
> > > > > > >>> > may
> > > > > > >>> > > > not.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure why
> the
> > > > > ongoing
> > > > > > >>> state
> > > > > > >>> > > > > matters
> > > > > > >>> > > > > > > for
> > > > > > >>> > > > > > > > >> this
> > > > > > >>> > > > > > > > >> > > > KIP.
> > > > > > >>> > > > > > > > >> > > > > It
> > > > > > >>> > > > > > > > >> > > > > > > > does
> > > > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure what
> > you
> > > > are
> > > > > > >>> > referring
> > > > > > >>> > > to
> > > > > > >>> > > > > > about
> > > > > > >>> > > > > > > > >> > writing
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > previous
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the prepare
> > > > marker.
> > > > > > >>> This is
> > > > > > >>> > > not
> > > > > > >>> > > > in
> > > > > > >>> > > > > > the
> > > > > > >>> > > > > > > > >> KIP.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the overflow
> > case,
> > > we
> > > > > > >>> write the
> > > > > > >>> > > > > > > > >> nextProducerId
> > > > > > >>> > > > > > > > >> > in
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so we know
> > > what
> > > > we
> > > > > > >>> > assigned
> > > > > > >>> > > > when
> > > > > > >>> > > > > > we
> > > > > > >>> > > > > > > > >> reload
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we complete,
> we
> > > > > > >>> transition
> > > > > > >>> > this
> > > > > > >>> > > > ID
> > > > > > >>> > > > > to
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> main
> > > > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the previous
> > > > producer
> > > > > > ID
> > > > > > >>> > field
> > > > > > >>> > > > > filled
> > > > > > >>> > > > > > > in.
> > > > > > >>> > > > > > > > >> This
> > > > > > >>> > > > > > > > >> > > is
> > > > > > >>> > > > > > > > >> > > > so
> > > > > > >>> > > > > > > > >> > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > can
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry case
> the
> > > > > > operation
> > > > > > >>> > > > completed
> > > > > > >>> > > > > > > > >> > successfully
> > > > > > >>> > > > > > > > >> > > > and
> > > > > > >>> > > > > > > > >> > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > don't
> > > > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer. The
> > > downgrade
> > > > > > >>> > > compatibility
> > > > > > >>> > > > I
> > > > > > >>> > > > > > > > mention
> > > > > > >>> > > > > > > > >> is
> > > > > > >>> > > > > > > > >> > > that
> > > > > > >>> > > > > > > > >> > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > keep
> > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer ID and
> > epoch
> > > in
> > > > > the
> > > > > > >>> main
> > > > > > >>> > > > > > > (non-tagged)
> > > > > > >>> > > > > > > > >> > fields
> > > > > > >>> > > > > > > > >> > > > as
> > > > > > >>> > > > > > > > >> > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > did
> > > > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on the server
> > > side.
> > > > > If
> > > > > > >>> the
> > > > > > >>> > > server
> > > > > > >>> > > > > > > > >> downgrades,
> > > > > > >>> > > > > > > > >> > we
> > > > > > >>> > > > > > > > >> > > > are
> > > > > > >>> > > > > > > > >> > > > > > > still
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This addresses
> both
> > > the
> > > > > > >>> prepare
> > > > > > >>> > and
> > > > > > >>> > > > > > complete
> > > > > > >>> > > > > > > > >> state
> > > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri, Jan 12,
> 2024
> > > at
> > > > > > >>> 10:21 AM
> > > > > > >>> > Jun
> > > > > > >>> > > > Rao
> > > > > > >>> > > > > > > > >> > > > > > > > > <jun@confluent.io.invalid
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks for the
> > > reply.
> > > > > > Sorry
> > > > > > >>> for
> > > > > > >>> > > the
> > > > > > >>> > > > > > > delay. I
> > > > > > >>> > > > > > > > >> > have a
> > > > > > >>> > > > > > > > >> > > > few
> > > > > > >>> > > > > > > > >> > > > > > > more
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I think the
> > > > > > motivation
> > > > > > >>> > > section
> > > > > > >>> > > > > > could
> > > > > > >>> > > > > > > be
> > > > > > >>> > > > > > > > >> > > > improved.
> > > > > > >>> > > > > > > > >> > > > > > One
> > > > > > >>> > > > > > > > >> > > > > > > of
> > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > motivations
> listed
> > > by
> > > > > the
> > > > > > >>> KIP is
> > > > > > >>> > > > "This
> > > > > > >>> > > > > > can
> > > > > > >>> > > > > > > > >> happen
> > > > > > >>> > > > > > > > >> > > > when
> > > > > > >>> > > > > > > > >> > > > > a
> > > > > > >>> > > > > > > > >> > > > > > > > > message
> > > > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or delayed due
> to
> > > > > > networking
> > > > > > >>> > > issues
> > > > > > >>> > > > > or a
> > > > > > >>> > > > > > > > >> network
> > > > > > >>> > > > > > > > >> > > > > > partition,
> > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > transaction
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts, and then
> > the
> > > > > > delayed
> > > > > > >>> > > message
> > > > > > >>> > > > > > > finally
> > > > > > >>> > > > > > > > >> > comes
> > > > > > >>> > > > > > > > >> > > > > in.".
> > > > > > >>> > > > > > > > >> > > > > > > This
> > > > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very accurate.
> > > Without
> > > > > > >>> KIP-890,
> > > > > > >>> > > > > > currently,
> > > > > > >>> > > > > > > > if
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts an
> ongoing
> > > > > > >>> transaction,
> > > > > > >>> > it
> > > > > > >>> > > > > > already
> > > > > > >>> > > > > > > > >> bumps
> > > > > > >>> > > > > > > > >> > up
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which prevents
> the
> > > > > delayed
> > > > > > >>> > produce
> > > > > > >>> > > > > > message
> > > > > > >>> > > > > > > > >> from
> > > > > > >>> > > > > > > > >> > > being
> > > > > > >>> > > > > > > > >> > > > > > added
> > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > partition. What
> > can
> > > > > cause
> > > > > > a
> > > > > > >>> > > hanging
> > > > > > >>> > > > > > > > >> transaction
> > > > > > >>> > > > > > > > >> > is
> > > > > > >>> > > > > > > > >> > > > that
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completes
> (either
> > > > aborts
> > > > > > or
> > > > > > >>> > > > commits) a
> > > > > > >>> > > > > > > > >> > transaction
> > > > > > >>> > > > > > > > >> > > > > before
> > > > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > successful ack
> on
> > > > > messages
> > > > > > >>> > > published
> > > > > > >>> > > > > in
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> same
> > > > > > >>> > > > > > > > >> > > txn.
> > > > > > >>> > > > > > > > >> > > > > In
> > > > > > >>> > > > > > > > >> > > > > > > this
> > > > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible for the
> > > > delayed
> > > > > > >>> message
> > > > > > >>> > > to
> > > > > > >>> > > > be
> > > > > > >>> > > > > > > > >> appended
> > > > > > >>> > > > > > > > >> > to
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker, causing
> a
> > > > > > >>> transaction to
> > > > > > >>> > > > hang.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A similar issue
> > (not
> > > > > > >>> mentioned
> > > > > > >>> > in
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > > >> motivation)
> > > > > > >>> > > > > > > > >> > > > could
> > > > > > >>> > > > > > > > >> > > > > > > > happen
> > > > > > >>> > > > > > > > >> > > > > > > > > on
> > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker in the
> > > > > > coordinator's
> > > > > > >>> log.
> > > > > > >>> > > For
> > > > > > >>> > > > > > > > example,
> > > > > > >>> > > > > > > > >> > it's
> > > > > > >>> > > > > > > > >> > > > > > possible
> > > > > > >>> > > > > > > > >> > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest to
> > be
> > > > > > delayed
> > > > > > >>> on
> > > > > > >>> > the
> > > > > > >>> > > > > > > > >> coordinator.
> > > > > > >>> > > > > > > > >> > By
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > time
> > > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest is
> > > > > > processed,
> > > > > > >>> it's
> > > > > > >>> > > > > > possible
> > > > > > >>> > > > > > > > that
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > >>> > > > > > > > >> > > > > > > > txn
> > > > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completed and a
> > new
> > > > txn
> > > > > > has
> > > > > > >>> > > started.
> > > > > > >>> > > > > > > > >> Currently,
> > > > > > >>> > > > > > > > >> > > since
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped on every
> > txn,
> > > > the
> > > > > > >>> delayed
> > > > > > >>> > > > > > > > EndTxnRequest
> > > > > > >>> > > > > > > > >> > will
> > > > > > >>> > > > > > > > >> > > > add
> > > > > > >>> > > > > > > > >> > > > > > an
> > > > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare marker
> > (and
> > > > > > >>> eventually a
> > > > > > >>> > > > > > complete
> > > > > > >>> > > > > > > > >> marker)
> > > > > > >>> > > > > > > > >> > > to
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't cause the
> > > > > > transaction
> > > > > > >>> to
> > > > > > >>> > > hang,
> > > > > > >>> > > > > but
> > > > > > >>> > > > > > > it
> > > > > > >>> > > > > > > > >> will
> > > > > > >>> > > > > > > > >> > > > break
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The proposal in
> > this
> > > > KIP
> > > > > > >>> will
> > > > > > >>> > > > address
> > > > > > >>> > > > > > this
> > > > > > >>> > > > > > > > >> issue
> > > > > > >>> > > > > > > > >> > > too.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101. "However, I
> > was
> > > > > > >>> writing it
> > > > > > >>> > so
> > > > > > >>> > > > > that
> > > > > > >>> > > > > > we
> > > > > > >>> > > > > > > > can
> > > > > > >>> > > > > > > > >> > > > > > distinguish
> > > > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old clients
> where
> > we
> > > > > don't
> > > > > > >>> have
> > > > > > >>> > > the
> > > > > > >>> > > > > > > ability
> > > > > > >>> > > > > > > > do
> > > > > > >>> > > > > > > > >> > this
> > > > > > >>> > > > > > > > >> > > > > > > operation
> > > > > > >>> > > > > > > > >> > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients that
> can.
> > > (Old
> > > > > > >>> clients
> > > > > > >>> > > don't
> > > > > > >>> > > > > > bump
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > epoch
> > > > > > >>> > > > > > > > >> > > > on
> > > > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > > > >>> > > > > > > > >> > > > > > > > > so
> > > > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for sure the
> > > write
> > > > > > >>> belongs
> > > > > > >>> > to
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > given
> > > > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I am
> > wondering
> > > > why
> > > > > > we
> > > > > > >>> need
> > > > > > >>> > > to
> > > > > > >>> > > > > > > > >> distinguish
> > > > > > >>> > > > > > > > >> > > > whether
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written by the
> old
> > > and
> > > > > the
> > > > > > >>> new
> > > > > > >>> > > > client.
> > > > > > >>> > > > > > > Could
> > > > > > >>> > > > > > > > >> you
> > > > > > >>> > > > > > > > >> > > > > describe
> > > > > > >>> > > > > > > > >> > > > > > > > what
> > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > differently if
> we
> > > know
> > > > > the
> > > > > > >>> > marker
> > > > > > >>> > > is
> > > > > > >>> > > > > > > written
> > > > > > >>> > > > > > > > >> by
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > client?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If we do
> > need
> > > a
> > > > > way
> > > > > > to
> > > > > > >>> > > > > distinguish
> > > > > > >>> > > > > > > > >> whether
> > > > > > >>> > > > > > > > >> > > the
> > > > > > >>> > > > > > > > >> > > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old and the
> > new
> > > > > > client.
> > > > > > >>> > Would
> > > > > > >>> > > it
> > > > > > >>> > > > > be
> > > > > > >>> > > > > > > > >> simpler
> > > > > > >>> > > > > > > > >> > to
> > > > > > >>> > > > > > > > >> > > > just
> > > > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field instead of
> > > > > > indirectly
> > > > > > >>> > > through
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> previous
> > > > > > >>> > > > > > > > >> > > > > produce
> > > > > > >>> > > > > > > > >> > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3 It's not
> > clear
> > > > to
> > > > > me
> > > > > > >>> why
> > > > > > >>> > we
> > > > > > >>> > > > only
> > > > > > >>> > > > > > add
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > previous
> > > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the complete
> > marker,
> > > > but
> > > > > > >>> not in
> > > > > > >>> > > the
> > > > > > >>> > > > > > > prepare
> > > > > > >>> > > > > > > > >> > marker.
> > > > > > >>> > > > > > > > >> > > > If
> > > > > > >>> > > > > > > > >> > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > want
> > > > > > >>> > > > > > > > >> > > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether a marker
> > is
> > > > > > written
> > > > > > >>> by
> > > > > > >>> > the
> > > > > > >>> > > > new
> > > > > > >>> > > > > > > > client
> > > > > > >>> > > > > > > > >> or
> > > > > > >>> > > > > > > > >> > > not,
> > > > > > >>> > > > > > > > >> > > > > it
> > > > > > >>> > > > > > > > >> > > > > > > > seems
> > > > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do this
> > > > consistently
> > > > > > for
> > > > > > >>> all
> > > > > > >>> > > > > markers.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4 What about
> > the
> > > > > > >>> > > > > TransactionLogValue
> > > > > > >>> > > > > > > > >> record
> > > > > > >>> > > > > > > > >> > > > > > > representing
> > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state? Should we
> > > also
> > > > > > >>> > distinguish
> > > > > > >>> > > > > > whether
> > > > > > >>> > > > > > > > it's
> > > > > > >>> > > > > > > > >> > > > written
> > > > > > >>> > > > > > > > >> > > > > by
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > old
> > > > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new client?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In the
> > overflow
> > > > > case,
> > > > > > >>> it's
> > > > > > >>> > > > still
> > > > > > >>> > > > > > not
> > > > > > >>> > > > > > > > >> clear
> > > > > > >>> > > > > > > > >> > to
> > > > > > >>> > > > > > > > >> > > me
> > > > > > >>> > > > > > > > >> > > > > why
> > > > > > >>> > > > > > > > >> > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous produce
> > Id
> > > in
> > > > > the
> > > > > > >>> > prepare
> > > > > > >>> > > > > > marker
> > > > > > >>> > > > > > > > >> while
> > > > > > >>> > > > > > > > >> > > > writing
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the complete
> > > > marker.
> > > > > > You
> > > > > > >>> > > > mentioned
> > > > > > >>> > > > > > that
> > > > > > >>> > > > > > > > >> it's
> > > > > > >>> > > > > > > > >> > for
> > > > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could
> downgrade
> > > > with
> > > > > > >>> either
> > > > > > >>> > the
> > > > > > >>> > > > > > prepare
> > > > > > >>> > > > > > > > >> marker
> > > > > > >>> > > > > > > > >> > > or
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In either case,
> > the
> > > > > > >>> downgraded
> > > > > > >>> > > > > > coordinator
> > > > > > >>> > > > > > > > >> should
> > > > > > >>> > > > > > > > >> > > see
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > same
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > (probably the
> > > previous
> > > > > > >>> produce
> > > > > > >>> > > Id),
> > > > > > >>> > > > > > right?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed, Dec 20,
> > 2023
> > > > at
> > > > > > >>> 6:00 PM
> > > > > > >>> > > > > Justine
> > > > > > >>> > > > > > > > Olshan
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> <jo...@confluent.io.invalid>
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey Jun,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks for
> > taking
> > > a
> > > > > look
> > > > > > >>> at
> > > > > > >>> > the
> > > > > > >>> > > > KIP
> > > > > > >>> > > > > > > again.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100. For the
> > epoch
> > > > > > >>> overflow
> > > > > > >>> > > case,
> > > > > > >>> > > > > only
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> > marker
> > > > > > >>> > > > > > > > >> > > > > will
> > > > > > >>> > > > > > > > >> > > > > > > have
> > > > > > >>> > > > > > > > >> > > > > > > > > max
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps the
> > behavior
> > > > of
> > > > > > the
> > > > > > >>> rest
> > > > > > >>> > > of
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> markers
> > > > > > >>> > > > > > > > >> > > where
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > last
> > > > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch of the
> > > > > transaction
> > > > > > >>> > > records +
> > > > > > >>> > > > > 1.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101. You are
> > > correct
> > > > > > that
> > > > > > >>> we
> > > > > > >>> > > don't
> > > > > > >>> > > > > > need
> > > > > > >>> > > > > > > to
> > > > > > >>> > > > > > > > >> > write
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the same.
> > > > However,
> > > > > I
> > > > > > >>> was
> > > > > > >>> > > > writing
> > > > > > >>> > > > > it
> > > > > > >>> > > > > > > so
> > > > > > >>> > > > > > > > >> that
> > > > > > >>> > > > > > > > >> > we
> > > > > > >>> > > > > > > > >> > > > can
> > > > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old clients
> > where
> > > we
> > > > > > don't
> > > > > > >>> > have
> > > > > > >>> > > > the
> > > > > > >>> > > > > > > > ability
> > > > > > >>> > > > > > > > >> do
> > > > > > >>> > > > > > > > >> > > this
> > > > > > >>> > > > > > > > >> > > > > > > > operation
> > > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > clients that
> > can.
> > > > (Old
> > > > > > >>> clients
> > > > > > >>> > > > don't
> > > > > > >>> > > > > > > bump
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > epoch
> > > > > > >>> > > > > > > > >> > > > > on
> > > > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say for sure
> the
> > > > write
> > > > > > >>> belongs
> > > > > > >>> > > to
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > given
> > > > > > >>> > > > > > > > >> > > > > > transaction).
> > > > > > >>> > > > > > > > >> > > > > > > > If
> > > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn request
> > > from
> > > > a
> > > > > > new
> > > > > > >>> > > client,
> > > > > > >>> > > > we
> > > > > > >>> > > > > > > will
> > > > > > >>> > > > > > > > >> fill
> > > > > > >>> > > > > > > > >> > > this
> > > > > > >>> > > > > > > > >> > > > > > > field.
> > > > > > >>> > > > > > > > >> > > > > > > > We
> > > > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > guarantee
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that any
> EndTxn
> > > > > requests
> > > > > > >>> with
> > > > > > >>> > > the
> > > > > > >>> > > > > same
> > > > > > >>> > > > > > > > epoch
> > > > > > >>> > > > > > > > >> > are
> > > > > > >>> > > > > > > > >> > > > from
> > > > > > >>> > > > > > > > >> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > same
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the same
> > > > transaction.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102. In
> prepare
> > > > phase,
> > > > > > we
> > > > > > >>> have
> > > > > > >>> > > the
> > > > > > >>> > > > > > same
> > > > > > >>> > > > > > > > >> > producer
> > > > > > >>> > > > > > > > >> > > ID
> > > > > > >>> > > > > > > > >> > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had. It is the
> > > > > producer
> > > > > > >>> ID and
> > > > > > >>> > > > epoch
> > > > > > >>> > > > > > > that
> > > > > > >>> > > > > > > > >> are
> > > > > > >>> > > > > > > > >> > on
> > > > > > >>> > > > > > > > >> > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > > > >>> > > > > > > > >> > > > > > > > > In
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase, we stay
> > the
> > > > > same
> > > > > > >>> unless
> > > > > > >>> > > it
> > > > > > >>> > > > is
> > > > > > >>> > > > > > the
> > > > > > >>> > > > > > > > >> > overflow
> > > > > > >>> > > > > > > > >> > > > > case.
> > > > > > >>> > > > > > > > >> > > > > > > In
> > > > > > >>> > > > > > > > >> > > > > > > > > that
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set the
> producer
> > > ID
> > > > to
> > > > > > >>> the new
> > > > > > >>> > > one
> > > > > > >>> > > > > we
> > > > > > >>> > > > > > > > >> generated
> > > > > > >>> > > > > > > > >> > > and
> > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > complete. This
> > is
> > > > for
> > > > > > >>> > downgrade
> > > > > > >>> > > > > > > > >> compatibility.
> > > > > > >>> > > > > > > > >> > > The
> > > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety guards
> > for
> > > > > > retries
> > > > > > >>> and
> > > > > > >>> > > > > > failovers.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In prepare
> phase
> > > for
> > > > > > epoch
> > > > > > >>> > > > overflow
> > > > > > >>> > > > > > case
> > > > > > >>> > > > > > > > >> only
> > > > > > >>> > > > > > > > >> > we
> > > > > > >>> > > > > > > > >> > > > > store
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID. This is
> for
> > > the
> > > > > case
> > > > > > >>> where
> > > > > > >>> > > we
> > > > > > >>> > > > > > reload
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > transaction
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > prepare state.
> > > Once
> > > > > the
> > > > > > >>> > > > transaction
> > > > > > >>> > > > > is
> > > > > > >>> > > > > > > > >> > committed,
> > > > > > >>> > > > > > > > >> > > > we
> > > > > > >>> > > > > > > > >> > > > > > can
> > > > > > >>> > > > > > > > >> > > > > > > > use
> > > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the client
> > > > already
> > > > > is
> > > > > > >>> > using.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In commit
> phase,
> > > we
> > > > > > store
> > > > > > >>> the
> > > > > > >>> > > > > previous
> > > > > > >>> > > > > > > > >> producer
> > > > > > >>> > > > > > > > >> > > ID
> > > > > > >>> > > > > > > > >> > > > in
> > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > >>> > > > > > > > >> > > > > > > > of
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I think it is
> > > easier
> > > > > to
> > > > > > >>> think
> > > > > > >>> > of
> > > > > > >>> > > > it
> > > > > > >>> > > > > as
> > > > > > >>> > > > > > > > just
> > > > > > >>> > > > > > > > >> how
> > > > > > >>> > > > > > > > >> > > we
> > > > > > >>> > > > > > > > >> > > > > were
> > > > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and epoch
> > before,
> > > > with
> > > > > > >>> some
> > > > > > >>> > > extra
> > > > > > >>> > > > > > > > bookeeping
> > > > > > >>> > > > > > > > >> > and
> > > > > > >>> > > > > > > > >> > > > edge
> > > > > > >>> > > > > > > > >> > > > > > > case
> > > > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged fields.
> > We
> > > > have
> > > > > > to
> > > > > > >>> do
> > > > > > >>> > it
> > > > > > >>> > > > this
> > > > > > >>> > > > > > way
> > > > > > >>> > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > > > >>> > > > > > > > >> > > > > > > > > with
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > downgrades.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103. Next
> > producer
> > > > ID
> > > > > is
> > > > > > >>> for
> > > > > > >>> > > > prepare
> > > > > > >>> > > > > > > > status
> > > > > > >>> > > > > > > > >> and
> > > > > > >>> > > > > > > > >> > > > > > previous
> > > > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after
> complete.
> > > The
> > > > > > >>> reason why
> > > > > > >>> > > we
> > > > > > >>> > > > > need
> > > > > > >>> > > > > > > two
> > > > > > >>> > > > > > > > >> > > separate
> > > > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > backwards
> > > > > compatibility.
> > > > > > >>> We
> > > > > > >>> > need
> > > > > > >>> > > > to
> > > > > > >>> > > > > > keep
> > > > > > >>> > > > > > > > the
> > > > > > >>> > > > > > > > >> > same
> > > > > > >>> > > > > > > > >> > > > > > > semantics
> > > > > > >>> > > > > > > > >> > > > > > > > > for
> > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged
> field
> > > in
> > > > > case
> > > > > > >>> we
> > > > > > >>> > > > > downgrade.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104. We set
> the
> > > > fields
> > > > > > as
> > > > > > >>> we
> > > > > > >>> > do
> > > > > > >>> > > in
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> > > > transactional
> > > > > > >>> > > > > > > > >> > > > > > > state
> > > > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do this for
> > > > > > compatibility
> > > > > > >>> --
> > > > > > >>> > if
> > > > > > >>> > > we
> > > > > > >>> > > > > > > > >> downgrade,
> > > > > > >>> > > > > > > > >> > we
> > > > > > >>> > > > > > > > >> > > > will
> > > > > > >>> > > > > > > > >> > > > > > > only
> > > > > > >>> > > > > > > > >> > > > > > > > > have
> > > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged
> > fields)
> > > > It
> > > > > > >>> will be
> > > > > > >>> > > the
> > > > > > >>> > > > > old
> > > > > > >>> > > > > > > > >> producer
> > > > > > >>> > > > > > > > >> > ID
> > > > > > >>> > > > > > > > >> > > > and
> > > > > > >>> > > > > > > > >> > > > > > max
> > > > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope this
> helps.
> > > Let
> > > > > me
> > > > > > >>> know
> > > > > > >>> > if
> > > > > > >>> > > > you
> > > > > > >>> > > > > > have
> > > > > > >>> > > > > > > > >> > further
> > > > > > >>> > > > > > > > >> > > > > > > questions.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Justine
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On Wed, Dec
> 20,
> > > 2023
> > > > > at
> > > > > > >>> > 3:33 PM
> > > > > > >>> > > > Jun
> > > > > > >>> > > > > > Rao
> > > > > > >>> > > > > > > > >> > > > > > > > > > <jun@confluent.io.invalid
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi, Justine,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It seems
> that
> > > you
> > > > > have
> > > > > > >>> made
> > > > > > >>> > > some
> > > > > > >>> > > > > > > changes
> > > > > > >>> > > > > > > > >> to
> > > > > > >>> > > > > > > > >> > > > KIP-890
> > > > > > >>> > > > > > > > >> > > > > > > since
> > > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > particular,
> we
> > > are
> > > > > > >>> changing
> > > > > > >>> > > the
> > > > > > >>> > > > > > format
> > > > > > >>> > > > > > > > of
> > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionLogValue.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > comments
> > related
> > > > to
> > > > > > >>> that.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100. Just to
> > be
> > > > > clear.
> > > > > > >>> The
> > > > > > >>> > > > > overflow
> > > > > > >>> > > > > > > case
> > > > > > >>> > > > > > > > >> > (i.e.
> > > > > > >>> > > > > > > > >> > > > > when a
> > > > > > >>> > > > > > > > >> > > > > > > new
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > generated)
> is
> > > when
> > > > > the
> > > > > > >>> > current
> > > > > > >>> > > > > epoch
> > > > > > >>> > > > > > > > >> equals
> > > > > > >>> > > > > > > > >> > to
> > > > > > >>> > > > > > > > >> > > > max
> > > > > > >>> > > > > > > > >> > > > > -
> > > > > > >>> > > > > > > > >> > > > > > 1
> > > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101. For the
> > > "not
> > > > > > epoch
> > > > > > >>> > > > overflow"
> > > > > > >>> > > > > > > case,
> > > > > > >>> > > > > > > > we
> > > > > > >>> > > > > > > > >> > > write
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > tagged field
> > in
> > > > the
> > > > > > >>> complete
> > > > > > >>> > > > > phase.
> > > > > > >>> > > > > > Do
> > > > > > >>> > > > > > > > we
> > > > > > >>> > > > > > > > >> > need
> > > > > > >>> > > > > > > > >> > > to
> > > > > > >>> > > > > > > > >> > > > > do
> > > > > > >>> > > > > > > > >> > > > > > > that
> > > > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > doesn't
> change
> > > in
> > > > > this
> > > > > > >>> case?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102. It
> seems
> > > that
> > > > > the
> > > > > > >>> > meaning
> > > > > > >>> > > > for
> > > > > > >>> > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId/ProducerEpoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > TransactionLogValue
> > > > > > >>> changes
> > > > > > >>> > > > > > depending
> > > > > > >>> > > > > > > on
> > > > > > >>> > > > > > > > >> the
> > > > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > > TransactionStatus
> > > > > > is
> > > > > > >>> > > > ongoing,
> > > > > > >>> > > > > > they
> > > > > > >>> > > > > > > > >> > > represent
> > > > > > >>> > > > > > > > >> > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > current
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the current
> > > > > > >>> ProducerEpoch.
> > > > > > >>> > > When
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > > > >>> > > > > > > > >> > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> PrepareCommit/PrepareAbort,
> > > > > > >>> > > they
> > > > > > >>> > > > > > > > represent
> > > > > > >>> > > > > > > > >> > the
> > > > > > >>> > > > > > > > >> > > > > > current
> > > > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > > > ProducerEpoch.
> > > > > > >>> When the
> > > > > > >>> > > > > > > > >> > TransactionStatus
> > > > > > >>> > > > > > > > >> > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > further
> depend
> > > on
> > > > > > >>> whether
> > > > > > >>> > the
> > > > > > >>> > > > > epoch
> > > > > > >>> > > > > > > > >> overflows
> > > > > > >>> > > > > > > > >> > > or
> > > > > > >>> > > > > > > > >> > > > > not.
> > > > > > >>> > > > > > > > >> > > > > > > If
> > > > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > overflow,
> they
> > > > > > represent
> > > > > > >>> > the
> > > > > > >>> > > > > > current
> > > > > > >>> > > > > > > > >> > > ProducerId
> > > > > > >>> > > > > > > > >> > > > > and
> > > > > > >>> > > > > > > > >> > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ProducerEpoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > (max).
> > > Otherwise,
> > > > > they
> > > > > > >>> > > represent
> > > > > > >>> > > > > the
> > > > > > >>> > > > > > > > newly
> > > > > > >>> > > > > > > > >> > > > > generated
> > > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> ProducerEpoch
> > of
> > > > 0.
> > > > > Is
> > > > > > >>> that
> > > > > > >>> > > > right?
> > > > > > >>> > > > > > > This
> > > > > > >>> > > > > > > > >> seems
> > > > > > >>> > > > > > > > >> > > not
> > > > > > >>> > > > > > > > >> > > > > > easy
> > > > > > >>> > > > > > > > >> > > > > > > to
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we provide
> > some
> > > > > > examples
> > > > > > >>> > like
> > > > > > >>> > > > what
> > > > > > >>> > > > > > > Artem
> > > > > > >>> > > > > > > > >> has
> > > > > > >>> > > > > > > > >> > > done
> > > > > > >>> > > > > > > > >> > > > > in
> > > > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > considered a
> > > > simpler
> > > > > > >>> design
> > > > > > >>> > > > where
> > > > > > >>> > > > > > > > >> > > > > > > > ProducerId/ProducerEpoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > represent
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the same
> value
> > > > (e.g.
> > > > > > >>> for the
> > > > > > >>> > > > > current
> > > > > > >>> > > > > > > > >> > > transaction)
> > > > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > TransactionStatus
> > > > > and
> > > > > > >>> epoch
> > > > > > >>> > > > > > overflow?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103. It's
> not
> > > > clear
> > > > > to
> > > > > > >>> me
> > > > > > >>> > why
> > > > > > >>> > > we
> > > > > > >>> > > > > > need
> > > > > > >>> > > > > > > 3
> > > > > > >>> > > > > > > > >> > fields:
> > > > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> PrevProducerId,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > NextProducerId.
> > > > > Could
> > > > > > we
> > > > > > >>> > just
> > > > > > >>> > > > have
> > > > > > >>> > > > > > > > >> ProducerId
> > > > > > >>> > > > > > > > >> > > and
> > > > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104. For
> > > > > > >>> > > WriteTxnMarkerRequests,
> > > > > > >>> > > > > if
> > > > > > >>> > > > > > > the
> > > > > > >>> > > > > > > > >> > > producer
> > > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we set the
> > > > > producerId
> > > > > > >>> and
> > > > > > >>> > the
> > > > > > >>> > > > > > > > >> producerEpoch?
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Thanks,
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > > >
> > > > > > >>> > > > > > > > >> > > > > > >
> > > > > > >>> > > > > > > > >> > > > > >
> > > > > > >>> > > > > > > > >> > > > >
> > > > > > >>> > > > > > > > >> > > >
> > > > > > >>> > > > > > > > >> > >
> > > > > > >>> > > > > > > > >> >
> > > > > > >>> > > > > > > > >>
> > > > > > >>> > > > > > > > >
> > > > > > >>> > > > > > > >
> > > > > > >>> > > > > > >
> > > > > > >>> > > > > >
> > > > > > >>> > > > >
> > > > > > >>> > > >
> > > > > > >>> > >
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Jun Rao <ju...@confluent.io.INVALID>]

Sounds good, Justine. It would be useful to document that in the KIP.

Thanks,

Jun

On Wed, Jul 10, 2024 at 2:59 PM Justine Olshan <jo...@confluent.io.invalid>
wrote:

> The client will send the newest EndTxn request version if and only if both
> the client and the server support kip-890 part 2.
> We set the value in the record based on the EndTxn version.
>
> Justine
>
> On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <ju...@confluent.io.invalid> wrote:
>
> > Hi, Justine,
> >
> > Thanks for the reply.
> >
> > 120. If the broker sends TV Y for the finalized version in
> > ApiVersionResponse, but the client doesn't support Y, how does the broker
> > know the TV that the client supports?
> >
> > Jun
> >
> > On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> > <jo...@confluent.io.invalid>
> > wrote:
> >
> > > Hey Jun,
> > >
> > > No worries. Work on this KIP has been blocked for a bit anyways --
> > catching
> > > up and rereading what I wrote :)
> > >
> > > 120. ClientTransactionProtocolVersion is the transaction version as
> > defined
> > > by the highest transaction version (feature version value) supported by
> > the
> > > client and the server. This works by the broker sending an
> > > ApiVersionsRequest to the client with the finalized version. Assuming
> > > kip-890 part 2 is enabled by transaction version Y, if this request
> > > contains finalized version Y and the client has the logic to set this
> > > field, it will set Y. If the server has Y - 1 (kip 890 part 2 not
> enable)
> > > the client will send Y - 1, even though the client has the ability to
> > > support kip-890 part 2.
> > >
> > > 121. You are correct that this is not needed. However, currently that
> > field
> > > is already being set in memory -- just not written to disk. I think it
> is
> > > ok to write it to disk though. Let me know if you think otherwise.
> > >
> > > Justine
> > >
> > > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao <ju...@confluent.io.invalid>
> > wrote:
> > >
> > > > Hi, Justine,
> > > >
> > > > Thanks for the update and sorry for the late reply.
> > > >
> > > > 120. I am wondering what value is used for
> > > > ClientTransactionProtocolVersion. Is it the version of the
> > EndTxnRequest?
> > > >
> > > > 121. Earlier, you made the change to set lastProducerId in PREPARE to
> > > > indicate that the marker is written for the new client. With the new
> > > > ClientTransactionProtocolVersion field, it seems this is no longer
> > > > necessary.
> > > >
> > > > Jun
> > > >
> > > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > > <jo...@confluent.io.invalid>
> > > > wrote:
> > > >
> > > > > Hi there -- another update!
> > > > >
> > > > > When looking into the implementation for the safe epoch bumps I
> > > realized
> > > > > that we are already populating previousProducerID in memory as part
> > of
> > > > > KIP-360.
> > > > > If we are to start using flexible fields, it is better to always
> use
> > > this
> > > > > information and have an explicit (tagged) field to indicate whether
> > the
> > > > > client supports KIP-890 part 2.
> > > > >
> > > > > I've included the extra field and how it is set in the KIP. I've
> also
> > > > > updated the KIP to explain that we will be setting the tagged
> fields
> > > when
> > > > > they are available for all transitions.
> > > > >
> > > > > Finally, I added clearer text about the transaction protocol
> versions
> > > > > included with this KIP. 1 for flexible transaction state records
> and
> > 2
> > > > for
> > > > > KIP-890 part 2 enablement.
> > > > >
> > > > > Justine
> > > > >
> > > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <
> jolshan@confluent.io
> > >
> > > > > wrote:
> > > > >
> > > > > > Hey there -- small update to the KIP,
> > > > > >
> > > > > > The KIP mentioned introducing ABORTABLE_ERROR and bumping
> > > > TxnOffsetCommit
> > > > > > and Produce requests. I've changed the name in the KIP to
> > > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > > AbortableTransactionException to match the pattern we had for
> other
> > > > > errors.
> > > > > > I also mentioned bumping all 6 transactional APIs so we can
> future
> > > > > > proof/support the error on the client going forward. If a future
> > > change
> > > > > > wants to have an error scenario that requires us to abort the
> > > > > transaction,
> > > > > > we can rely on the 3.8+ clients to support it. We ran into issues
> > > > finding
> > > > > > good/generic error codes that older clients could support while
> > > working
> > > > > on
> > > > > > this KIP, so this should help in the future.
> > > > > >
> > > > > > The features discussion is still ongoing in KIP-1022. Will update
> > > again
> > > > > > here when that concludes.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> > jolshan@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > >> I don't think AddPartitions is a good example since we currenly
> > > don't
> > > > > >> gate the version on TV or MV. (We only set a different flag
> > > depending
> > > > on
> > > > > >> the TV)
> > > > > >>
> > > > > >> Even if we did want to gate it on TV, I think the idea is to
> move
> > > away
> > > > > >> from MV gating inter broker protocols. Ideally we can get to a
> > state
> > > > > where
> > > > > >> MV is just used for metadata changes.
> > > > > >>
> > > > > >> I think some of this discussion might fit more with the feature
> > > > version
> > > > > >> KIP, so I can try to open that up soon. Until we settle that,
> some
> > > of
> > > > > the
> > > > > >> work in KIP-890 is blocked.
> > > > > >>
> > > > > >> Justine
> > > > > >>
> > > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao <jun@confluent.io.invalid
> >
> > > > > wrote:
> > > > > >>
> > > > > >>> Hi, Justine,
> > > > > >>>
> > > > > >>> Thanks for the reply.
> > > > > >>>
> > > > > >>> Since AddPartitions is an inter broker request, will its
> version
> > be
> > > > > gated
> > > > > >>> only by TV or other features like MV too? For example, if we
> need
> > > to
> > > > > >>> change
> > > > > >>> the protocol for AddPartitions for reasons other than txn
> > > > verification
> > > > > in
> > > > > >>> the future, will the new version be gated by a new MV? If so,
> > does
> > > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > > >>>
> > > > > >>> Jun
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > > >>> <jo...@confluent.io.invalid>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> > One TV gates the flexible feature version (no rpcs involved,
> > only
> > > > the
> > > > > >>> > transactional records that should only be gated by TV)
> > > > > >>> > Another TV gates the ability to turn on kip-890 part 2. This
> > > would
> > > > > >>> gate the
> > > > > >>> > version of Produce and EndTxn (likely only used by
> > transactions),
> > > > and
> > > > > >>> > specifies a flag in AddPartitionsToTxn though the version is
> > > > already
> > > > > >>> used
> > > > > >>> > without TV.
> > > > > >>> >
> > > > > >>> > I think the only concern is the Produce request and we could
> > > > consider
> > > > > >>> work
> > > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > > >>> >
> > > > > >>> > Justine
> > > > > >>> >
> > > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> > <jun@confluent.io.invalid
> > > >
> > > > > >>> wrote:
> > > > > >>> >
> > > > > >>> > > Hi, Justine,
> > > > > >>> > >
> > > > > >>> > > Which PRC/record protocols will TV guard? Going forward,
> will
> > > > those
> > > > > >>> > > PRC/record protocols only be guarded by TV and not by other
> > > > > features
> > > > > >>> like
> > > > > >>> > > MV?
> > > > > >>> > >
> > > > > >>> > > Thanks,
> > > > > >>> > >
> > > > > >>> > > Jun
> > > > > >>> > >
> > > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > > >>> > <jolshan@confluent.io.invalid
> > > > > >>> > > >
> > > > > >>> > > wrote:
> > > > > >>> > >
> > > > > >>> > > > Hi Jun,
> > > > > >>> > > >
> > > > > >>> > > > Sorry I think I misunderstood your question or answered
> > > > > >>> incorrectly.
> > > > > >>> > The
> > > > > >>> > > TV
> > > > > >>> > > > version should ideally be fully independent from MV.
> > > > > >>> > > > At least for the changes I proposed, TV should not affect
> > MV
> > > > and
> > > > > MV
> > > > > >>> > > should
> > > > > >>> > > > not affect TV/
> > > > > >>> > > >
> > > > > >>> > > > I think if we downgrade TV, only that feature should
> > > downgrade.
> > > > > >>> > Likewise
> > > > > >>> > > > the same with MV. The finalizedFeatures should just
> reflect
> > > the
> > > > > >>> feature
> > > > > >>> > > > downgrade we made.
> > > > > >>> > > >
> > > > > >>> > > > I also plan to write a new KIP for managing the disk
> format
> > > and
> > > > > >>> upgrade
> > > > > >>> > > > tool as we will need new flags to support these features.
> > > That
> > > > > >>> should
> > > > > >>> > > help
> > > > > >>> > > > clarify some things.
> > > > > >>> > > >
> > > > > >>> > > > Justine
> > > > > >>> > > >
> > > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > > <jun@confluent.io.invalid
> > > > > >
> > > > > >>> > > wrote:
> > > > > >>> > > >
> > > > > >>> > > > > Hi, Justine,
> > > > > >>> > > > >
> > > > > >>> > > > > Thanks for the reply.
> > > > > >>> > > > >
> > > > > >>> > > > > So, if we downgrade TV, we could implicitly downgrade
> > > another
> > > > > >>> feature
> > > > > >>> > > > (say
> > > > > >>> > > > > MV) that has dependency (e.g. RPC). What would we
> return
> > > for
> > > > > >>> > > > > FinalizedFeatures for MV in ApiVersionsResponse in that
> > > case?
> > > > > >>> > > > >
> > > > > >>> > > > > Thanks,
> > > > > >>> > > > >
> > > > > >>> > > > > Jun
> > > > > >>> > > > >
> > > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > > >>> > > > <jolshan@confluent.io.invalid
> > > > > >>> > > > > >
> > > > > >>> > > > > wrote:
> > > > > >>> > > > >
> > > > > >>> > > > > > Hey Jun,
> > > > > >>> > > > > >
> > > > > >>> > > > > > Yes, the idea is that if we downgrade TV (transaction
> > > > > version)
> > > > > >>> we
> > > > > >>> > > will
> > > > > >>> > > > > stop
> > > > > >>> > > > > > using the add partitions to txn optimization and stop
> > > > writing
> > > > > >>> the
> > > > > >>> > > > > flexible
> > > > > >>> > > > > > feature version of the log.
> > > > > >>> > > > > > In the compatibility section I included some
> > explanations
> > > > on
> > > > > >>> how
> > > > > >>> > this
> > > > > >>> > > > is
> > > > > >>> > > > > > done.
> > > > > >>> > > > > >
> > > > > >>> > > > > > Thanks,
> > > > > >>> > > > > > Justine
> > > > > >>> > > > > >
> > > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > > >>> <ju...@confluent.io.invalid>
> > > > > >>> > > > > wrote:
> > > > > >>> > > > > >
> > > > > >>> > > > > > > Hi, Justine,
> > > > > >>> > > > > > >
> > > > > >>> > > > > > > Thanks for the update.
> > > > > >>> > > > > > >
> > > > > >>> > > > > > > If we ever downgrade the transaction feature, any
> > > feature
> > > > > >>> > depending
> > > > > >>> > > > on
> > > > > >>> > > > > > > changes on top of those RPC/record
> > > > > >>> > > > > > > (AddPartitionsToTxnRequest/TransactionLogValue)
> > changes
> > > > > made
> > > > > >>> in
> > > > > >>> > > > KIP-890
> > > > > >>> > > > > > > will be automatically downgraded too?
> > > > > >>> > > > > > >
> > > > > >>> > > > > > > Jun
> > > > > >>> > > > > > >
> > > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine Olshan
> > > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > > >>> > > > > > > wrote:
> > > > > >>> > > > > > >
> > > > > >>> > > > > > > > Hey Jun,
> > > > > >>> > > > > > > >
> > > > > >>> > > > > > > > I wanted to get back to you about your questions
> > > about
> > > > > >>> MV/IBP.
> > > > > >>> > > > > > > >
> > > > > >>> > > > > > > > Looking at the options, I think it makes the most
> > > sense
> > > > > to
> > > > > >>> > > create a
> > > > > >>> > > > > > > > separate feature for transactions and use that to
> > > > version
> > > > > >>> gate
> > > > > >>> > > the
> > > > > >>> > > > > > > features
> > > > > >>> > > > > > > > we need to version gate (flexible transactional
> > state
> > > > > >>> records
> > > > > >>> > and
> > > > > >>> > > > > using
> > > > > >>> > > > > > > the
> > > > > >>> > > > > > > > new protocol)
> > > > > >>> > > > > > > > I've updated the KIP to include this change.
> > > Hopefully
> > > > > >>> that's
> > > > > >>> > > > > > everything
> > > > > >>> > > > > > > we
> > > > > >>> > > > > > > > need for this KIP :)
> > > > > >>> > > > > > > >
> > > > > >>> > > > > > > > Justine
> > > > > >>> > > > > > > >
> > > > > >>> > > > > > > >
> > > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine Olshan <
> > > > > >>> > > > jolshan@confluent.io
> > > > > >>> > > > > >
> > > > > >>> > > > > > > > wrote:
> > > > > >>> > > > > > > >
> > > > > >>> > > > > > > > > Thanks Jun,
> > > > > >>> > > > > > > > >
> > > > > >>> > > > > > > > > I will update the KIP with the prev field for
> > > prepare
> > > > > as
> > > > > >>> > well.
> > > > > >>> > > > > > > > >
> > > > > >>> > > > > > > > > PREPARE
> > > > > >>> > > > > > > > > producerId: x
> > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > >>> > > > > > > > > nextProducerId (tagged field): empty or z if y
> > will
> > > > > >>> overflow
> > > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > > >>> > > > > > > > >
> > > > > >>> > > > > > > > > COMPLETE
> > > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we overflowed
> > > > > >>> > > > > > > > >
> > > > > >>> > > > > > > > > Thanks again,
> > > > > >>> > > > > > > > > Justine
> > > > > >>> > > > > > > > >
> > > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun Rao
> > > > > >>> > > <jun@confluent.io.invalid
> > > > > >>> > > > >
> > > > > >>> > > > > > > > wrote:
> > > > > >>> > > > > > > > >
> > > > > >>> > > > > > > > >> Hi, Justine,
> > > > > >>> > > > > > > > >>
> > > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > > >>> > > > > > > > >> (1) My point was that the coordinator could
> fail
> > > > right
> > > > > >>> after
> > > > > >>> > > > > writing
> > > > > >>> > > > > > > the
> > > > > >>> > > > > > > > >> prepare marker. When the new txn coordinator
> > > > generates
> > > > > >>> the
> > > > > >>> > > > > complete
> > > > > >>> > > > > > > > marker
> > > > > >>> > > > > > > > >> after the failover, it needs some field from
> the
> > > > > prepare
> > > > > >>> > > marker
> > > > > >>> > > > to
> > > > > >>> > > > > > > > >> determine whether it's written by the new
> > client.
> > > > > >>> > > > > > > > >>
> > > > > >>> > > > > > > > >> (2) The changing of the behavior sounds good
> to
> > > me.
> > > > We
> > > > > >>> only
> > > > > >>> > > want
> > > > > >>> > > > > to
> > > > > >>> > > > > > > > return
> > > > > >>> > > > > > > > >> success if the prepare state is written by the
> > new
> > > > > >>> client.
> > > > > >>> > So,
> > > > > >>> > > > in
> > > > > >>> > > > > > the
> > > > > >>> > > > > > > > >> non-overflow case, it seems that we also need
> > sth
> > > in
> > > > > the
> > > > > >>> > > prepare
> > > > > >>> > > > > > > marker
> > > > > >>> > > > > > > > to
> > > > > >>> > > > > > > > >> tell us whether it's written by the new
> client.
> > > > > >>> > > > > > > > >>
> > > > > >>> > > > > > > > >> 112. Thanks for the explanation. That sounds
> > good
> > > to
> > > > > me.
> > > > > >>> > > > > > > > >>
> > > > > >>> > > > > > > > >> Jun
> > > > > >>> > > > > > > > >>
> > > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM Justine
> Olshan
> > > > > >>> > > > > > > > >> <jo...@confluent.io.invalid> wrote:
> > > > > >>> > > > > > > > >>
> > > > > >>> > > > > > > > >> > 101.3 I realized that I actually have two
> > > > questions.
> > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need to
> > write
> > > > the
> > > > > >>> > > previous
> > > > > >>> > > > > > > produce
> > > > > >>> > > > > > > > Id
> > > > > >>> > > > > > > > >> > tagged field in the end maker so that we
> know
> > if
> > > > the
> > > > > >>> > marker
> > > > > >>> > > is
> > > > > >>> > > > > > from
> > > > > >>> > > > > > > > the
> > > > > >>> > > > > > > > >> new
> > > > > >>> > > > > > > > >> > client. Since the end maker is derived from
> > the
> > > > > >>> prepare
> > > > > >>> > > > marker,
> > > > > >>> > > > > > > should
> > > > > >>> > > > > > > > >> we
> > > > > >>> > > > > > > > >> > write the previous produce Id in the prepare
> > > > marker
> > > > > >>> field
> > > > > >>> > > too?
> > > > > >>> > > > > > > > >> Otherwise,
> > > > > >>> > > > > > > > >> > we will lose this information when deriving
> > the
> > > > end
> > > > > >>> > marker.
> > > > > >>> > > > > > > > >> >
> > > > > >>> > > > > > > > >> > The "previous" producer ID is in the normal
> > > > producer
> > > > > >>> ID
> > > > > >>> > > field.
> > > > > >>> > > > > So
> > > > > >>> > > > > > > yes,
> > > > > >>> > > > > > > > >> we
> > > > > >>> > > > > > > > >> > need it in prepare and that was always the
> > plan.
> > > > > >>> > > > > > > > >> >
> > > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I will
> enumerate
> > > the
> > > > > >>> fields
> > > > > >>> > and
> > > > > >>> > > > add
> > > > > >>> > > > > > > them
> > > > > >>> > > > > > > > to
> > > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > > >>> > > > > > > > >> > Say we have producer ID x and epoch y. When
> we
> > > > > >>> overflow
> > > > > >>> > > epoch
> > > > > >>> > > > y
> > > > > >>> > > > > we
> > > > > >>> > > > > > > get
> > > > > >>> > > > > > > > >> > producer ID Z.
> > > > > >>> > > > > > > > >> >
> > > > > >>> > > > > > > > >> > PREPARE
> > > > > >>> > > > > > > > >> > producerId: x
> > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged field):
> empty
> > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty or z
> if y
> > > > will
> > > > > >>> > overflow
> > > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > > >>> > > > > > > > >> >
> > > > > >>> > > > > > > > >> > COMPLETE
> > > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > > >>> > > > > > > > >> > previous/lastProducerId (tagged field): x
> > > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we overflowed
> > > > > >>> > > > > > > > >> >
> > > > > >>> > > > > > > > >> > (2) In the prepare phase, if we retry and
> see
> > > > epoch
> > > > > -
> > > > > >>> 1 +
> > > > > >>> > ID
> > > > > >>> > > > in
> > > > > >>> > > > > > last
> > > > > >>> > > > > > > > >> seen
> > > > > >>> > > > > > > > >> > fields and are issuing the same command (ie
> > > commit
> > > > > not
> > > > > >>> > > abort),
> > > > > >>> > > > > we
> > > > > >>> > > > > > > > return
> > > > > >>> > > > > > > > >> > success. The logic before KIP-890 seems to
> > > return
> > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > >>> > > > > > > > >> > in this case. Are we intentionally making
> this
> > > > > change?
> > > > > >>> > > > > > > > >> >
> > > > > >>> > > > > > > > >> > Hmm -- we would fence the producer if the
> > epoch
> > > is
> > > > > >>> bumped
> > > > > >>> > > and
> > > > > >>> > > > we
> > > > > >>> > > > > > > get a
> > > > > >>> > > > > > > > >> > lower epoch. Yes -- we are intentionally
> > adding
> > > > this
> > > > > >>> to
> > > > > >>> > > > prevent
> > > > > >>> > > > > > > > fencing.
> > > > > >>> > > > > > > > >> >
> > > > > >>> > > > > > > > >> >
> > > > > >>> > > > > > > > >> > 112. We already merged the code that adds
> the
> > > > > >>> VerifyOnly
> > > > > >>> > > field
> > > > > >>> > > > > in
> > > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which is an inter
> > > > broker
> > > > > >>> > request.
> > > > > >>> > > > It
> > > > > >>> > > > > > > seems
> > > > > >>> > > > > > > > >> that
> > > > > >>> > > > > > > > >> > we didn't bump up the IBP for that. Do you
> > know
> > > > why?
> > > > > >>> > > > > > > > >> >
> > > > > >>> > > > > > > > >> > We no longer need IBP for all interbroker
> > > requests
> > > > > as
> > > > > >>> > > > > ApiVersions
> > > > > >>> > > > > > > > should
> > > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > > >>> > > > > > > > >> > We also handle unsupported version errors
> > > > correctly
> > > > > >>> if we
> > > > > >>> > > > > receive
> > > > > >>> > > > > > > them
> > > > > >>> > > > > > > > >> in
> > > > > >>> > > > > > > > >> > edge cases like upgrades/downgrades.
> > > > > >>> > > > > > > > >> >
> > > > > >>> > > > > > > > >> > Justine
> > > > > >>> > > > > > > > >> >
> > > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM Jun Rao
> > > > > >>> > > > > <jun@confluent.io.invalid
> > > > > >>> > > > > > >
> > > > > >>> > > > > > > > >> wrote:
> > > > > >>> > > > > > > > >> >
> > > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > > >>> > > > > > > > >> > >
> > > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > > >>> > > > > > > > >> > >
> > > > > >>> > > > > > > > >> > > 101.3 I realized that I actually have two
> > > > > questions.
> > > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need to
> > write
> > > > the
> > > > > >>> > > previous
> > > > > >>> > > > > > > produce
> > > > > >>> > > > > > > > Id
> > > > > >>> > > > > > > > >> > > tagged field in the end maker so that we
> > know
> > > if
> > > > > the
> > > > > >>> > > marker
> > > > > >>> > > > is
> > > > > >>> > > > > > > from
> > > > > >>> > > > > > > > >> the
> > > > > >>> > > > > > > > >> > new
> > > > > >>> > > > > > > > >> > > client. Since the end maker is derived
> from
> > > the
> > > > > >>> prepare
> > > > > >>> > > > > marker,
> > > > > >>> > > > > > > > >> should we
> > > > > >>> > > > > > > > >> > > write the previous produce Id in the
> prepare
> > > > > marker
> > > > > >>> > field
> > > > > >>> > > > too?
> > > > > >>> > > > > > > > >> Otherwise,
> > > > > >>> > > > > > > > >> > > we will lose this information when
> deriving
> > > the
> > > > > end
> > > > > >>> > > marker.
> > > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we retry and
> > see
> > > > > epoch
> > > > > >>> - 1
> > > > > >>> > +
> > > > > >>> > > ID
> > > > > >>> > > > > in
> > > > > >>> > > > > > > last
> > > > > >>> > > > > > > > >> seen
> > > > > >>> > > > > > > > >> > > fields and are issuing the same command
> (ie
> > > > commit
> > > > > >>> not
> > > > > >>> > > > abort),
> > > > > >>> > > > > > we
> > > > > >>> > > > > > > > >> return
> > > > > >>> > > > > > > > >> > > success. The logic before KIP-890 seems to
> > > > return
> > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > >>> > > > > > > > >> > > in this case. Are we intentionally making
> > this
> > > > > >>> change?
> > > > > >>> > > > > > > > >> > >
> > > > > >>> > > > > > > > >> > > 112. We already merged the code that adds
> > the
> > > > > >>> VerifyOnly
> > > > > >>> > > > field
> > > > > >>> > > > > > in
> > > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which is an
> inter
> > > > > broker
> > > > > >>> > > request.
> > > > > >>> > > > > It
> > > > > >>> > > > > > > > seems
> > > > > >>> > > > > > > > >> > that
> > > > > >>> > > > > > > > >> > > we didn't bump up the IBP for that. Do you
> > > know
> > > > > why?
> > > > > >>> > > > > > > > >> > >
> > > > > >>> > > > > > > > >> > > Jun
> > > > > >>> > > > > > > > >> > >
> > > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM Justine
> > Olshan
> > > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > > >>> > > > > > > > >> > > wrote:
> > > > > >>> > > > > > > > >> > >
> > > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > > >>> > > > > > > > >> > > >
> > > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen" to
> "current
> > > > > >>> producer id
> > > > > >>> > > and
> > > > > >>> > > > > > > epoch"
> > > > > >>> > > > > > > > if
> > > > > >>> > > > > > > > >> > that
> > > > > >>> > > > > > > > >> > > > was the part that was confusing
> > > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > > >>> > > > > > > > >> > > > 112 We still need it. But I am still
> > > > finalizing
> > > > > >>> the
> > > > > >>> > > > design.
> > > > > >>> > > > > I
> > > > > >>> > > > > > > will
> > > > > >>> > > > > > > > >> > update
> > > > > >>> > > > > > > > >> > > > the KIP once I get the information
> > > finalized.
> > > > > >>> Sorry
> > > > > >>> > for
> > > > > >>> > > > the
> > > > > >>> > > > > > > > delays.
> > > > > >>> > > > > > > > >> > > >
> > > > > >>> > > > > > > > >> > > > Justine
> > > > > >>> > > > > > > > >> > > >
> > > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at 10:50 AM Jun Rao
> > > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > > >>> > > > > > > > >
> > > > > >>> > > > > > > > >> > > wrote:
> > > > > >>> > > > > > > > >> > > >
> > > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > > >>> > > > > > > > >> > > > >
> > > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > > >>> > > > > > > > >> > > > >
> > > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow case, the
> > > previous
> > > > > ID
> > > > > >>> is
> > > > > >>> > the
> > > > > >>> > > > > same
> > > > > >>> > > > > > as
> > > > > >>> > > > > > > > the
> > > > > >>> > > > > > > > >> > > > produce
> > > > > >>> > > > > > > > >> > > > > ID for the complete marker too, but we
> > set
> > > > the
> > > > > >>> > > previous
> > > > > >>> > > > ID
> > > > > >>> > > > > > in
> > > > > >>> > > > > > > > the
> > > > > >>> > > > > > > > >> > > > complete
> > > > > >>> > > > > > > > >> > > > > marker. Earlier you mentioned that
> this
> > is
> > > > to
> > > > > >>> know
> > > > > >>> > > that
> > > > > >>> > > > > the
> > > > > >>> > > > > > > > >> marker is
> > > > > >>> > > > > > > > >> > > > > written by the new client so that we
> > could
> > > > > >>> return
> > > > > >>> > > > success
> > > > > >>> > > > > on
> > > > > >>> > > > > > > > >> retried
> > > > > >>> > > > > > > > >> > > > > endMarker requests. I was trying to
> > > > understand
> > > > > >>> why
> > > > > >>> > > this
> > > > > >>> > > > is
> > > > > >>> > > > > > not
> > > > > >>> > > > > > > > >> needed
> > > > > >>> > > > > > > > >> > > for
> > > > > >>> > > > > > > > >> > > > > the prepare marker since retry can
> > happen
> > > in
> > > > > the
> > > > > >>> > > prepare
> > > > > >>> > > > > > state
> > > > > >>> > > > > > > > >> too.
> > > > > >>> > > > > > > > >> > Is
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > reason that in the prepare state, we
> > > return
> > > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > > >>> > > > > > > > >> > > > instead
> > > > > >>> > > > > > > > >> > > > > of success on retried endMaker
> requests?
> > > If
> > > > > so,
> > > > > >>> > should
> > > > > >>> > > > we
> > > > > >>> > > > > > > change
> > > > > >>> > > > > > > > >> "If
> > > > > >>> > > > > > > > >> > we
> > > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID in last
> > seen
> > > > > >>> fields and
> > > > > >>> > > are
> > > > > >>> > > > > > > issuing
> > > > > >>> > > > > > > > >> the
> > > > > >>> > > > > > > > >> > > same
> > > > > >>> > > > > > > > >> > > > > command (ie commit not abort) we can
> > > return
> > > > > >>> (with
> > > > > >>> > the
> > > > > >>> > > > new
> > > > > >>> > > > > > > > epoch)"
> > > > > >>> > > > > > > > >> > > > > accordingly?
> > > > > >>> > > > > > > > >> > > > >
> > > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP, a delayed
> > > > endMaker
> > > > > >>> > request
> > > > > >>> > > > > > carries
> > > > > >>> > > > > > > > the
> > > > > >>> > > > > > > > >> > same
> > > > > >>> > > > > > > > >> > > > > epoch and won't be fenced. This can
> > > > > >>> commit/abort a
> > > > > >>> > > > future
> > > > > >>> > > > > > > > >> transaction
> > > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure if we have
> > > seen
> > > > > >>> this in
> > > > > >>> > > > > practice
> > > > > >>> > > > > > > > >> though.
> > > > > >>> > > > > > > > >> > > > >
> > > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would be useful
> to
> > > make
> > > > > it
> > > > > >>> > clear
> > > > > >>> > > > that
> > > > > >>> > > > > > we
> > > > > >>> > > > > > > > can
> > > > > >>> > > > > > > > >> now
> > > > > >>> > > > > > > > >> > > > > populate the lastSeen field from the
> log
> > > > > >>> reliably.
> > > > > >>> > > > > > > > >> > > > >
> > > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > > >>> > AddPartitionsToTxnRequest
> > > > > >>> > > > > since
> > > > > >>> > > > > > > > it's
> > > > > >>> > > > > > > > >> > > called
> > > > > >>> > > > > > > > >> > > > > across brokers and we are changing its
> > > > schema.
> > > > > >>> Are
> > > > > >>> > you
> > > > > >>> > > > > > saying
> > > > > >>> > > > > > > we
> > > > > >>> > > > > > > > >> > don't
> > > > > >>> > > > > > > > >> > > > need
> > > > > >>> > > > > > > > >> > > > > it any more? I thought that we already
> > > > > >>> implemented
> > > > > >>> > the
> > > > > >>> > > > > > server
> > > > > >>> > > > > > > > side
> > > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > > >>> > AddPartitionsToTxnRequest
> > > > > >>> > > > > across
> > > > > >>> > > > > > > > >> brokers.
> > > > > >>> > > > > > > > >> > > > >
> > > > > >>> > > > > > > > >> > > > > Jun
> > > > > >>> > > > > > > > >> > > > >
> > > > > >>> > > > > > > > >> > > > >
> > > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at 5:05 PM
> Justine
> > > > Olshan
> > > > > >>> > > > > > > > >> > > > > <jo...@confluent.io.invalid>
> > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > >>> > > > > > > > >> > > > >
> > > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > > >>> > > > > > > > >> > > > > >
> > > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the previous ID
> in
> > > the
> > > > > >>> Prepare
> > > > > >>> > > > field
> > > > > >>> > > > > > > since
> > > > > >>> > > > > > > > we
> > > > > >>> > > > > > > > >> > > don't
> > > > > >>> > > > > > > > >> > > > > need
> > > > > >>> > > > > > > > >> > > > > > it. It is the same producer ID as
> the
> > > main
> > > > > >>> > producer
> > > > > >>> > > ID
> > > > > >>> > > > > > > field.
> > > > > >>> > > > > > > > >> > > > > >
> > > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to reread
> your
> > > > > message
> > > > > >>> > about
> > > > > >>> > > > > > delayed
> > > > > >>> > > > > > > > >> > markers.
> > > > > >>> > > > > > > > >> > > > If
> > > > > >>> > > > > > > > >> > > > > we
> > > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn marker
> after
> > > the
> > > > > >>> > > transaction
> > > > > >>> > > > is
> > > > > >>> > > > > > > > already
> > > > > >>> > > > > > > > >> > > > > complete?
> > > > > >>> > > > > > > > >> > > > > > So we will commit the next
> transaction
> > > > early
> > > > > >>> > without
> > > > > >>> > > > the
> > > > > >>> > > > > > > fixes
> > > > > >>> > > > > > > > >> in
> > > > > >>> > > > > > > > >> > > part
> > > > > >>> > > > > > > > >> > > > 2?
> > > > > >>> > > > > > > > >> > > > > >
> > > > > >>> > > > > > > > >> > > > > > 111 Yes -- this terminology was used
> > in
> > > a
> > > > > >>> previous
> > > > > >>> > > KIP
> > > > > >>> > > > > and
> > > > > >>> > > > > > > > never
> > > > > >>> > > > > > > > >> > > > > > implemented it in the log -- only in
> > > > memory
> > > > > >>> > > > > > > > >> > > > > >
> > > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which interbroker
> protocol
> > > are
> > > > > you
> > > > > >>> > > > referring
> > > > > >>> > > > > > to?
> > > > > >>> > > > > > > I
> > > > > >>> > > > > > > > am
> > > > > >>> > > > > > > > >> > > > working
> > > > > >>> > > > > > > > >> > > > > on
> > > > > >>> > > > > > > > >> > > > > > the design for the work to remove
> the
> > > > extra
> > > > > >>> add
> > > > > >>> > > > > partitions
> > > > > >>> > > > > > > > call
> > > > > >>> > > > > > > > >> > and I
> > > > > >>> > > > > > > > >> > > > > right
> > > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I have yet
> to
> > > > > update
> > > > > >>> that
> > > > > >>> > > > > section
> > > > > >>> > > > > > > as
> > > > > >>> > > > > > > > I
> > > > > >>> > > > > > > > >> > > > finalize
> > > > > >>> > > > > > > > >> > > > > > the design so please stay tuned. Was
> > > there
> > > > > >>> > anything
> > > > > >>> > > > else
> > > > > >>> > > > > > you
> > > > > >>> > > > > > > > >> > thought
> > > > > >>> > > > > > > > >> > > > > needed
> > > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > > >>> > > > > > > > >> > > > > >
> > > > > >>> > > > > > > > >> > > > > > Justine
> > > > > >>> > > > > > > > >> > > > > >
> > > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at 3:07 PM Jun
> > Rao
> > > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > > >>> > > > > > > > >> > > > > wrote:
> > > > > >>> > > > > > > > >> > > > > >
> > > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > > >>> > > > > > > > >> > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > I don't see this create any issue.
> > It
> > > > just
> > > > > >>> makes
> > > > > >>> > > it
> > > > > >>> > > > a
> > > > > >>> > > > > > bit
> > > > > >>> > > > > > > > >> hard to
> > > > > >>> > > > > > > > >> > > > > explain
> > > > > >>> > > > > > > > >> > > > > > > what this non-tagged produce id
> > field
> > > > > >>> means. We
> > > > > >>> > > are
> > > > > >>> > > > > > > > >> essentially
> > > > > >>> > > > > > > > >> > > > trying
> > > > > >>> > > > > > > > >> > > > > to
> > > > > >>> > > > > > > > >> > > > > > > combine two actions (completing a
> > txn
> > > > and
> > > > > >>> init a
> > > > > >>> > > new
> > > > > >>> > > > > > > produce
> > > > > >>> > > > > > > > >> Id)
> > > > > >>> > > > > > > > >> > > in a
> > > > > >>> > > > > > > > >> > > > > > > single record. But, this may be
> fine
> > > > too.
> > > > > >>> > > > > > > > >> > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > A few other follow up comments.
> > > > > >>> > > > > > > > >> > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason that we
> > only
> > > > set
> > > > > >>> the
> > > > > >>> > > > previous
> > > > > >>> > > > > > > > >> produce id
> > > > > >>> > > > > > > > >> > > > > tagged
> > > > > >>> > > > > > > > >> > > > > > > field in the complete marker, but
> > not
> > > in
> > > > > the
> > > > > >>> > > prepare
> > > > > >>> > > > > > > marker,
> > > > > >>> > > > > > > > >> is
> > > > > >>> > > > > > > > >> > > that
> > > > > >>> > > > > > > > >> > > > in
> > > > > >>> > > > > > > > >> > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > prepare state, we always return
> > > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > > >>> > > > > > on
> > > > > >>> > > > > > > > >> > retried
> > > > > >>> > > > > > > > >> > > > > > endMaker
> > > > > >>> > > > > > > > >> > > > > > > requests?
> > > > > >>> > > > > > > > >> > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > 110. "I believe your second point
> is
> > > > > >>> mentioned
> > > > > >>> > in
> > > > > >>> > > > the
> > > > > >>> > > > > > > KIP. I
> > > > > >>> > > > > > > > >> can
> > > > > >>> > > > > > > > >> > > add
> > > > > >>> > > > > > > > >> > > > > more
> > > > > >>> > > > > > > > >> > > > > > > text on
> > > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > > >>> > > > > > > > >> > > > > > > > The delayed message case can
> also
> > > > > violate
> > > > > >>> EOS
> > > > > >>> > if
> > > > > >>> > > > the
> > > > > >>> > > > > > > > delayed
> > > > > >>> > > > > > > > >> > > > message
> > > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > > addPartitionsToTxn
> > > > > >>> > request
> > > > > >>> > > > > comes
> > > > > >>> > > > > > > in.
> > > > > >>> > > > > > > > >> > > > > Effectively
> > > > > >>> > > > > > > > >> > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > may see a message from a previous
> > > > > (aborted)
> > > > > >>> > > > > transaction
> > > > > >>> > > > > > > > become
> > > > > >>> > > > > > > > >> > part
> > > > > >>> > > > > > > > >> > > > of
> > > > > >>> > > > > > > > >> > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > > >>> > > > > > > > >> > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > The above is the case when a
> delayed
> > > > > >>> message is
> > > > > >>> > > > > appended
> > > > > >>> > > > > > > to
> > > > > >>> > > > > > > > >> the
> > > > > >>> > > > > > > > >> > > data
> > > > > >>> > > > > > > > >> > > > > > > partition. What I mentioned is a
> > > > slightly
> > > > > >>> > > different
> > > > > >>> > > > > case
> > > > > >>> > > > > > > > when
> > > > > >>> > > > > > > > >> a
> > > > > >>> > > > > > > > >> > > > delayed
> > > > > >>> > > > > > > > >> > > > > > > marker is appended to the
> > transaction
> > > > log
> > > > > >>> > > partition.
> > > > > >>> > > > > > > > >> > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once we move
> past
> > > the
> > > > > >>> Prepare
> > > > > >>> > > and
> > > > > >>> > > > > > > > Complete
> > > > > >>> > > > > > > > >> > > states,
> > > > > >>> > > > > > > > >> > > > > we
> > > > > >>> > > > > > > > >> > > > > > > don’t need to worry about lastSeen
> > > > fields
> > > > > >>> and
> > > > > >>> > > clear
> > > > > >>> > > > > > them,
> > > > > >>> > > > > > > > just
> > > > > >>> > > > > > > > >> > > handle
> > > > > >>> > > > > > > > >> > > > > > state
> > > > > >>> > > > > > > > >> > > > > > > transitions as normal.". Is the
> > > lastSeen
> > > > > >>> field
> > > > > >>> > the
> > > > > >>> > > > > same
> > > > > >>> > > > > > as
> > > > > >>> > > > > > > > the
> > > > > >>> > > > > > > > >> > > > previous
> > > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field in
> > > > > >>> TransactionLogValue?
> > > > > >>> > > > > > > > >> > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > 112. Since the kip changes the
> > > > > inter-broker
> > > > > >>> > > > protocol,
> > > > > >>> > > > > > > should
> > > > > >>> > > > > > > > >> we
> > > > > >>> > > > > > > > >> > > bump
> > > > > >>> > > > > > > > >> > > > up
> > > > > >>> > > > > > > > >> > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this feature
> only
> > > for
> > > > > the
> > > > > >>> > KRaft
> > > > > >>> > > > > mode?
> > > > > >>> > > > > > > > >> > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > > >>> > > > > > > > >> > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > Jun
> > > > > >>> > > > > > > > >> > > > > > >
> > > > > >>> > > > > > > > >> > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at 11:13 AM
> > > Justine
> > > > > >>> Olshan
> > > > > >>> > > > > > > > >> > > > > > > <jo...@confluent.io.invalid>
> > wrote:
> > > > > >>> > > > > > > > >> > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > > >>> > > > > > > > >> > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > I'm glad we are getting to
> > > convergence
> > > > > on
> > > > > >>> the
> > > > > >>> > > > > design.
> > > > > >>> > > > > > :)
> > > > > >>> > > > > > > > >> > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > While I understand it seems a
> > little
> > > > > >>> "weird".
> > > > > >>> > > I'm
> > > > > >>> > > > > not
> > > > > >>> > > > > > > sure
> > > > > >>> > > > > > > > >> what
> > > > > >>> > > > > > > > >> > > the
> > > > > >>> > > > > > > > >> > > > > > > benefit
> > > > > >>> > > > > > > > >> > > > > > > > of writing an extra record to
> the
> > > log.
> > > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool to
> describe
> > > > > >>> transactions
> > > > > >>> > > > won't
> > > > > >>> > > > > > > work
> > > > > >>> > > > > > > > >> (ie,
> > > > > >>> > > > > > > > >> > > the
> > > > > >>> > > > > > > > >> > > > > > > complete
> > > > > >>> > > > > > > > >> > > > > > > > state is needed to calculate the
> > > time
> > > > > >>> since
> > > > > >>> > the
> > > > > >>> > > > > > > > transaction
> > > > > >>> > > > > > > > >> > > > > completed?)
> > > > > >>> > > > > > > > >> > > > > > > > If we have a reason like this,
> it
> > is
> > > > > >>> enough to
> > > > > >>> > > > > > convince
> > > > > >>> > > > > > > me
> > > > > >>> > > > > > > > >> we
> > > > > >>> > > > > > > > >> > > need
> > > > > >>> > > > > > > > >> > > > > such
> > > > > >>> > > > > > > > >> > > > > > > an
> > > > > >>> > > > > > > > >> > > > > > > > extra record. It seems like it
> > would
> > > > be
> > > > > >>> > > replacing
> > > > > >>> > > > > the
> > > > > >>> > > > > > > > record
> > > > > >>> > > > > > > > >> > > > written
> > > > > >>> > > > > > > > >> > > > > on
> > > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is this correct?
> > > > > >>> > > > > > > > >> > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > > >>> > > > > > > > >> > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024 at 5:14 PM
> > Jun
> > > > Rao
> > > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > > >>> > > > > > > > >> > > >
> > > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > > >>> > > > > > > > >> > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > Thanks for the explanation. I
> > > > > >>> understand the
> > > > > >>> > > > > > intention
> > > > > >>> > > > > > > > >> now.
> > > > > >>> > > > > > > > >> > In
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > > >>> > > > > > > > >> > > > > > > > > case, we set the non-tagged
> > field
> > > to
> > > > > >>> the old
> > > > > >>> > > pid
> > > > > >>> > > > > > (and
> > > > > >>> > > > > > > > the
> > > > > >>> > > > > > > > >> max
> > > > > >>> > > > > > > > >> > > > > epoch)
> > > > > >>> > > > > > > > >> > > > > > in
> > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > prepare marker so that we
> could
> > > > > >>> correctly
> > > > > >>> > > write
> > > > > >>> > > > > the
> > > > > >>> > > > > > > > >> marker to
> > > > > >>> > > > > > > > >> > > the
> > > > > >>> > > > > > > > >> > > > > > data
> > > > > >>> > > > > > > > >> > > > > > > > > partition if the broker
> > > downgrades.
> > > > > When
> > > > > >>> > > writing
> > > > > >>> > > > > the
> > > > > >>> > > > > > > > >> complete
> > > > > >>> > > > > > > > >> > > > > marker,
> > > > > >>> > > > > > > > >> > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > know the marker has already
> been
> > > > > >>> written to
> > > > > >>> > > the
> > > > > >>> > > > > data
> > > > > >>> > > > > > > > >> > partition.
> > > > > >>> > > > > > > > >> > > > We
> > > > > >>> > > > > > > > >> > > > > > set
> > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to the new
> pid
> > to
> > > > > avoid
> > > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > > >>> > > > > > > > >> > > > > > in
> > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > client if the broker
> downgrades.
> > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > The above seems to work. It's
> > > just a
> > > > > bit
> > > > > >>> > > > > > inconsistent
> > > > > >>> > > > > > > > for
> > > > > >>> > > > > > > > >> a
> > > > > >>> > > > > > > > >> > > > prepare
> > > > > >>> > > > > > > > >> > > > > > > > marker
> > > > > >>> > > > > > > > >> > > > > > > > > and a complete marker to use
> > > > different
> > > > > >>> pids
> > > > > >>> > in
> > > > > >>> > > > > this
> > > > > >>> > > > > > > > >> special
> > > > > >>> > > > > > > > >> > > case.
> > > > > >>> > > > > > > > >> > > > > If
> > > > > >>> > > > > > > > >> > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > downgrade with the complete
> > > marker,
> > > > it
> > > > > >>> seems
> > > > > >>> > > > that
> > > > > >>> > > > > we
> > > > > >>> > > > > > > > will
> > > > > >>> > > > > > > > >> > never
> > > > > >>> > > > > > > > >> > > > be
> > > > > >>> > > > > > > > >> > > > > > able
> > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > >>> > > > > > > > >> > > > > > > > > write the complete marker with
> > the
> > > > old
> > > > > >>> pid.
> > > > > >>> > > Not
> > > > > >>> > > > > sure
> > > > > >>> > > > > > > if
> > > > > >>> > > > > > > > it
> > > > > >>> > > > > > > > >> > > causes
> > > > > >>> > > > > > > > >> > > > > any
> > > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems a bit
> weird.
> > > > > >>> Instead of
> > > > > >>> > > > > writing
> > > > > >>> > > > > > > the
> > > > > >>> > > > > > > > >> > > complete
> > > > > >>> > > > > > > > >> > > > > > marker
> > > > > >>> > > > > > > > >> > > > > > > > > with the new pid, could we
> write
> > > two
> > > > > >>> > records:
> > > > > >>> > > a
> > > > > >>> > > > > > > complete
> > > > > >>> > > > > > > > >> > marker
> > > > > >>> > > > > > > > >> > > > > with
> > > > > >>> > > > > > > > >> > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > old pid followed by a
> > > > > >>> TransactionLogValue
> > > > > >>> > with
> > > > > >>> > > > the
> > > > > >>> > > > > > new
> > > > > >>> > > > > > > > pid
> > > > > >>> > > > > > > > >> > and
> > > > > >>> > > > > > > > >> > > an
> > > > > >>> > > > > > > > >> > > > > > empty
> > > > > >>> > > > > > > > >> > > > > > > > > state? We could make the two
> > > records
> > > > > in
> > > > > >>> the
> > > > > >>> > > same
> > > > > >>> > > > > > batch
> > > > > >>> > > > > > > > so
> > > > > >>> > > > > > > > >> > that
> > > > > >>> > > > > > > > >> > > > they
> > > > > >>> > > > > > > > >> > > > > > > will
> > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > >>> > > > > > > > >> > > > > > > > > added to the log atomically.
> > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024 at
> 5:40 PM
> > > > > Justine
> > > > > >>> > Olshan
> > > > > >>> > > > > > > > >> > > > > > > > > <jolshan@confluent.io.invalid
> >
> > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare marker is
> > > written,
> > > > > >>> but the
> > > > > >>> > > > > endTxn
> > > > > >>> > > > > > > > >> response
> > > > > >>> > > > > > > > >> > is
> > > > > >>> > > > > > > > >> > > > not
> > > > > >>> > > > > > > > >> > > > > > > > > received
> > > > > >>> > > > > > > > >> > > > > > > > > > by the client when the
> server
> > > > > >>> downgrades
> > > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare marker is
> > > > written,
> > > > > >>> the
> > > > > >>> > > endTxn
> > > > > >>> > > > > > > > response
> > > > > >>> > > > > > > > >> is
> > > > > >>> > > > > > > > >> > > > > received
> > > > > >>> > > > > > > > >> > > > > > > by
> > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > client when the server
> > > downgrades.
> > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > I think I am still a little
> > > > > confused.
> > > > > >>> In
> > > > > >>> > > both
> > > > > >>> > > > of
> > > > > >>> > > > > > > these
> > > > > >>> > > > > > > > >> > cases,
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > transaction log has the old
> > > > producer
> > > > > >>> ID.
> > > > > >>> > We
> > > > > >>> > > > > don't
> > > > > >>> > > > > > > > write
> > > > > >>> > > > > > > > >> the
> > > > > >>> > > > > > > > >> > > new
> > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > > >>> > > > > > > > >> > > > > > > > > > in the prepare marker's non
> > > tagged
> > > > > >>> fields.
> > > > > >>> > > > > > > > >> > > > > > > > > > If the server downgrades
> now,
> > it
> > > > > would
> > > > > >>> > read
> > > > > >>> > > > the
> > > > > >>> > > > > > > > records
> > > > > >>> > > > > > > > >> not
> > > > > >>> > > > > > > > >> > > in
> > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > >>> > > > > > > > >> > > > > > > > > > fields and the complete
> marker
> > > > will
> > > > > >>> also
> > > > > >>> > > have
> > > > > >>> > > > > the
> > > > > >>> > > > > > > old
> > > > > >>> > > > > > > > >> > > producer
> > > > > >>> > > > > > > > >> > > > > ID.
> > > > > >>> > > > > > > > >> > > > > > > > > > (If we had used the new
> > producer
> > > > ID,
> > > > > >>> we
> > > > > >>> > > would
> > > > > >>> > > > > not
> > > > > >>> > > > > > > have
> > > > > >>> > > > > > > > >> > > > > > transactional
> > > > > >>> > > > > > > > >> > > > > > > > > > correctness since the
> producer
> > > id
> > > > > >>> doesn't
> > > > > >>> > > > match
> > > > > >>> > > > > > the
> > > > > >>> > > > > > > > >> > > transaction
> > > > > >>> > > > > > > > >> > > > > and
> > > > > >>> > > > > > > > >> > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > state would not be correct
> on
> > > the
> > > > > data
> > > > > >>> > > > > partition.)
> > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > In the overflow case, I'd
> > expect
> > > > the
> > > > > >>> > > following
> > > > > >>> > > > > to
> > > > > >>> > > > > > > > >> happen on
> > > > > >>> > > > > > > > >> > > the
> > > > > >>> > > > > > > > >> > > > > > > client
> > > > > >>> > > > > > > > >> > > > > > > > > side
> > > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we retry EndTxn
> --
> > it
> > > > is
> > > > > >>> the
> > > > > >>> > same
> > > > > >>> > > > > > > producer
> > > > > >>> > > > > > > > ID
> > > > > >>> > > > > > > > >> > and
> > > > > >>> > > > > > > > >> > > > > epoch
> > > > > >>> > > > > > > > >> > > > > > -
> > > > > >>> > > > > > > > >> > > > > > > 1
> > > > > >>> > > > > > > > >> > > > > > > > > this
> > > > > >>> > > > > > > > >> > > > > > > > > > would fence the producer
> > > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't retry
> > EndTxn
> > > > and
> > > > > >>> use
> > > > > >>> > the
> > > > > >>> > > > new
> > > > > >>> > > > > > > > >> producer id
> > > > > >>> > > > > > > > >> > > > which
> > > > > >>> > > > > > > > >> > > > > > > would
> > > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > > InvalidPidMappingException
> > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have special
> > > handling
> > > > > for
> > > > > >>> > when
> > > > > >>> > > a
> > > > > >>> > > > > > server
> > > > > >>> > > > > > > > >> > > > downgrades.
> > > > > >>> > > > > > > > >> > > > > > When
> > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > >>> > > > > > > > >> > > > > > > > > > reconnects we could get an
> API
> > > > > version
> > > > > >>> > > request
> > > > > >>> > > > > > > showing
> > > > > >>> > > > > > > > >> > > KIP-890
> > > > > >>> > > > > > > > >> > > > > > part 2
> > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > not supported. In that case,
> > we
> > > > can
> > > > > >>> call
> > > > > >>> > > > > > > > initProducerId
> > > > > >>> > > > > > > > >> to
> > > > > >>> > > > > > > > >> > > > abort
> > > > > >>> > > > > > > > >> > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In the
> overflow
> > > > case,
> > > > > >>> this
> > > > > >>> > > > > correctly
> > > > > >>> > > > > > > > gives
> > > > > >>> > > > > > > > >> > us a
> > > > > >>> > > > > > > > >> > > > new
> > > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > I guess the corresponding
> case
> > > > would
> > > > > >>> be
> > > > > >>> > > where
> > > > > >>> > > > > the
> > > > > >>> > > > > > > > >> *complete
> > > > > >>> > > > > > > > >> > > > > marker
> > > > > >>> > > > > > > > >> > > > > > > *is
> > > > > >>> > > > > > > > >> > > > > > > > > > written but the endTxn is
> not
> > > > > >>> received by
> > > > > >>> > > the
> > > > > >>> > > > > > client
> > > > > >>> > > > > > > > and
> > > > > >>> > > > > > > > >> > the
> > > > > >>> > > > > > > > >> > > > > server
> > > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This would
> result
> > in
> > > > the
> > > > > >>> > > > transaction
> > > > > >>> > > > > > > > >> > coordinator
> > > > > >>> > > > > > > > >> > > > > having
> > > > > >>> > > > > > > > >> > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > new
> > > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old one.  If
> > the
> > > > > client
> > > > > >>> > > > retries,
> > > > > >>> > > > > it
> > > > > >>> > > > > > > > will
> > > > > >>> > > > > > > > >> > > receive
> > > > > >>> > > > > > > > >> > > > > an
> > > > > >>> > > > > > > > >> > > > > > > > > > InvalidPidMappingException.
> > The
> > > > > >>> > > InitProducerId
> > > > > >>> > > > > > > > scenario
> > > > > >>> > > > > > > > >> > above
> > > > > >>> > > > > > > > >> > > > > would
> > > > > >>> > > > > > > > >> > > > > > > > help
> > > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my
> compatibility
> > > > story
> > > > > is
> > > > > >>> > meant
> > > > > >>> > > > to
> > > > > >>> > > > > > > > support
> > > > > >>> > > > > > > > >> > > > > downgrades
> > > > > >>> > > > > > > > >> > > > > > > > server
> > > > > >>> > > > > > > > >> > > > > > > > > > side in keeping the
> > > transactional
> > > > > >>> > > correctness.
> > > > > >>> > > > > > > Keeping
> > > > > >>> > > > > > > > >> the
> > > > > >>> > > > > > > > >> > > > client
> > > > > >>> > > > > > > > >> > > > > > > from
> > > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is not the
> > > > priority.
> > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I can also
> > add
> > > > text
> > > > > >>> in
> > > > > >>> > the
> > > > > >>> > > > KIP
> > > > > >>> > > > > > > about
> > > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > > >>> > > > > > > > >> > > > > > > if
> > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > think that fixes some edge
> > > cases.
> > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12, 2024 at
> > 4:10 PM
> > > > Jun
> > > > > >>> Rao
> > > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > > >>> > > > > > > > >> > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the reply.
> > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we don't need
> > to
> > > > > >>> optimize
> > > > > >>> > for
> > > > > >>> > > > > > fencing
> > > > > >>> > > > > > > > >> during
> > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > >>> > > > > > > > >> > > > > > > > > > > Regarding consistency,
> there
> > > are
> > > > > two
> > > > > >>> > > > possible
> > > > > >>> > > > > > > cases:
> > > > > >>> > > > > > > > >> (1)
> > > > > >>> > > > > > > > >> > > the
> > > > > >>> > > > > > > > >> > > > > > > prepare
> > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > >>> > > > > > > > >> > > > > > > > > > > is written, but the endTxn
> > > > > response
> > > > > >>> is
> > > > > >>> > not
> > > > > >>> > > > > > > received
> > > > > >>> > > > > > > > by
> > > > > >>> > > > > > > > >> > the
> > > > > >>> > > > > > > > >> > > > > client
> > > > > >>> > > > > > > > >> > > > > > > > when
> > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > server downgrades; (2)
> the
> > > > > prepare
> > > > > >>> > marker
> > > > > >>> > > > is
> > > > > >>> > > > > > > > written,
> > > > > >>> > > > > > > > >> > the
> > > > > >>> > > > > > > > >> > > > > endTxn
> > > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > > >>> > > > > > > > >> > > > > > > > > > > is received by the client
> > when
> > > > the
> > > > > >>> > server
> > > > > >>> > > > > > > > downgrades.
> > > > > >>> > > > > > > > >> In
> > > > > >>> > > > > > > > >> > > (1),
> > > > > >>> > > > > > > > >> > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > client
> > > > > >>> > > > > > > > >> > > > > > > > > > > will have the old produce
> Id
> > > and
> > > > > in
> > > > > >>> (2),
> > > > > >>> > > the
> > > > > >>> > > > > > > client
> > > > > >>> > > > > > > > >> will
> > > > > >>> > > > > > > > >> > > have
> > > > > >>> > > > > > > > >> > > > > the
> > > > > >>> > > > > > > > >> > > > > > > new
> > > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If we
> downgrade
> > > > right
> > > > > >>> after
> > > > > >>> > > the
> > > > > >>> > > > > > > prepare
> > > > > >>> > > > > > > > >> > marker,
> > > > > >>> > > > > > > > >> > > > we
> > > > > >>> > > > > > > > >> > > > > > > can't
> > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > >>> > > > > > > > >> > > > > > > > > > > consistent to both (1) and
> > (2)
> > > > > >>> since we
> > > > > >>> > > can
> > > > > >>> > > > > only
> > > > > >>> > > > > > > put
> > > > > >>> > > > > > > > >> one
> > > > > >>> > > > > > > > >> > > > value
> > > > > >>> > > > > > > > >> > > > > in
> > > > > >>> > > > > > > > >> > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > existing produce Id field.
> > > It's
> > > > > >>> also not
> > > > > >>> > > > clear
> > > > > >>> > > > > > > which
> > > > > >>> > > > > > > > >> case
> > > > > >>> > > > > > > > >> > > is
> > > > > >>> > > > > > > > >> > > > > more
> > > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > > >>> > > > > > > > >> > > > > > > > > > > So we could probably be
> > > > consistent
> > > > > >>> with
> > > > > >>> > > > either
> > > > > >>> > > > > > > case.
> > > > > >>> > > > > > > > >> By
> > > > > >>> > > > > > > > >> > > > putting
> > > > > >>> > > > > > > > >> > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > new
> > > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in the prepare
> > > > marker,
> > > > > >>> we
> > > > > >>> > are
> > > > > >>> > > > > > > consistent
> > > > > >>> > > > > > > > >> with
> > > > > >>> > > > > > > > >> > > > case
> > > > > >>> > > > > > > > >> > > > > > (2)
> > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > > it
> > > > > >>> > > > > > > > >> > > > > > > > > > > also has the slight
> benefit
> > > that
> > > > > the
> > > > > >>> > > produce
> > > > > >>> > > > > > field
> > > > > >>> > > > > > > > in
> > > > > >>> > > > > > > > >> the
> > > > > >>> > > > > > > > >> > > > > prepare
> > > > > >>> > > > > > > > >> > > > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > > > > complete marker are
> > consistent
> > > > in
> > > > > >>> the
> > > > > >>> > > > overflow
> > > > > >>> > > > > > > case.
> > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12, 2024 at
> > > 3:11 PM
> > > > > >>> Justine
> > > > > >>> > > > Olshan
> > > > > >>> > > > > > > > >> > > > > > > > > > >
> > <jolshan@confluent.io.invalid
> > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > In the case you
> describe,
> > we
> > > > > would
> > > > > >>> > need
> > > > > >>> > > to
> > > > > >>> > > > > > have
> > > > > >>> > > > > > > a
> > > > > >>> > > > > > > > >> > delayed
> > > > > >>> > > > > > > > >> > > > > > > request,
> > > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > > >>> > > > > > > > >> > > > > > > > > > > > successful EndTxn, and a
> > > > > >>> successful
> > > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > > >>> > > > > > > > >> > > and
> > > > > >>> > > > > > > > >> > > > > then
> > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn request
> go
> > > > > through
> > > > > >>> for
> > > > > >>> > a
> > > > > >>> > > > > given
> > > > > >>> > > > > > > > >> producer.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to figure out
> > if
> > > it
> > > > > is
> > > > > >>> > > possible
> > > > > >>> > > > > for
> > > > > >>> > > > > > > the
> > > > > >>> > > > > > > > >> > client
> > > > > >>> > > > > > > > >> > > > to
> > > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > > >>> > > > > > > > >> > > > > > > > > > > > a previous request is
> > > delayed
> > > > > >>> > somewhere.
> > > > > >>> > > > But
> > > > > >>> > > > > > > yes,
> > > > > >>> > > > > > > > in
> > > > > >>> > > > > > > > >> > this
> > > > > >>> > > > > > > > >> > > > > case
> > > > > >>> > > > > > > > >> > > > > > I
> > > > > >>> > > > > > > > >> > > > > > > > > think
> > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > > > would fence the client.
> > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the overflow
> case.
> > > In
> > > > > the
> > > > > >>> > > overflow
> > > > > >>> > > > > > case,
> > > > > >>> > > > > > > > the
> > > > > >>> > > > > > > > >> > > > producer
> > > > > >>> > > > > > > > >> > > > > > ID
> > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are different on
> the
> > > > > marker
> > > > > >>> and
> > > > > >>> > on
> > > > > >>> > > > the
> > > > > >>> > > > > > new
> > > > > >>> > > > > > > > >> > > > transaction.
> > > > > >>> > > > > > > > >> > > > > > So
> > > > > >>> > > > > > > > >> > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to use the
> max
> > > > epoch
> > > > > >>> but
> > > > > >>> > the
> > > > > >>> > > > new
> > > > > >>> > > > > > > > >> > transaction
> > > > > >>> > > > > > > > >> > > > > should
> > > > > >>> > > > > > > > >> > > > > > > > start
> > > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and epoch 0
> in
> > > the
> > > > > >>> > > > transactional
> > > > > >>> > > > > > > state.
> > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > In the server downgrade
> > > case,
> > > > we
> > > > > >>> want
> > > > > >>> > to
> > > > > >>> > > > see
> > > > > >>> > > > > > the
> > > > > >>> > > > > > > > >> > producer
> > > > > >>> > > > > > > > >> > > > ID
> > > > > >>> > > > > > > > >> > > > > as
> > > > > >>> > > > > > > > >> > > > > > > > that
> > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > > > what the client will
> have.
> > > If
> > > > we
> > > > > >>> > > complete
> > > > > >>> > > > > the
> > > > > >>> > > > > > > > >> commit,
> > > > > >>> > > > > > > > >> > and
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > > >>> > > > > > > > >> > > > > > > > > > > > state is reloaded, we
> need
> > > the
> > > > > new
> > > > > >>> > > > producer
> > > > > >>> > > > > ID
> > > > > >>> > > > > > > in
> > > > > >>> > > > > > > > >> the
> > > > > >>> > > > > > > > >> > > state
> > > > > >>> > > > > > > > >> > > > > so
> > > > > >>> > > > > > > > >> > > > > > > > there
> > > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid producer ID
> > > > mapping.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > The server downgrade
> cases
> > > are
> > > > > >>> > > considering
> > > > > >>> > > > > > > > >> > transactional
> > > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing from
> > previous
> > > > > >>> behavior
> > > > > >>> > --
> > > > > >>> > > > and
> > > > > >>> > > > > > are
> > > > > >>> > > > > > > > not
> > > > > >>> > > > > > > > >> > > > > concerned
> > > > > >>> > > > > > > > >> > > > > > > > about
> > > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the safety
> from
> > > > > fencing
> > > > > >>> > > retries
> > > > > >>> > > > > (as
> > > > > >>> > > > > > > we
> > > > > >>> > > > > > > > >> have
> > > > > >>> > > > > > > > >> > > > > > downgraded
> > > > > >>> > > > > > > > >> > > > > > > > so
> > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to support).
> > > > Perhaps
> > > > > >>> this
> > > > > >>> > is
> > > > > >>> > > a
> > > > > >>> > > > > > trade
> > > > > >>> > > > > > > > off,
> > > > > >>> > > > > > > > >> > but
> > > > > >>> > > > > > > > >> > > I
> > > > > >>> > > > > > > > >> > > > > > think
> > > > > >>> > > > > > > > >> > > > > > > it
> > > > > >>> > > > > > > > >> > > > > > > > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client
> downgrades,
> > > it
> > > > > will
> > > > > >>> > have
> > > > > >>> > > > > > > restarted
> > > > > >>> > > > > > > > >> and
> > > > > >>> > > > > > > > >> > it
> > > > > >>> > > > > > > > >> > > is
> > > > > >>> > > > > > > > >> > > > > ok
> > > > > >>> > > > > > > > >> > > > > > > for
> > > > > >>> > > > > > > > >> > > > > > > > it
> > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > >>> > > > > > > > >> > > > > > > > > > > > have a new producer ID
> > too).
> > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan 12, 2024 at
> > > > 11:42 AM
> > > > > >>> Jun
> > > > > >>> > Rao
> > > > > >>> > > > > > > > >> > > > > > > <jun@confluent.io.invalid
> > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for the reply.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If the marker
> is
> > > > > written
> > > > > >>> by
> > > > > >>> > the
> > > > > >>> > > > new
> > > > > >>> > > > > > > > >> client, we
> > > > > >>> > > > > > > > >> > > can
> > > > > >>> > > > > > > > >> > > > > as
> > > > > >>> > > > > > > > >> > > > > > I
> > > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > the last email
> guarantee
> > > > that
> > > > > >>> any
> > > > > >>> > > EndTxn
> > > > > >>> > > > > > > > requests
> > > > > >>> > > > > > > > >> > with
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > > same
> > > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > from the same producer
> > and
> > > > the
> > > > > >>> same
> > > > > >>> > > > > > > transaction.
> > > > > >>> > > > > > > > >> Then
> > > > > >>> > > > > > > > >> > > we
> > > > > >>> > > > > > > > >> > > > > > don't
> > > > > >>> > > > > > > > >> > > > > > > > have
> > > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > return a fenced error
> > but
> > > > can
> > > > > >>> handle
> > > > > >>> > > > > > > gracefully
> > > > > >>> > > > > > > > as
> > > > > >>> > > > > > > > >> > > > > described
> > > > > >>> > > > > > > > >> > > > > > in
> > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > When a delayed EndTnx
> > > > request
> > > > > is
> > > > > >>> > > > > processed,
> > > > > >>> > > > > > > the
> > > > > >>> > > > > > > > >> txn
> > > > > >>> > > > > > > > >> > > state
> > > > > >>> > > > > > > > >> > > > > > could
> > > > > >>> > > > > > > > >> > > > > > > > be
> > > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next txn. I
> > guess
> > > in
> > > > > >>> this
> > > > > >>> > case
> > > > > >>> > > > we
> > > > > >>> > > > > > > still
> > > > > >>> > > > > > > > >> > return
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > > > fenced
> > > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed request?
> > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry, my
> question
> > > was
> > > > > >>> > > inaccurate.
> > > > > >>> > > > > What
> > > > > >>> > > > > > > you
> > > > > >>> > > > > > > > >> > > > described
> > > > > >>> > > > > > > > >> > > > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > "The downgrade
> > > > compatibility I
> > > > > >>> > mention
> > > > > >>> > > > is
> > > > > >>> > > > > > that
> > > > > >>> > > > > > > > we
> > > > > >>> > > > > > > > >> > keep
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > > same
> > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in the main
> > > > > >>> (non-tagged)
> > > > > >>> > > > fields
> > > > > >>> > > > > as
> > > > > >>> > > > > > > we
> > > > > >>> > > > > > > > >> did
> > > > > >>> > > > > > > > >> > > > before
> > > > > >>> > > > > > > > >> > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > code
> > > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > server side." If we
> want
> > > to
> > > > do
> > > > > >>> this,
> > > > > >>> > > it
> > > > > >>> > > > > > seems
> > > > > >>> > > > > > > > >> that we
> > > > > >>> > > > > > > > >> > > > > should
> > > > > >>> > > > > > > > >> > > > > > > use
> > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > current produce Id and
> > max
> > > > > >>> epoch in
> > > > > >>> > > the
> > > > > >>> > > > > > > existing
> > > > > >>> > > > > > > > >> > > > producerId
> > > > > >>> > > > > > > > >> > > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerEpoch fields
> for
> > > > both
> > > > > >>> the
> > > > > >>> > > > prepare
> > > > > >>> > > > > > and
> > > > > >>> > > > > > > > the
> > > > > >>> > > > > > > > >> > > > complete
> > > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > The downgrade can
> happen
> > > > after
> > > > > >>> the
> > > > > >>> > > > > complete
> > > > > >>> > > > > > > > >> marker is
> > > > > >>> > > > > > > > >> > > > > > written.
> > > > > >>> > > > > > > > >> > > > > > > > With
> > > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > you described, the
> > > > downgraded
> > > > > >>> > > > coordinator
> > > > > >>> > > > > > will
> > > > > >>> > > > > > > > see
> > > > > >>> > > > > > > > >> > the
> > > > > >>> > > > > > > > >> > > > new
> > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of the old
> one.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan 12, 2024
> at
> > > > > 10:44 AM
> > > > > >>> > > Justine
> > > > > >>> > > > > > > Olshan
> > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > <jolshan@confluent.io.invalid
> > > > > >
> > > > > >>> > wrote:
> > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can update the
> > > > > description.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe your
> second
> > > > point
> > > > > is
> > > > > >>> > > > mentioned
> > > > > >>> > > > > > in
> > > > > >>> > > > > > > > the
> > > > > >>> > > > > > > > >> > KIP.
> > > > > >>> > > > > > > > >> > > I
> > > > > >>> > > > > > > > >> > > > > can
> > > > > >>> > > > > > > > >> > > > > > > add
> > > > > >>> > > > > > > > >> > > > > > > > > more
> > > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it is
> helpful.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The delayed
> message
> > > case
> > > > > can
> > > > > >>> > also
> > > > > >>> > > > > > violate
> > > > > >>> > > > > > > > EOS
> > > > > >>> > > > > > > > >> if
> > > > > >>> > > > > > > > >> > > the
> > > > > >>> > > > > > > > >> > > > > > > delayed
> > > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in after the
> > next
> > > > > >>> > > > > addPartitionsToTxn
> > > > > >>> > > > > > > > >> request
> > > > > >>> > > > > > > > >> > > > comes
> > > > > >>> > > > > > > > >> > > > > > in.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a message
> > from a
> > > > > >>> previous
> > > > > >>> > > > > > (aborted)
> > > > > >>> > > > > > > > >> > > transaction
> > > > > >>> > > > > > > > >> > > > > > > become
> > > > > >>> > > > > > > > >> > > > > > > > > part
> > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > next transaction.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the marker is
> > written
> > > > by
> > > > > >>> the
> > > > > >>> > new
> > > > > >>> > > > > > client,
> > > > > >>> > > > > > > we
> > > > > >>> > > > > > > > >> can
> > > > > >>> > > > > > > > >> > > as I
> > > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email guarantee
> > > that
> > > > > any
> > > > > >>> > EndTxn
> > > > > >>> > > > > > > requests
> > > > > >>> > > > > > > > >> with
> > > > > >>> > > > > > > > >> > > the
> > > > > >>> > > > > > > > >> > > > > same
> > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same producer
> and
> > > the
> > > > > same
> > > > > >>> > > > > > transaction.
> > > > > >>> > > > > > > > >> Then we
> > > > > >>> > > > > > > > >> > > > don't
> > > > > >>> > > > > > > > >> > > > > > > have
> > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced error but can
> > > > handle
> > > > > >>> > > gracefully
> > > > > >>> > > > > as
> > > > > >>> > > > > > > > >> described
> > > > > >>> > > > > > > > >> > > in
> > > > > >>> > > > > > > > >> > > > > the
> > > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't think a
> > boolean
> > > is
> > > > > >>> useful
> > > > > >>> > > > since
> > > > > >>> > > > > it
> > > > > >>> > > > > > > is
> > > > > >>> > > > > > > > >> > > directly
> > > > > >>> > > > > > > > >> > > > > > > encoded
> > > > > >>> > > > > > > > >> > > > > > > > by
> > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence or lack of
> > the
> > > > > >>> tagged
> > > > > >>> > > field
> > > > > >>> > > > > > being
> > > > > >>> > > > > > > > >> > written.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the prepare
> marker
> > we
> > > > > will
> > > > > >>> have
> > > > > >>> > > the
> > > > > >>> > > > > > same
> > > > > >>> > > > > > > > >> > producer
> > > > > >>> > > > > > > > >> > > ID
> > > > > >>> > > > > > > > >> > > > > in
> > > > > >>> > > > > > > > >> > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In the
> Complete
> > > > state
> > > > > >>> we
> > > > > >>> > may
> > > > > >>> > > > not.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure why the
> > > > ongoing
> > > > > >>> state
> > > > > >>> > > > > matters
> > > > > >>> > > > > > > for
> > > > > >>> > > > > > > > >> this
> > > > > >>> > > > > > > > >> > > > KIP.
> > > > > >>> > > > > > > > >> > > > > It
> > > > > >>> > > > > > > > >> > > > > > > > does
> > > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure what
> you
> > > are
> > > > > >>> > referring
> > > > > >>> > > to
> > > > > >>> > > > > > about
> > > > > >>> > > > > > > > >> > writing
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > > > > previous
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the prepare
> > > marker.
> > > > > >>> This is
> > > > > >>> > > not
> > > > > >>> > > > in
> > > > > >>> > > > > > the
> > > > > >>> > > > > > > > >> KIP.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the overflow
> case,
> > we
> > > > > >>> write the
> > > > > >>> > > > > > > > >> nextProducerId
> > > > > >>> > > > > > > > >> > in
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so we know
> > what
> > > we
> > > > > >>> > assigned
> > > > > >>> > > > when
> > > > > >>> > > > > > we
> > > > > >>> > > > > > > > >> reload
> > > > > >>> > > > > > > > >> > > the
> > > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we complete, we
> > > > > >>> transition
> > > > > >>> > this
> > > > > >>> > > > ID
> > > > > >>> > > > > to
> > > > > >>> > > > > > > the
> > > > > >>> > > > > > > > >> main
> > > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the previous
> > > producer
> > > > > ID
> > > > > >>> > field
> > > > > >>> > > > > filled
> > > > > >>> > > > > > > in.
> > > > > >>> > > > > > > > >> This
> > > > > >>> > > > > > > > >> > > is
> > > > > >>> > > > > > > > >> > > > so
> > > > > >>> > > > > > > > >> > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > can
> > > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry case the
> > > > > operation
> > > > > >>> > > > completed
> > > > > >>> > > > > > > > >> > successfully
> > > > > >>> > > > > > > > >> > > > and
> > > > > >>> > > > > > > > >> > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > don't
> > > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer. The
> > downgrade
> > > > > >>> > > compatibility
> > > > > >>> > > > I
> > > > > >>> > > > > > > > mention
> > > > > >>> > > > > > > > >> is
> > > > > >>> > > > > > > > >> > > that
> > > > > >>> > > > > > > > >> > > > > we
> > > > > >>> > > > > > > > >> > > > > > > keep
> > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer ID and
> epoch
> > in
> > > > the
> > > > > >>> main
> > > > > >>> > > > > > > (non-tagged)
> > > > > >>> > > > > > > > >> > fields
> > > > > >>> > > > > > > > >> > > > as
> > > > > >>> > > > > > > > >> > > > > we
> > > > > >>> > > > > > > > >> > > > > > > did
> > > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on the server
> > side.
> > > > If
> > > > > >>> the
> > > > > >>> > > server
> > > > > >>> > > > > > > > >> downgrades,
> > > > > >>> > > > > > > > >> > we
> > > > > >>> > > > > > > > >> > > > are
> > > > > >>> > > > > > > > >> > > > > > > still
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This addresses both
> > the
> > > > > >>> prepare
> > > > > >>> > and
> > > > > >>> > > > > > complete
> > > > > >>> > > > > > > > >> state
> > > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri, Jan 12, 2024
> > at
> > > > > >>> 10:21 AM
> > > > > >>> > Jun
> > > > > >>> > > > Rao
> > > > > >>> > > > > > > > >> > > > > > > > > <jun@confluent.io.invalid
> > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi, Justine,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks for the
> > reply.
> > > > > Sorry
> > > > > >>> for
> > > > > >>> > > the
> > > > > >>> > > > > > > delay. I
> > > > > >>> > > > > > > > >> > have a
> > > > > >>> > > > > > > > >> > > > few
> > > > > >>> > > > > > > > >> > > > > > > more
> > > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I think the
> > > > > motivation
> > > > > >>> > > section
> > > > > >>> > > > > > could
> > > > > >>> > > > > > > be
> > > > > >>> > > > > > > > >> > > > improved.
> > > > > >>> > > > > > > > >> > > > > > One
> > > > > >>> > > > > > > > >> > > > > > > of
> > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > motivations listed
> > by
> > > > the
> > > > > >>> KIP is
> > > > > >>> > > > "This
> > > > > >>> > > > > > can
> > > > > >>> > > > > > > > >> happen
> > > > > >>> > > > > > > > >> > > > when
> > > > > >>> > > > > > > > >> > > > > a
> > > > > >>> > > > > > > > >> > > > > > > > > message
> > > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or delayed due to
> > > > > networking
> > > > > >>> > > issues
> > > > > >>> > > > > or a
> > > > > >>> > > > > > > > >> network
> > > > > >>> > > > > > > > >> > > > > > partition,
> > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > transaction
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts, and then
> the
> > > > > delayed
> > > > > >>> > > message
> > > > > >>> > > > > > > finally
> > > > > >>> > > > > > > > >> > comes
> > > > > >>> > > > > > > > >> > > > > in.".
> > > > > >>> > > > > > > > >> > > > > > > This
> > > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very accurate.
> > Without
> > > > > >>> KIP-890,
> > > > > >>> > > > > > currently,
> > > > > >>> > > > > > > > if
> > > > > >>> > > > > > > > >> the
> > > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts an ongoing
> > > > > >>> transaction,
> > > > > >>> > it
> > > > > >>> > > > > > already
> > > > > >>> > > > > > > > >> bumps
> > > > > >>> > > > > > > > >> > up
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > > > epoch
> > > > > >>> > > > > > > > >> > > > > > > > in
> > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which prevents the
> > > > delayed
> > > > > >>> > produce
> > > > > >>> > > > > > message
> > > > > >>> > > > > > > > >> from
> > > > > >>> > > > > > > > >> > > being
> > > > > >>> > > > > > > > >> > > > > > added
> > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > partition. What
> can
> > > > cause
> > > > > a
> > > > > >>> > > hanging
> > > > > >>> > > > > > > > >> transaction
> > > > > >>> > > > > > > > >> > is
> > > > > >>> > > > > > > > >> > > > that
> > > > > >>> > > > > > > > >> > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completes (either
> > > aborts
> > > > > or
> > > > > >>> > > > commits) a
> > > > > >>> > > > > > > > >> > transaction
> > > > > >>> > > > > > > > >> > > > > before
> > > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > successful ack on
> > > > messages
> > > > > >>> > > published
> > > > > >>> > > > > in
> > > > > >>> > > > > > > the
> > > > > >>> > > > > > > > >> same
> > > > > >>> > > > > > > > >> > > txn.
> > > > > >>> > > > > > > > >> > > > > In
> > > > > >>> > > > > > > > >> > > > > > > this
> > > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible for the
> > > delayed
> > > > > >>> message
> > > > > >>> > > to
> > > > > >>> > > > be
> > > > > >>> > > > > > > > >> appended
> > > > > >>> > > > > > > > >> > to
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker, causing a
> > > > > >>> transaction to
> > > > > >>> > > > hang.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A similar issue
> (not
> > > > > >>> mentioned
> > > > > >>> > in
> > > > > >>> > > > the
> > > > > >>> > > > > > > > >> motivation)
> > > > > >>> > > > > > > > >> > > > could
> > > > > >>> > > > > > > > >> > > > > > > > happen
> > > > > >>> > > > > > > > >> > > > > > > > > on
> > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker in the
> > > > > coordinator's
> > > > > >>> log.
> > > > > >>> > > For
> > > > > >>> > > > > > > > example,
> > > > > >>> > > > > > > > >> > it's
> > > > > >>> > > > > > > > >> > > > > > possible
> > > > > >>> > > > > > > > >> > > > > > > > for
> > > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest to
> be
> > > > > delayed
> > > > > >>> on
> > > > > >>> > the
> > > > > >>> > > > > > > > >> coordinator.
> > > > > >>> > > > > > > > >> > By
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > > time
> > > > > >>> > > > > > > > >> > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest is
> > > > > processed,
> > > > > >>> it's
> > > > > >>> > > > > > possible
> > > > > >>> > > > > > > > that
> > > > > >>> > > > > > > > >> > the
> > > > > >>> > > > > > > > >> > > > > > previous
> > > > > >>> > > > > > > > >> > > > > > > > txn
> > > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completed and a
> new
> > > txn
> > > > > has
> > > > > >>> > > started.
> > > > > >>> > > > > > > > >> Currently,
> > > > > >>> > > > > > > > >> > > since
> > > > > >>> > > > > > > > >> > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped on every
> txn,
> > > the
> > > > > >>> delayed
> > > > > >>> > > > > > > > EndTxnRequest
> > > > > >>> > > > > > > > >> > will
> > > > > >>> > > > > > > > >> > > > add
> > > > > >>> > > > > > > > >> > > > > > an
> > > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare marker
> (and
> > > > > >>> eventually a
> > > > > >>> > > > > > complete
> > > > > >>> > > > > > > > >> marker)
> > > > > >>> > > > > > > > >> > > to
> > > > > >>> > > > > > > > >> > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't cause the
> > > > > transaction
> > > > > >>> to
> > > > > >>> > > hang,
> > > > > >>> > > > > but
> > > > > >>> > > > > > > it
> > > > > >>> > > > > > > > >> will
> > > > > >>> > > > > > > > >> > > > break
> > > > > >>> > > > > > > > >> > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The proposal in
> this
> > > KIP
> > > > > >>> will
> > > > > >>> > > > address
> > > > > >>> > > > > > this
> > > > > >>> > > > > > > > >> issue
> > > > > >>> > > > > > > > >> > > too.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101. "However, I
> was
> > > > > >>> writing it
> > > > > >>> > so
> > > > > >>> > > > > that
> > > > > >>> > > > > > we
> > > > > >>> > > > > > > > can
> > > > > >>> > > > > > > > >> > > > > > distinguish
> > > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old clients where
> we
> > > > don't
> > > > > >>> have
> > > > > >>> > > the
> > > > > >>> > > > > > > ability
> > > > > >>> > > > > > > > do
> > > > > >>> > > > > > > > >> > this
> > > > > >>> > > > > > > > >> > > > > > > operation
> > > > > >>> > > > > > > > >> > > > > > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients that can.
> > (Old
> > > > > >>> clients
> > > > > >>> > > don't
> > > > > >>> > > > > > bump
> > > > > >>> > > > > > > > the
> > > > > >>> > > > > > > > >> > epoch
> > > > > >>> > > > > > > > >> > > > on
> > > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > > >>> > > > > > > > >> > > > > > > > > so
> > > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for sure the
> > write
> > > > > >>> belongs
> > > > > >>> > to
> > > > > >>> > > > the
> > > > > >>> > > > > > > given
> > > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I am
> wondering
> > > why
> > > > > we
> > > > > >>> need
> > > > > >>> > > to
> > > > > >>> > > > > > > > >> distinguish
> > > > > >>> > > > > > > > >> > > > whether
> > > > > >>> > > > > > > > >> > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written by the old
> > and
> > > > the
> > > > > >>> new
> > > > > >>> > > > client.
> > > > > >>> > > > > > > Could
> > > > > >>> > > > > > > > >> you
> > > > > >>> > > > > > > > >> > > > > describe
> > > > > >>> > > > > > > > >> > > > > > > > what
> > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > differently if we
> > know
> > > > the
> > > > > >>> > marker
> > > > > >>> > > is
> > > > > >>> > > > > > > written
> > > > > >>> > > > > > > > >> by
> > > > > >>> > > > > > > > >> > the
> > > > > >>> > > > > > > > >> > > > new
> > > > > >>> > > > > > > > >> > > > > > > > client?
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If we do
> need
> > a
> > > > way
> > > > > to
> > > > > >>> > > > > distinguish
> > > > > >>> > > > > > > > >> whether
> > > > > >>> > > > > > > > >> > > the
> > > > > >>> > > > > > > > >> > > > > > marker
> > > > > >>> > > > > > > > >> > > > > > > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old and the
> new
> > > > > client.
> > > > > >>> > Would
> > > > > >>> > > it
> > > > > >>> > > > > be
> > > > > >>> > > > > > > > >> simpler
> > > > > >>> > > > > > > > >> > to
> > > > > >>> > > > > > > > >> > > > just
> > > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field instead of
> > > > > indirectly
> > > > > >>> > > through
> > > > > >>> > > > > the
> > > > > >>> > > > > > > > >> previous
> > > > > >>> > > > > > > > >> > > > > produce
> > > > > >>> > > > > > > > >> > > > > > ID
> > > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3 It's not
> clear
> > > to
> > > > me
> > > > > >>> why
> > > > > >>> > we
> > > > > >>> > > > only
> > > > > >>> > > > > > add
> > > > > >>> > > > > > > > the
> > > > > >>> > > > > > > > >> > > > previous
> > > > > >>> > > > > > > > >> > > > > > > > produce
> > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the complete
> marker,
> > > but
> > > > > >>> not in
> > > > > >>> > > the
> > > > > >>> > > > > > > prepare
> > > > > >>> > > > > > > > >> > marker.
> > > > > >>> > > > > > > > >> > > > If
> > > > > >>> > > > > > > > >> > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > want
> > > > > >>> > > > > > > > >> > > > > > > > > to
> > > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether a marker
> is
> > > > > written
> > > > > >>> by
> > > > > >>> > the
> > > > > >>> > > > new
> > > > > >>> > > > > > > > client
> > > > > >>> > > > > > > > >> or
> > > > > >>> > > > > > > > >> > > not,
> > > > > >>> > > > > > > > >> > > > > it
> > > > > >>> > > > > > > > >> > > > > > > > seems
> > > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do this
> > > consistently
> > > > > for
> > > > > >>> all
> > > > > >>> > > > > markers.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4 What about
> the
> > > > > >>> > > > > TransactionLogValue
> > > > > >>> > > > > > > > >> record
> > > > > >>> > > > > > > > >> > > > > > > representing
> > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state? Should we
> > also
> > > > > >>> > distinguish
> > > > > >>> > > > > > whether
> > > > > >>> > > > > > > > it's
> > > > > >>> > > > > > > > >> > > > written
> > > > > >>> > > > > > > > >> > > > > by
> > > > > >>> > > > > > > > >> > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > old
> > > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new client?
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In the
> overflow
> > > > case,
> > > > > >>> it's
> > > > > >>> > > > still
> > > > > >>> > > > > > not
> > > > > >>> > > > > > > > >> clear
> > > > > >>> > > > > > > > >> > to
> > > > > >>> > > > > > > > >> > > me
> > > > > >>> > > > > > > > >> > > > > why
> > > > > >>> > > > > > > > >> > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous produce
> Id
> > in
> > > > the
> > > > > >>> > prepare
> > > > > >>> > > > > > marker
> > > > > >>> > > > > > > > >> while
> > > > > >>> > > > > > > > >> > > > writing
> > > > > >>> > > > > > > > >> > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the complete
> > > marker.
> > > > > You
> > > > > >>> > > > mentioned
> > > > > >>> > > > > > that
> > > > > >>> > > > > > > > >> it's
> > > > > >>> > > > > > > > >> > for
> > > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could downgrade
> > > with
> > > > > >>> either
> > > > > >>> > the
> > > > > >>> > > > > > prepare
> > > > > >>> > > > > > > > >> marker
> > > > > >>> > > > > > > > >> > > or
> > > > > >>> > > > > > > > >> > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In either case,
> the
> > > > > >>> downgraded
> > > > > >>> > > > > > coordinator
> > > > > >>> > > > > > > > >> should
> > > > > >>> > > > > > > > >> > > see
> > > > > >>> > > > > > > > >> > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > same
> > > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > (probably the
> > previous
> > > > > >>> produce
> > > > > >>> > > Id),
> > > > > >>> > > > > > right?
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed, Dec 20,
> 2023
> > > at
> > > > > >>> 6:00 PM
> > > > > >>> > > > > Justine
> > > > > >>> > > > > > > > Olshan
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > >>> <jo...@confluent.io.invalid>
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey Jun,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks for
> taking
> > a
> > > > look
> > > > > >>> at
> > > > > >>> > the
> > > > > >>> > > > KIP
> > > > > >>> > > > > > > again.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100. For the
> epoch
> > > > > >>> overflow
> > > > > >>> > > case,
> > > > > >>> > > > > only
> > > > > >>> > > > > > > the
> > > > > >>> > > > > > > > >> > marker
> > > > > >>> > > > > > > > >> > > > > will
> > > > > >>> > > > > > > > >> > > > > > > have
> > > > > >>> > > > > > > > >> > > > > > > > > max
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps the
> behavior
> > > of
> > > > > the
> > > > > >>> rest
> > > > > >>> > > of
> > > > > >>> > > > > the
> > > > > >>> > > > > > > > >> markers
> > > > > >>> > > > > > > > >> > > where
> > > > > >>> > > > > > > > >> > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > last
> > > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch of the
> > > > transaction
> > > > > >>> > > records +
> > > > > >>> > > > > 1.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101. You are
> > correct
> > > > > that
> > > > > >>> we
> > > > > >>> > > don't
> > > > > >>> > > > > > need
> > > > > >>> > > > > > > to
> > > > > >>> > > > > > > > >> > write
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > > > > producer
> > > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the same.
> > > However,
> > > > I
> > > > > >>> was
> > > > > >>> > > > writing
> > > > > >>> > > > > it
> > > > > >>> > > > > > > so
> > > > > >>> > > > > > > > >> that
> > > > > >>> > > > > > > > >> > we
> > > > > >>> > > > > > > > >> > > > can
> > > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old clients
> where
> > we
> > > > > don't
> > > > > >>> > have
> > > > > >>> > > > the
> > > > > >>> > > > > > > > ability
> > > > > >>> > > > > > > > >> do
> > > > > >>> > > > > > > > >> > > this
> > > > > >>> > > > > > > > >> > > > > > > > operation
> > > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > clients that
> can.
> > > (Old
> > > > > >>> clients
> > > > > >>> > > > don't
> > > > > >>> > > > > > > bump
> > > > > >>> > > > > > > > >> the
> > > > > >>> > > > > > > > >> > > epoch
> > > > > >>> > > > > > > > >> > > > > on
> > > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say for sure the
> > > write
> > > > > >>> belongs
> > > > > >>> > > to
> > > > > >>> > > > > the
> > > > > >>> > > > > > > > given
> > > > > >>> > > > > > > > >> > > > > > transaction).
> > > > > >>> > > > > > > > >> > > > > > > > If
> > > > > >>> > > > > > > > >> > > > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn request
> > from
> > > a
> > > > > new
> > > > > >>> > > client,
> > > > > >>> > > > we
> > > > > >>> > > > > > > will
> > > > > >>> > > > > > > > >> fill
> > > > > >>> > > > > > > > >> > > this
> > > > > >>> > > > > > > > >> > > > > > > field.
> > > > > >>> > > > > > > > >> > > > > > > > We
> > > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > guarantee
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that any EndTxn
> > > > requests
> > > > > >>> with
> > > > > >>> > > the
> > > > > >>> > > > > same
> > > > > >>> > > > > > > > epoch
> > > > > >>> > > > > > > > >> > are
> > > > > >>> > > > > > > > >> > > > from
> > > > > >>> > > > > > > > >> > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > same
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the same
> > > transaction.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102. In prepare
> > > phase,
> > > > > we
> > > > > >>> have
> > > > > >>> > > the
> > > > > >>> > > > > > same
> > > > > >>> > > > > > > > >> > producer
> > > > > >>> > > > > > > > >> > > ID
> > > > > >>> > > > > > > > >> > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had. It is the
> > > > producer
> > > > > >>> ID and
> > > > > >>> > > > epoch
> > > > > >>> > > > > > > that
> > > > > >>> > > > > > > > >> are
> > > > > >>> > > > > > > > >> > on
> > > > > >>> > > > > > > > >> > > > the
> > > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > > >>> > > > > > > > >> > > > > > > > > In
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase, we stay
> the
> > > > same
> > > > > >>> unless
> > > > > >>> > > it
> > > > > >>> > > > is
> > > > > >>> > > > > > the
> > > > > >>> > > > > > > > >> > overflow
> > > > > >>> > > > > > > > >> > > > > case.
> > > > > >>> > > > > > > > >> > > > > > > In
> > > > > >>> > > > > > > > >> > > > > > > > > that
> > > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set the producer
> > ID
> > > to
> > > > > >>> the new
> > > > > >>> > > one
> > > > > >>> > > > > we
> > > > > >>> > > > > > > > >> generated
> > > > > >>> > > > > > > > >> > > and
> > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > >>> > > > > > > > >> > > > > > > > to
> > > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > complete. This
> is
> > > for
> > > > > >>> > downgrade
> > > > > >>> > > > > > > > >> compatibility.
> > > > > >>> > > > > > > > >> > > The
> > > > > >>> > > > > > > > >> > > > > > tagged
> > > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety guards
> for
> > > > > retries
> > > > > >>> and
> > > > > >>> > > > > > failovers.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In prepare phase
> > for
> > > > > epoch
> > > > > >>> > > > overflow
> > > > > >>> > > > > > case
> > > > > >>> > > > > > > > >> only
> > > > > >>> > > > > > > > >> > we
> > > > > >>> > > > > > > > >> > > > > store
> > > > > >>> > > > > > > > >> > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID. This is for
> > the
> > > > case
> > > > > >>> where
> > > > > >>> > > we
> > > > > >>> > > > > > reload
> > > > > >>> > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > transaction
> > > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > prepare state.
> > Once
> > > > the
> > > > > >>> > > > transaction
> > > > > >>> > > > > is
> > > > > >>> > > > > > > > >> > committed,
> > > > > >>> > > > > > > > >> > > > we
> > > > > >>> > > > > > > > >> > > > > > can
> > > > > >>> > > > > > > > >> > > > > > > > use
> > > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the client
> > > already
> > > > is
> > > > > >>> > using.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In commit phase,
> > we
> > > > > store
> > > > > >>> the
> > > > > >>> > > > > previous
> > > > > >>> > > > > > > > >> producer
> > > > > >>> > > > > > > > >> > > ID
> > > > > >>> > > > > > > > >> > > > in
> > > > > >>> > > > > > > > >> > > > > > > case
> > > > > >>> > > > > > > > >> > > > > > > > of
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I think it is
> > easier
> > > > to
> > > > > >>> think
> > > > > >>> > of
> > > > > >>> > > > it
> > > > > >>> > > > > as
> > > > > >>> > > > > > > > just
> > > > > >>> > > > > > > > >> how
> > > > > >>> > > > > > > > >> > > we
> > > > > >>> > > > > > > > >> > > > > were
> > > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and epoch
> before,
> > > with
> > > > > >>> some
> > > > > >>> > > extra
> > > > > >>> > > > > > > > bookeeping
> > > > > >>> > > > > > > > >> > and
> > > > > >>> > > > > > > > >> > > > edge
> > > > > >>> > > > > > > > >> > > > > > > case
> > > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged fields.
> We
> > > have
> > > > > to
> > > > > >>> do
> > > > > >>> > it
> > > > > >>> > > > this
> > > > > >>> > > > > > way
> > > > > >>> > > > > > > > for
> > > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > > >>> > > > > > > > >> > > > > > > > > with
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > downgrades.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103. Next
> producer
> > > ID
> > > > is
> > > > > >>> for
> > > > > >>> > > > prepare
> > > > > >>> > > > > > > > status
> > > > > >>> > > > > > > > >> and
> > > > > >>> > > > > > > > >> > > > > > previous
> > > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after complete.
> > The
> > > > > >>> reason why
> > > > > >>> > > we
> > > > > >>> > > > > need
> > > > > >>> > > > > > > two
> > > > > >>> > > > > > > > >> > > separate
> > > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > backwards
> > > > compatibility.
> > > > > >>> We
> > > > > >>> > need
> > > > > >>> > > > to
> > > > > >>> > > > > > keep
> > > > > >>> > > > > > > > the
> > > > > >>> > > > > > > > >> > same
> > > > > >>> > > > > > > > >> > > > > > > semantics
> > > > > >>> > > > > > > > >> > > > > > > > > for
> > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged field
> > in
> > > > case
> > > > > >>> we
> > > > > >>> > > > > downgrade.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104. We set the
> > > fields
> > > > > as
> > > > > >>> we
> > > > > >>> > do
> > > > > >>> > > in
> > > > > >>> > > > > the
> > > > > >>> > > > > > > > >> > > > transactional
> > > > > >>> > > > > > > > >> > > > > > > state
> > > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do this for
> > > > > compatibility
> > > > > >>> --
> > > > > >>> > if
> > > > > >>> > > we
> > > > > >>> > > > > > > > >> downgrade,
> > > > > >>> > > > > > > > >> > we
> > > > > >>> > > > > > > > >> > > > will
> > > > > >>> > > > > > > > >> > > > > > > only
> > > > > >>> > > > > > > > >> > > > > > > > > have
> > > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged
> fields)
> > > It
> > > > > >>> will be
> > > > > >>> > > the
> > > > > >>> > > > > old
> > > > > >>> > > > > > > > >> producer
> > > > > >>> > > > > > > > >> > ID
> > > > > >>> > > > > > > > >> > > > and
> > > > > >>> > > > > > > > >> > > > > > max
> > > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope this helps.
> > Let
> > > > me
> > > > > >>> know
> > > > > >>> > if
> > > > > >>> > > > you
> > > > > >>> > > > > > have
> > > > > >>> > > > > > > > >> > further
> > > > > >>> > > > > > > > >> > > > > > > questions.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Justine
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On Wed, Dec 20,
> > 2023
> > > > at
> > > > > >>> > 3:33 PM
> > > > > >>> > > > Jun
> > > > > >>> > > > > > Rao
> > > > > >>> > > > > > > > >> > > > > > > > > > <jun@confluent.io.invalid
> > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi, Justine,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It seems that
> > you
> > > > have
> > > > > >>> made
> > > > > >>> > > some
> > > > > >>> > > > > > > changes
> > > > > >>> > > > > > > > >> to
> > > > > >>> > > > > > > > >> > > > KIP-890
> > > > > >>> > > > > > > > >> > > > > > > since
> > > > > >>> > > > > > > > >> > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > particular, we
> > are
> > > > > >>> changing
> > > > > >>> > > the
> > > > > >>> > > > > > format
> > > > > >>> > > > > > > > of
> > > > > >>> > > > > > > > >> > > > > > > > > > TransactionLogValue.
> > > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > comments
> related
> > > to
> > > > > >>> that.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100. Just to
> be
> > > > clear.
> > > > > >>> The
> > > > > >>> > > > > overflow
> > > > > >>> > > > > > > case
> > > > > >>> > > > > > > > >> > (i.e.
> > > > > >>> > > > > > > > >> > > > > when a
> > > > > >>> > > > > > > > >> > > > > > > new
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > generated) is
> > when
> > > > the
> > > > > >>> > current
> > > > > >>> > > > > epoch
> > > > > >>> > > > > > > > >> equals
> > > > > >>> > > > > > > > >> > to
> > > > > >>> > > > > > > > >> > > > max
> > > > > >>> > > > > > > > >> > > > > -
> > > > > >>> > > > > > > > >> > > > > > 1
> > > > > >>> > > > > > > > >> > > > > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101. For the
> > "not
> > > > > epoch
> > > > > >>> > > > overflow"
> > > > > >>> > > > > > > case,
> > > > > >>> > > > > > > > we
> > > > > >>> > > > > > > > >> > > write
> > > > > >>> > > > > > > > >> > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > tagged field
> in
> > > the
> > > > > >>> complete
> > > > > >>> > > > > phase.
> > > > > >>> > > > > > Do
> > > > > >>> > > > > > > > we
> > > > > >>> > > > > > > > >> > need
> > > > > >>> > > > > > > > >> > > to
> > > > > >>> > > > > > > > >> > > > > do
> > > > > >>> > > > > > > > >> > > > > > > that
> > > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > doesn't change
> > in
> > > > this
> > > > > >>> case?
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102. It seems
> > that
> > > > the
> > > > > >>> > meaning
> > > > > >>> > > > for
> > > > > >>> > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId/ProducerEpoch
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > TransactionLogValue
> > > > > >>> changes
> > > > > >>> > > > > > depending
> > > > > >>> > > > > > > on
> > > > > >>> > > > > > > > >> the
> > > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > > TransactionStatus
> > > > > is
> > > > > >>> > > > ongoing,
> > > > > >>> > > > > > they
> > > > > >>> > > > > > > > >> > > represent
> > > > > >>> > > > > > > > >> > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > current
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the current
> > > > > >>> ProducerEpoch.
> > > > > >>> > > When
> > > > > >>> > > > > the
> > > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > > >>> > > > > > > > >> > > > > > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > >>> PrepareCommit/PrepareAbort,
> > > > > >>> > > they
> > > > > >>> > > > > > > > represent
> > > > > >>> > > > > > > > >> > the
> > > > > >>> > > > > > > > >> > > > > > current
> > > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > > ProducerEpoch.
> > > > > >>> When the
> > > > > >>> > > > > > > > >> > TransactionStatus
> > > > > >>> > > > > > > > >> > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > further depend
> > on
> > > > > >>> whether
> > > > > >>> > the
> > > > > >>> > > > > epoch
> > > > > >>> > > > > > > > >> overflows
> > > > > >>> > > > > > > > >> > > or
> > > > > >>> > > > > > > > >> > > > > not.
> > > > > >>> > > > > > > > >> > > > > > > If
> > > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > overflow, they
> > > > > represent
> > > > > >>> > the
> > > > > >>> > > > > > current
> > > > > >>> > > > > > > > >> > > ProducerId
> > > > > >>> > > > > > > > >> > > > > and
> > > > > >>> > > > > > > > >> > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > next
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ProducerEpoch
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > (max).
> > Otherwise,
> > > > they
> > > > > >>> > > represent
> > > > > >>> > > > > the
> > > > > >>> > > > > > > > newly
> > > > > >>> > > > > > > > >> > > > > generated
> > > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > ProducerEpoch
> of
> > > 0.
> > > > Is
> > > > > >>> that
> > > > > >>> > > > right?
> > > > > >>> > > > > > > This
> > > > > >>> > > > > > > > >> seems
> > > > > >>> > > > > > > > >> > > not
> > > > > >>> > > > > > > > >> > > > > > easy
> > > > > >>> > > > > > > > >> > > > > > > to
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we provide
> some
> > > > > examples
> > > > > >>> > like
> > > > > >>> > > > what
> > > > > >>> > > > > > > Artem
> > > > > >>> > > > > > > > >> has
> > > > > >>> > > > > > > > >> > > done
> > > > > >>> > > > > > > > >> > > > > in
> > > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > considered a
> > > simpler
> > > > > >>> design
> > > > > >>> > > > where
> > > > > >>> > > > > > > > >> > > > > > > > ProducerId/ProducerEpoch
> > > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > represent
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the same value
> > > (e.g.
> > > > > >>> for the
> > > > > >>> > > > > current
> > > > > >>> > > > > > > > >> > > transaction)
> > > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > TransactionStatus
> > > > and
> > > > > >>> epoch
> > > > > >>> > > > > > overflow?
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103. It's not
> > > clear
> > > > to
> > > > > >>> me
> > > > > >>> > why
> > > > > >>> > > we
> > > > > >>> > > > > > need
> > > > > >>> > > > > > > 3
> > > > > >>> > > > > > > > >> > fields:
> > > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > PrevProducerId,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> NextProducerId.
> > > > Could
> > > > > we
> > > > > >>> > just
> > > > > >>> > > > have
> > > > > >>> > > > > > > > >> ProducerId
> > > > > >>> > > > > > > > >> > > and
> > > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104. For
> > > > > >>> > > WriteTxnMarkerRequests,
> > > > > >>> > > > > if
> > > > > >>> > > > > > > the
> > > > > >>> > > > > > > > >> > > producer
> > > > > >>> > > > > > > > >> > > > > > epoch
> > > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we set the
> > > > producerId
> > > > > >>> and
> > > > > >>> > the
> > > > > >>> > > > > > > > >> producerEpoch?
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Thanks,
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > > >
> > > > > >>> > > > > > > > >> > > > > > >
> > > > > >>> > > > > > > > >> > > > > >
> > > > > >>> > > > > > > > >> > > > >
> > > > > >>> > > > > > > > >> > > >
> > > > > >>> > > > > > > > >> > >
> > > > > >>> > > > > > > > >> >
> > > > > >>> > > > > > > > >>
> > > > > >>> > > > > > > > >
> > > > > >>> > > > > > > >
> > > > > >>> > > > > > >
> > > > > >>> > > > > >
> > > > > >>> > > > >
> > > > > >>> > > >
> > > > > >>> > >
> > > > > >>> >
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Justine Olshan <jo...@confluent.io.INVALID>]

The client will send the newest EndTxn request version if and only if both
the client and the server support kip-890 part 2.
We set the value in the record based on the EndTxn version.

Justine

On Wed, Jul 10, 2024 at 2:50 PM Jun Rao <ju...@confluent.io.invalid> wrote:

> Hi, Justine,
>
> Thanks for the reply.
>
> 120. If the broker sends TV Y for the finalized version in
> ApiVersionResponse, but the client doesn't support Y, how does the broker
> know the TV that the client supports?
>
> Jun
>
> On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan
> <jo...@confluent.io.invalid>
> wrote:
>
> > Hey Jun,
> >
> > No worries. Work on this KIP has been blocked for a bit anyways --
> catching
> > up and rereading what I wrote :)
> >
> > 120. ClientTransactionProtocolVersion is the transaction version as
> defined
> > by the highest transaction version (feature version value) supported by
> the
> > client and the server. This works by the broker sending an
> > ApiVersionsRequest to the client with the finalized version. Assuming
> > kip-890 part 2 is enabled by transaction version Y, if this request
> > contains finalized version Y and the client has the logic to set this
> > field, it will set Y. If the server has Y - 1 (kip 890 part 2 not enable)
> > the client will send Y - 1, even though the client has the ability to
> > support kip-890 part 2.
> >
> > 121. You are correct that this is not needed. However, currently that
> field
> > is already being set in memory -- just not written to disk. I think it is
> > ok to write it to disk though. Let me know if you think otherwise.
> >
> > Justine
> >
> > On Wed, Jul 10, 2024 at 2:16 PM Jun Rao <ju...@confluent.io.invalid>
> wrote:
> >
> > > Hi, Justine,
> > >
> > > Thanks for the update and sorry for the late reply.
> > >
> > > 120. I am wondering what value is used for
> > > ClientTransactionProtocolVersion. Is it the version of the
> EndTxnRequest?
> > >
> > > 121. Earlier, you made the change to set lastProducerId in PREPARE to
> > > indicate that the marker is written for the new client. With the new
> > > ClientTransactionProtocolVersion field, it seems this is no longer
> > > necessary.
> > >
> > > Jun
> > >
> > > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > > <jo...@confluent.io.invalid>
> > > wrote:
> > >
> > > > Hi there -- another update!
> > > >
> > > > When looking into the implementation for the safe epoch bumps I
> > realized
> > > > that we are already populating previousProducerID in memory as part
> of
> > > > KIP-360.
> > > > If we are to start using flexible fields, it is better to always use
> > this
> > > > information and have an explicit (tagged) field to indicate whether
> the
> > > > client supports KIP-890 part 2.
> > > >
> > > > I've included the extra field and how it is set in the KIP. I've also
> > > > updated the KIP to explain that we will be setting the tagged fields
> > when
> > > > they are available for all transitions.
> > > >
> > > > Finally, I added clearer text about the transaction protocol versions
> > > > included with this KIP. 1 for flexible transaction state records and
> 2
> > > for
> > > > KIP-890 part 2 enablement.
> > > >
> > > > Justine
> > > >
> > > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <jolshan@confluent.io
> >
> > > > wrote:
> > > >
> > > > > Hey there -- small update to the KIP,
> > > > >
> > > > > The KIP mentioned introducing ABORTABLE_ERROR and bumping
> > > TxnOffsetCommit
> > > > > and Produce requests. I've changed the name in the KIP to
> > > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > > AbortableTransactionException to match the pattern we had for other
> > > > errors.
> > > > > I also mentioned bumping all 6 transactional APIs so we can future
> > > > > proof/support the error on the client going forward. If a future
> > change
> > > > > wants to have an error scenario that requires us to abort the
> > > > transaction,
> > > > > we can rely on the 3.8+ clients to support it. We ran into issues
> > > finding
> > > > > good/generic error codes that older clients could support while
> > working
> > > > on
> > > > > this KIP, so this should help in the future.
> > > > >
> > > > > The features discussion is still ongoing in KIP-1022. Will update
> > again
> > > > > here when that concludes.
> > > > >
> > > > > Justine
> > > > >
> > > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <
> jolshan@confluent.io>
> > > > > wrote:
> > > > >
> > > > >> I don't think AddPartitions is a good example since we currenly
> > don't
> > > > >> gate the version on TV or MV. (We only set a different flag
> > depending
> > > on
> > > > >> the TV)
> > > > >>
> > > > >> Even if we did want to gate it on TV, I think the idea is to move
> > away
> > > > >> from MV gating inter broker protocols. Ideally we can get to a
> state
> > > > where
> > > > >> MV is just used for metadata changes.
> > > > >>
> > > > >> I think some of this discussion might fit more with the feature
> > > version
> > > > >> KIP, so I can try to open that up soon. Until we settle that, some
> > of
> > > > the
> > > > >> work in KIP-890 is blocked.
> > > > >>
> > > > >> Justine
> > > > >>
> > > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao <ju...@confluent.io.invalid>
> > > > wrote:
> > > > >>
> > > > >>> Hi, Justine,
> > > > >>>
> > > > >>> Thanks for the reply.
> > > > >>>
> > > > >>> Since AddPartitions is an inter broker request, will its version
> be
> > > > gated
> > > > >>> only by TV or other features like MV too? For example, if we need
> > to
> > > > >>> change
> > > > >>> the protocol for AddPartitions for reasons other than txn
> > > verification
> > > > in
> > > > >>> the future, will the new version be gated by a new MV? If so,
> does
> > > > >>> downgrading a TV imply potential downgrade of MV too?
> > > > >>>
> > > > >>> Jun
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > > >>> <jo...@confluent.io.invalid>
> > > > >>> wrote:
> > > > >>>
> > > > >>> > One TV gates the flexible feature version (no rpcs involved,
> only
> > > the
> > > > >>> > transactional records that should only be gated by TV)
> > > > >>> > Another TV gates the ability to turn on kip-890 part 2. This
> > would
> > > > >>> gate the
> > > > >>> > version of Produce and EndTxn (likely only used by
> transactions),
> > > and
> > > > >>> > specifies a flag in AddPartitionsToTxn though the version is
> > > already
> > > > >>> used
> > > > >>> > without TV.
> > > > >>> >
> > > > >>> > I think the only concern is the Produce request and we could
> > > consider
> > > > >>> work
> > > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > > >>> >
> > > > >>> > Justine
> > > > >>> >
> > > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao
> <jun@confluent.io.invalid
> > >
> > > > >>> wrote:
> > > > >>> >
> > > > >>> > > Hi, Justine,
> > > > >>> > >
> > > > >>> > > Which PRC/record protocols will TV guard? Going forward, will
> > > those
> > > > >>> > > PRC/record protocols only be guarded by TV and not by other
> > > > features
> > > > >>> like
> > > > >>> > > MV?
> > > > >>> > >
> > > > >>> > > Thanks,
> > > > >>> > >
> > > > >>> > > Jun
> > > > >>> > >
> > > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > > >>> > <jolshan@confluent.io.invalid
> > > > >>> > > >
> > > > >>> > > wrote:
> > > > >>> > >
> > > > >>> > > > Hi Jun,
> > > > >>> > > >
> > > > >>> > > > Sorry I think I misunderstood your question or answered
> > > > >>> incorrectly.
> > > > >>> > The
> > > > >>> > > TV
> > > > >>> > > > version should ideally be fully independent from MV.
> > > > >>> > > > At least for the changes I proposed, TV should not affect
> MV
> > > and
> > > > MV
> > > > >>> > > should
> > > > >>> > > > not affect TV/
> > > > >>> > > >
> > > > >>> > > > I think if we downgrade TV, only that feature should
> > downgrade.
> > > > >>> > Likewise
> > > > >>> > > > the same with MV. The finalizedFeatures should just reflect
> > the
> > > > >>> feature
> > > > >>> > > > downgrade we made.
> > > > >>> > > >
> > > > >>> > > > I also plan to write a new KIP for managing the disk format
> > and
> > > > >>> upgrade
> > > > >>> > > > tool as we will need new flags to support these features.
> > That
> > > > >>> should
> > > > >>> > > help
> > > > >>> > > > clarify some things.
> > > > >>> > > >
> > > > >>> > > > Justine
> > > > >>> > > >
> > > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > > <jun@confluent.io.invalid
> > > > >
> > > > >>> > > wrote:
> > > > >>> > > >
> > > > >>> > > > > Hi, Justine,
> > > > >>> > > > >
> > > > >>> > > > > Thanks for the reply.
> > > > >>> > > > >
> > > > >>> > > > > So, if we downgrade TV, we could implicitly downgrade
> > another
> > > > >>> feature
> > > > >>> > > > (say
> > > > >>> > > > > MV) that has dependency (e.g. RPC). What would we return
> > for
> > > > >>> > > > > FinalizedFeatures for MV in ApiVersionsResponse in that
> > case?
> > > > >>> > > > >
> > > > >>> > > > > Thanks,
> > > > >>> > > > >
> > > > >>> > > > > Jun
> > > > >>> > > > >
> > > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > > >>> > > > <jolshan@confluent.io.invalid
> > > > >>> > > > > >
> > > > >>> > > > > wrote:
> > > > >>> > > > >
> > > > >>> > > > > > Hey Jun,
> > > > >>> > > > > >
> > > > >>> > > > > > Yes, the idea is that if we downgrade TV (transaction
> > > > version)
> > > > >>> we
> > > > >>> > > will
> > > > >>> > > > > stop
> > > > >>> > > > > > using the add partitions to txn optimization and stop
> > > writing
> > > > >>> the
> > > > >>> > > > > flexible
> > > > >>> > > > > > feature version of the log.
> > > > >>> > > > > > In the compatibility section I included some
> explanations
> > > on
> > > > >>> how
> > > > >>> > this
> > > > >>> > > > is
> > > > >>> > > > > > done.
> > > > >>> > > > > >
> > > > >>> > > > > > Thanks,
> > > > >>> > > > > > Justine
> > > > >>> > > > > >
> > > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > > >>> <ju...@confluent.io.invalid>
> > > > >>> > > > > wrote:
> > > > >>> > > > > >
> > > > >>> > > > > > > Hi, Justine,
> > > > >>> > > > > > >
> > > > >>> > > > > > > Thanks for the update.
> > > > >>> > > > > > >
> > > > >>> > > > > > > If we ever downgrade the transaction feature, any
> > feature
> > > > >>> > depending
> > > > >>> > > > on
> > > > >>> > > > > > > changes on top of those RPC/record
> > > > >>> > > > > > > (AddPartitionsToTxnRequest/TransactionLogValue)
> changes
> > > > made
> > > > >>> in
> > > > >>> > > > KIP-890
> > > > >>> > > > > > > will be automatically downgraded too?
> > > > >>> > > > > > >
> > > > >>> > > > > > > Jun
> > > > >>> > > > > > >
> > > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine Olshan
> > > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > > >>> > > > > > > wrote:
> > > > >>> > > > > > >
> > > > >>> > > > > > > > Hey Jun,
> > > > >>> > > > > > > >
> > > > >>> > > > > > > > I wanted to get back to you about your questions
> > about
> > > > >>> MV/IBP.
> > > > >>> > > > > > > >
> > > > >>> > > > > > > > Looking at the options, I think it makes the most
> > sense
> > > > to
> > > > >>> > > create a
> > > > >>> > > > > > > > separate feature for transactions and use that to
> > > version
> > > > >>> gate
> > > > >>> > > the
> > > > >>> > > > > > > features
> > > > >>> > > > > > > > we need to version gate (flexible transactional
> state
> > > > >>> records
> > > > >>> > and
> > > > >>> > > > > using
> > > > >>> > > > > > > the
> > > > >>> > > > > > > > new protocol)
> > > > >>> > > > > > > > I've updated the KIP to include this change.
> > Hopefully
> > > > >>> that's
> > > > >>> > > > > > everything
> > > > >>> > > > > > > we
> > > > >>> > > > > > > > need for this KIP :)
> > > > >>> > > > > > > >
> > > > >>> > > > > > > > Justine
> > > > >>> > > > > > > >
> > > > >>> > > > > > > >
> > > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine Olshan <
> > > > >>> > > > jolshan@confluent.io
> > > > >>> > > > > >
> > > > >>> > > > > > > > wrote:
> > > > >>> > > > > > > >
> > > > >>> > > > > > > > > Thanks Jun,
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > > > I will update the KIP with the prev field for
> > prepare
> > > > as
> > > > >>> > well.
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > > > PREPARE
> > > > >>> > > > > > > > > producerId: x
> > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > >>> > > > > > > > > nextProducerId (tagged field): empty or z if y
> will
> > > > >>> overflow
> > > > >>> > > > > > > > > producerEpoch: y + 1
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > > > COMPLETE
> > > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we overflowed
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > > > Thanks again,
> > > > >>> > > > > > > > > Justine
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun Rao
> > > > >>> > > <jun@confluent.io.invalid
> > > > >>> > > > >
> > > > >>> > > > > > > > wrote:
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > > >> Hi, Justine,
> > > > >>> > > > > > > > >>
> > > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > > >>> > > > > > > > >> (1) My point was that the coordinator could fail
> > > right
> > > > >>> after
> > > > >>> > > > > writing
> > > > >>> > > > > > > the
> > > > >>> > > > > > > > >> prepare marker. When the new txn coordinator
> > > generates
> > > > >>> the
> > > > >>> > > > > complete
> > > > >>> > > > > > > > marker
> > > > >>> > > > > > > > >> after the failover, it needs some field from the
> > > > prepare
> > > > >>> > > marker
> > > > >>> > > > to
> > > > >>> > > > > > > > >> determine whether it's written by the new
> client.
> > > > >>> > > > > > > > >>
> > > > >>> > > > > > > > >> (2) The changing of the behavior sounds good to
> > me.
> > > We
> > > > >>> only
> > > > >>> > > want
> > > > >>> > > > > to
> > > > >>> > > > > > > > return
> > > > >>> > > > > > > > >> success if the prepare state is written by the
> new
> > > > >>> client.
> > > > >>> > So,
> > > > >>> > > > in
> > > > >>> > > > > > the
> > > > >>> > > > > > > > >> non-overflow case, it seems that we also need
> sth
> > in
> > > > the
> > > > >>> > > prepare
> > > > >>> > > > > > > marker
> > > > >>> > > > > > > > to
> > > > >>> > > > > > > > >> tell us whether it's written by the new client.
> > > > >>> > > > > > > > >>
> > > > >>> > > > > > > > >> 112. Thanks for the explanation. That sounds
> good
> > to
> > > > me.
> > > > >>> > > > > > > > >>
> > > > >>> > > > > > > > >> Jun
> > > > >>> > > > > > > > >>
> > > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM Justine Olshan
> > > > >>> > > > > > > > >> <jo...@confluent.io.invalid> wrote:
> > > > >>> > > > > > > > >>
> > > > >>> > > > > > > > >> > 101.3 I realized that I actually have two
> > > questions.
> > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need to
> write
> > > the
> > > > >>> > > previous
> > > > >>> > > > > > > produce
> > > > >>> > > > > > > > Id
> > > > >>> > > > > > > > >> > tagged field in the end maker so that we know
> if
> > > the
> > > > >>> > marker
> > > > >>> > > is
> > > > >>> > > > > > from
> > > > >>> > > > > > > > the
> > > > >>> > > > > > > > >> new
> > > > >>> > > > > > > > >> > client. Since the end maker is derived from
> the
> > > > >>> prepare
> > > > >>> > > > marker,
> > > > >>> > > > > > > should
> > > > >>> > > > > > > > >> we
> > > > >>> > > > > > > > >> > write the previous produce Id in the prepare
> > > marker
> > > > >>> field
> > > > >>> > > too?
> > > > >>> > > > > > > > >> Otherwise,
> > > > >>> > > > > > > > >> > we will lose this information when deriving
> the
> > > end
> > > > >>> > marker.
> > > > >>> > > > > > > > >> >
> > > > >>> > > > > > > > >> > The "previous" producer ID is in the normal
> > > producer
> > > > >>> ID
> > > > >>> > > field.
> > > > >>> > > > > So
> > > > >>> > > > > > > yes,
> > > > >>> > > > > > > > >> we
> > > > >>> > > > > > > > >> > need it in prepare and that was always the
> plan.
> > > > >>> > > > > > > > >> >
> > > > >>> > > > > > > > >> > Maybe it is a bit unclear so I will enumerate
> > the
> > > > >>> fields
> > > > >>> > and
> > > > >>> > > > add
> > > > >>> > > > > > > them
> > > > >>> > > > > > > > to
> > > > >>> > > > > > > > >> > the KIP if that helps.
> > > > >>> > > > > > > > >> > Say we have producer ID x and epoch y. When we
> > > > >>> overflow
> > > > >>> > > epoch
> > > > >>> > > > y
> > > > >>> > > > > we
> > > > >>> > > > > > > get
> > > > >>> > > > > > > > >> > producer ID Z.
> > > > >>> > > > > > > > >> >
> > > > >>> > > > > > > > >> > PREPARE
> > > > >>> > > > > > > > >> > producerId: x
> > > > >>> > > > > > > > >> > previous/lastProducerId (tagged field): empty
> > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty or z if y
> > > will
> > > > >>> > overflow
> > > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > > >>> > > > > > > > >> >
> > > > >>> > > > > > > > >> > COMPLETE
> > > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > > >>> > > > > > > > >> > previous/lastProducerId (tagged field): x
> > > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we overflowed
> > > > >>> > > > > > > > >> >
> > > > >>> > > > > > > > >> > (2) In the prepare phase, if we retry and see
> > > epoch
> > > > -
> > > > >>> 1 +
> > > > >>> > ID
> > > > >>> > > > in
> > > > >>> > > > > > last
> > > > >>> > > > > > > > >> seen
> > > > >>> > > > > > > > >> > fields and are issuing the same command (ie
> > commit
> > > > not
> > > > >>> > > abort),
> > > > >>> > > > > we
> > > > >>> > > > > > > > return
> > > > >>> > > > > > > > >> > success. The logic before KIP-890 seems to
> > return
> > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > >>> > > > > > > > >> > in this case. Are we intentionally making this
> > > > change?
> > > > >>> > > > > > > > >> >
> > > > >>> > > > > > > > >> > Hmm -- we would fence the producer if the
> epoch
> > is
> > > > >>> bumped
> > > > >>> > > and
> > > > >>> > > > we
> > > > >>> > > > > > > get a
> > > > >>> > > > > > > > >> > lower epoch. Yes -- we are intentionally
> adding
> > > this
> > > > >>> to
> > > > >>> > > > prevent
> > > > >>> > > > > > > > fencing.
> > > > >>> > > > > > > > >> >
> > > > >>> > > > > > > > >> >
> > > > >>> > > > > > > > >> > 112. We already merged the code that adds the
> > > > >>> VerifyOnly
> > > > >>> > > field
> > > > >>> > > > > in
> > > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which is an inter
> > > broker
> > > > >>> > request.
> > > > >>> > > > It
> > > > >>> > > > > > > seems
> > > > >>> > > > > > > > >> that
> > > > >>> > > > > > > > >> > we didn't bump up the IBP for that. Do you
> know
> > > why?
> > > > >>> > > > > > > > >> >
> > > > >>> > > > > > > > >> > We no longer need IBP for all interbroker
> > requests
> > > > as
> > > > >>> > > > > ApiVersions
> > > > >>> > > > > > > > should
> > > > >>> > > > > > > > >> > correctly gate versioning.
> > > > >>> > > > > > > > >> > We also handle unsupported version errors
> > > correctly
> > > > >>> if we
> > > > >>> > > > > receive
> > > > >>> > > > > > > them
> > > > >>> > > > > > > > >> in
> > > > >>> > > > > > > > >> > edge cases like upgrades/downgrades.
> > > > >>> > > > > > > > >> >
> > > > >>> > > > > > > > >> > Justine
> > > > >>> > > > > > > > >> >
> > > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM Jun Rao
> > > > >>> > > > > <jun@confluent.io.invalid
> > > > >>> > > > > > >
> > > > >>> > > > > > > > >> wrote:
> > > > >>> > > > > > > > >> >
> > > > >>> > > > > > > > >> > > Hi, Justine,
> > > > >>> > > > > > > > >> > >
> > > > >>> > > > > > > > >> > > Thanks for the reply.
> > > > >>> > > > > > > > >> > >
> > > > >>> > > > > > > > >> > > 101.3 I realized that I actually have two
> > > > questions.
> > > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need to
> write
> > > the
> > > > >>> > > previous
> > > > >>> > > > > > > produce
> > > > >>> > > > > > > > Id
> > > > >>> > > > > > > > >> > > tagged field in the end maker so that we
> know
> > if
> > > > the
> > > > >>> > > marker
> > > > >>> > > > is
> > > > >>> > > > > > > from
> > > > >>> > > > > > > > >> the
> > > > >>> > > > > > > > >> > new
> > > > >>> > > > > > > > >> > > client. Since the end maker is derived from
> > the
> > > > >>> prepare
> > > > >>> > > > > marker,
> > > > >>> > > > > > > > >> should we
> > > > >>> > > > > > > > >> > > write the previous produce Id in the prepare
> > > > marker
> > > > >>> > field
> > > > >>> > > > too?
> > > > >>> > > > > > > > >> Otherwise,
> > > > >>> > > > > > > > >> > > we will lose this information when deriving
> > the
> > > > end
> > > > >>> > > marker.
> > > > >>> > > > > > > > >> > > (2) In the prepare phase, if we retry and
> see
> > > > epoch
> > > > >>> - 1
> > > > >>> > +
> > > > >>> > > ID
> > > > >>> > > > > in
> > > > >>> > > > > > > last
> > > > >>> > > > > > > > >> seen
> > > > >>> > > > > > > > >> > > fields and are issuing the same command (ie
> > > commit
> > > > >>> not
> > > > >>> > > > abort),
> > > > >>> > > > > > we
> > > > >>> > > > > > > > >> return
> > > > >>> > > > > > > > >> > > success. The logic before KIP-890 seems to
> > > return
> > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > >>> > > > > > > > >> > > in this case. Are we intentionally making
> this
> > > > >>> change?
> > > > >>> > > > > > > > >> > >
> > > > >>> > > > > > > > >> > > 112. We already merged the code that adds
> the
> > > > >>> VerifyOnly
> > > > >>> > > > field
> > > > >>> > > > > > in
> > > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which is an inter
> > > > broker
> > > > >>> > > request.
> > > > >>> > > > > It
> > > > >>> > > > > > > > seems
> > > > >>> > > > > > > > >> > that
> > > > >>> > > > > > > > >> > > we didn't bump up the IBP for that. Do you
> > know
> > > > why?
> > > > >>> > > > > > > > >> > >
> > > > >>> > > > > > > > >> > > Jun
> > > > >>> > > > > > > > >> > >
> > > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM Justine
> Olshan
> > > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > > >>> > > > > > > > >> > > wrote:
> > > > >>> > > > > > > > >> > >
> > > > >>> > > > > > > > >> > > > Hi Jun,
> > > > >>> > > > > > > > >> > > >
> > > > >>> > > > > > > > >> > > > 101.3 I can change "last seen" to "current
> > > > >>> producer id
> > > > >>> > > and
> > > > >>> > > > > > > epoch"
> > > > >>> > > > > > > > if
> > > > >>> > > > > > > > >> > that
> > > > >>> > > > > > > > >> > > > was the part that was confusing
> > > > >>> > > > > > > > >> > > > 110 I can mention this
> > > > >>> > > > > > > > >> > > > 111 I can do that
> > > > >>> > > > > > > > >> > > > 112 We still need it. But I am still
> > > finalizing
> > > > >>> the
> > > > >>> > > > design.
> > > > >>> > > > > I
> > > > >>> > > > > > > will
> > > > >>> > > > > > > > >> > update
> > > > >>> > > > > > > > >> > > > the KIP once I get the information
> > finalized.
> > > > >>> Sorry
> > > > >>> > for
> > > > >>> > > > the
> > > > >>> > > > > > > > delays.
> > > > >>> > > > > > > > >> > > >
> > > > >>> > > > > > > > >> > > > Justine
> > > > >>> > > > > > > > >> > > >
> > > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at 10:50 AM Jun Rao
> > > > >>> > > > > > > <jun@confluent.io.invalid
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > > >> > > wrote:
> > > > >>> > > > > > > > >> > > >
> > > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > > >>> > > > > > > > >> > > > >
> > > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > > >>> > > > > > > > >> > > > >
> > > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow case, the
> > previous
> > > > ID
> > > > >>> is
> > > > >>> > the
> > > > >>> > > > > same
> > > > >>> > > > > > as
> > > > >>> > > > > > > > the
> > > > >>> > > > > > > > >> > > > produce
> > > > >>> > > > > > > > >> > > > > ID for the complete marker too, but we
> set
> > > the
> > > > >>> > > previous
> > > > >>> > > > ID
> > > > >>> > > > > > in
> > > > >>> > > > > > > > the
> > > > >>> > > > > > > > >> > > > complete
> > > > >>> > > > > > > > >> > > > > marker. Earlier you mentioned that this
> is
> > > to
> > > > >>> know
> > > > >>> > > that
> > > > >>> > > > > the
> > > > >>> > > > > > > > >> marker is
> > > > >>> > > > > > > > >> > > > > written by the new client so that we
> could
> > > > >>> return
> > > > >>> > > > success
> > > > >>> > > > > on
> > > > >>> > > > > > > > >> retried
> > > > >>> > > > > > > > >> > > > > endMarker requests. I was trying to
> > > understand
> > > > >>> why
> > > > >>> > > this
> > > > >>> > > > is
> > > > >>> > > > > > not
> > > > >>> > > > > > > > >> needed
> > > > >>> > > > > > > > >> > > for
> > > > >>> > > > > > > > >> > > > > the prepare marker since retry can
> happen
> > in
> > > > the
> > > > >>> > > prepare
> > > > >>> > > > > > state
> > > > >>> > > > > > > > >> too.
> > > > >>> > > > > > > > >> > Is
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > reason that in the prepare state, we
> > return
> > > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > > >>> > > > > > > > >> > > > instead
> > > > >>> > > > > > > > >> > > > > of success on retried endMaker requests?
> > If
> > > > so,
> > > > >>> > should
> > > > >>> > > > we
> > > > >>> > > > > > > change
> > > > >>> > > > > > > > >> "If
> > > > >>> > > > > > > > >> > we
> > > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID in last
> seen
> > > > >>> fields and
> > > > >>> > > are
> > > > >>> > > > > > > issuing
> > > > >>> > > > > > > > >> the
> > > > >>> > > > > > > > >> > > same
> > > > >>> > > > > > > > >> > > > > command (ie commit not abort) we can
> > return
> > > > >>> (with
> > > > >>> > the
> > > > >>> > > > new
> > > > >>> > > > > > > > epoch)"
> > > > >>> > > > > > > > >> > > > > accordingly?
> > > > >>> > > > > > > > >> > > > >
> > > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP, a delayed
> > > endMaker
> > > > >>> > request
> > > > >>> > > > > > carries
> > > > >>> > > > > > > > the
> > > > >>> > > > > > > > >> > same
> > > > >>> > > > > > > > >> > > > > epoch and won't be fenced. This can
> > > > >>> commit/abort a
> > > > >>> > > > future
> > > > >>> > > > > > > > >> transaction
> > > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure if we have
> > seen
> > > > >>> this in
> > > > >>> > > > > practice
> > > > >>> > > > > > > > >> though.
> > > > >>> > > > > > > > >> > > > >
> > > > >>> > > > > > > > >> > > > > 111. Sounds good. It would be useful to
> > make
> > > > it
> > > > >>> > clear
> > > > >>> > > > that
> > > > >>> > > > > > we
> > > > >>> > > > > > > > can
> > > > >>> > > > > > > > >> now
> > > > >>> > > > > > > > >> > > > > populate the lastSeen field from the log
> > > > >>> reliably.
> > > > >>> > > > > > > > >> > > > >
> > > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > > >>> > AddPartitionsToTxnRequest
> > > > >>> > > > > since
> > > > >>> > > > > > > > it's
> > > > >>> > > > > > > > >> > > called
> > > > >>> > > > > > > > >> > > > > across brokers and we are changing its
> > > schema.
> > > > >>> Are
> > > > >>> > you
> > > > >>> > > > > > saying
> > > > >>> > > > > > > we
> > > > >>> > > > > > > > >> > don't
> > > > >>> > > > > > > > >> > > > need
> > > > >>> > > > > > > > >> > > > > it any more? I thought that we already
> > > > >>> implemented
> > > > >>> > the
> > > > >>> > > > > > server
> > > > >>> > > > > > > > side
> > > > >>> > > > > > > > >> > > > > verification logic based on
> > > > >>> > AddPartitionsToTxnRequest
> > > > >>> > > > > across
> > > > >>> > > > > > > > >> brokers.
> > > > >>> > > > > > > > >> > > > >
> > > > >>> > > > > > > > >> > > > > Jun
> > > > >>> > > > > > > > >> > > > >
> > > > >>> > > > > > > > >> > > > >
> > > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at 5:05 PM Justine
> > > Olshan
> > > > >>> > > > > > > > >> > > > > <jo...@confluent.io.invalid>
> > > > >>> > > > > > > > >> > > > > wrote:
> > > > >>> > > > > > > > >> > > > >
> > > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > > >>> > > > > > > > >> > > > > >
> > > > >>> > > > > > > > >> > > > > > 101.3 We don't set the previous ID in
> > the
> > > > >>> Prepare
> > > > >>> > > > field
> > > > >>> > > > > > > since
> > > > >>> > > > > > > > we
> > > > >>> > > > > > > > >> > > don't
> > > > >>> > > > > > > > >> > > > > need
> > > > >>> > > > > > > > >> > > > > > it. It is the same producer ID as the
> > main
> > > > >>> > producer
> > > > >>> > > ID
> > > > >>> > > > > > > field.
> > > > >>> > > > > > > > >> > > > > >
> > > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to reread your
> > > > message
> > > > >>> > about
> > > > >>> > > > > > delayed
> > > > >>> > > > > > > > >> > markers.
> > > > >>> > > > > > > > >> > > > If
> > > > >>> > > > > > > > >> > > > > we
> > > > >>> > > > > > > > >> > > > > > receive a delayed endTxn marker after
> > the
> > > > >>> > > transaction
> > > > >>> > > > is
> > > > >>> > > > > > > > already
> > > > >>> > > > > > > > >> > > > > complete?
> > > > >>> > > > > > > > >> > > > > > So we will commit the next transaction
> > > early
> > > > >>> > without
> > > > >>> > > > the
> > > > >>> > > > > > > fixes
> > > > >>> > > > > > > > >> in
> > > > >>> > > > > > > > >> > > part
> > > > >>> > > > > > > > >> > > > 2?
> > > > >>> > > > > > > > >> > > > > >
> > > > >>> > > > > > > > >> > > > > > 111 Yes -- this terminology was used
> in
> > a
> > > > >>> previous
> > > > >>> > > KIP
> > > > >>> > > > > and
> > > > >>> > > > > > > > never
> > > > >>> > > > > > > > >> > > > > > implemented it in the log -- only in
> > > memory
> > > > >>> > > > > > > > >> > > > > >
> > > > >>> > > > > > > > >> > > > > > 112 Hmm -- which interbroker protocol
> > are
> > > > you
> > > > >>> > > > referring
> > > > >>> > > > > > to?
> > > > >>> > > > > > > I
> > > > >>> > > > > > > > am
> > > > >>> > > > > > > > >> > > > working
> > > > >>> > > > > > > > >> > > > > on
> > > > >>> > > > > > > > >> > > > > > the design for the work to remove the
> > > extra
> > > > >>> add
> > > > >>> > > > > partitions
> > > > >>> > > > > > > > call
> > > > >>> > > > > > > > >> > and I
> > > > >>> > > > > > > > >> > > > > right
> > > > >>> > > > > > > > >> > > > > > now the design bumps MV. I have yet to
> > > > update
> > > > >>> that
> > > > >>> > > > > section
> > > > >>> > > > > > > as
> > > > >>> > > > > > > > I
> > > > >>> > > > > > > > >> > > > finalize
> > > > >>> > > > > > > > >> > > > > > the design so please stay tuned. Was
> > there
> > > > >>> > anything
> > > > >>> > > > else
> > > > >>> > > > > > you
> > > > >>> > > > > > > > >> > thought
> > > > >>> > > > > > > > >> > > > > needed
> > > > >>> > > > > > > > >> > > > > > MV bump?
> > > > >>> > > > > > > > >> > > > > >
> > > > >>> > > > > > > > >> > > > > > Justine
> > > > >>> > > > > > > > >> > > > > >
> > > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at 3:07 PM Jun
> Rao
> > > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > > >>> > > > > > > > >> > > > > wrote:
> > > > >>> > > > > > > > >> > > > > >
> > > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > > >>> > > > > > > > >> > > > > > >
> > > > >>> > > > > > > > >> > > > > > > I don't see this create any issue.
> It
> > > just
> > > > >>> makes
> > > > >>> > > it
> > > > >>> > > > a
> > > > >>> > > > > > bit
> > > > >>> > > > > > > > >> hard to
> > > > >>> > > > > > > > >> > > > > explain
> > > > >>> > > > > > > > >> > > > > > > what this non-tagged produce id
> field
> > > > >>> means. We
> > > > >>> > > are
> > > > >>> > > > > > > > >> essentially
> > > > >>> > > > > > > > >> > > > trying
> > > > >>> > > > > > > > >> > > > > to
> > > > >>> > > > > > > > >> > > > > > > combine two actions (completing a
> txn
> > > and
> > > > >>> init a
> > > > >>> > > new
> > > > >>> > > > > > > produce
> > > > >>> > > > > > > > >> Id)
> > > > >>> > > > > > > > >> > > in a
> > > > >>> > > > > > > > >> > > > > > > single record. But, this may be fine
> > > too.
> > > > >>> > > > > > > > >> > > > > > >
> > > > >>> > > > > > > > >> > > > > > > A few other follow up comments.
> > > > >>> > > > > > > > >> > > > > > >
> > > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason that we
> only
> > > set
> > > > >>> the
> > > > >>> > > > previous
> > > > >>> > > > > > > > >> produce id
> > > > >>> > > > > > > > >> > > > > tagged
> > > > >>> > > > > > > > >> > > > > > > field in the complete marker, but
> not
> > in
> > > > the
> > > > >>> > > prepare
> > > > >>> > > > > > > marker,
> > > > >>> > > > > > > > >> is
> > > > >>> > > > > > > > >> > > that
> > > > >>> > > > > > > > >> > > > in
> > > > >>> > > > > > > > >> > > > > > the
> > > > >>> > > > > > > > >> > > > > > > prepare state, we always return
> > > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > > >>> > > > > > on
> > > > >>> > > > > > > > >> > retried
> > > > >>> > > > > > > > >> > > > > > endMaker
> > > > >>> > > > > > > > >> > > > > > > requests?
> > > > >>> > > > > > > > >> > > > > > >
> > > > >>> > > > > > > > >> > > > > > > 110. "I believe your second point is
> > > > >>> mentioned
> > > > >>> > in
> > > > >>> > > > the
> > > > >>> > > > > > > KIP. I
> > > > >>> > > > > > > > >> can
> > > > >>> > > > > > > > >> > > add
> > > > >>> > > > > > > > >> > > > > more
> > > > >>> > > > > > > > >> > > > > > > text on
> > > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > > >>> > > > > > > > >> > > > > > > > The delayed message case can also
> > > > violate
> > > > >>> EOS
> > > > >>> > if
> > > > >>> > > > the
> > > > >>> > > > > > > > delayed
> > > > >>> > > > > > > > >> > > > message
> > > > >>> > > > > > > > >> > > > > > > comes in after the next
> > > addPartitionsToTxn
> > > > >>> > request
> > > > >>> > > > > comes
> > > > >>> > > > > > > in.
> > > > >>> > > > > > > > >> > > > > Effectively
> > > > >>> > > > > > > > >> > > > > > we
> > > > >>> > > > > > > > >> > > > > > > may see a message from a previous
> > > > (aborted)
> > > > >>> > > > > transaction
> > > > >>> > > > > > > > become
> > > > >>> > > > > > > > >> > part
> > > > >>> > > > > > > > >> > > > of
> > > > >>> > > > > > > > >> > > > > > the
> > > > >>> > > > > > > > >> > > > > > > next transaction."
> > > > >>> > > > > > > > >> > > > > > >
> > > > >>> > > > > > > > >> > > > > > > The above is the case when a delayed
> > > > >>> message is
> > > > >>> > > > > appended
> > > > >>> > > > > > > to
> > > > >>> > > > > > > > >> the
> > > > >>> > > > > > > > >> > > data
> > > > >>> > > > > > > > >> > > > > > > partition. What I mentioned is a
> > > slightly
> > > > >>> > > different
> > > > >>> > > > > case
> > > > >>> > > > > > > > when
> > > > >>> > > > > > > > >> a
> > > > >>> > > > > > > > >> > > > delayed
> > > > >>> > > > > > > > >> > > > > > > marker is appended to the
> transaction
> > > log
> > > > >>> > > partition.
> > > > >>> > > > > > > > >> > > > > > >
> > > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once we move past
> > the
> > > > >>> Prepare
> > > > >>> > > and
> > > > >>> > > > > > > > Complete
> > > > >>> > > > > > > > >> > > states,
> > > > >>> > > > > > > > >> > > > > we
> > > > >>> > > > > > > > >> > > > > > > don’t need to worry about lastSeen
> > > fields
> > > > >>> and
> > > > >>> > > clear
> > > > >>> > > > > > them,
> > > > >>> > > > > > > > just
> > > > >>> > > > > > > > >> > > handle
> > > > >>> > > > > > > > >> > > > > > state
> > > > >>> > > > > > > > >> > > > > > > transitions as normal.". Is the
> > lastSeen
> > > > >>> field
> > > > >>> > the
> > > > >>> > > > > same
> > > > >>> > > > > > as
> > > > >>> > > > > > > > the
> > > > >>> > > > > > > > >> > > > previous
> > > > >>> > > > > > > > >> > > > > > > Produce Id tagged field in
> > > > >>> TransactionLogValue?
> > > > >>> > > > > > > > >> > > > > > >
> > > > >>> > > > > > > > >> > > > > > > 112. Since the kip changes the
> > > > inter-broker
> > > > >>> > > > protocol,
> > > > >>> > > > > > > should
> > > > >>> > > > > > > > >> we
> > > > >>> > > > > > > > >> > > bump
> > > > >>> > > > > > > > >> > > > up
> > > > >>> > > > > > > > >> > > > > > the
> > > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this feature only
> > for
> > > > the
> > > > >>> > KRaft
> > > > >>> > > > > mode?
> > > > >>> > > > > > > > >> > > > > > >
> > > > >>> > > > > > > > >> > > > > > > Thanks,
> > > > >>> > > > > > > > >> > > > > > >
> > > > >>> > > > > > > > >> > > > > > > Jun
> > > > >>> > > > > > > > >> > > > > > >
> > > > >>> > > > > > > > >> > > > > > >
> > > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at 11:13 AM
> > Justine
> > > > >>> Olshan
> > > > >>> > > > > > > > >> > > > > > > <jo...@confluent.io.invalid>
> wrote:
> > > > >>> > > > > > > > >> > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > > >>> > > > > > > > >> > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > I'm glad we are getting to
> > convergence
> > > > on
> > > > >>> the
> > > > >>> > > > > design.
> > > > >>> > > > > > :)
> > > > >>> > > > > > > > >> > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > While I understand it seems a
> little
> > > > >>> "weird".
> > > > >>> > > I'm
> > > > >>> > > > > not
> > > > >>> > > > > > > sure
> > > > >>> > > > > > > > >> what
> > > > >>> > > > > > > > >> > > the
> > > > >>> > > > > > > > >> > > > > > > benefit
> > > > >>> > > > > > > > >> > > > > > > > of writing an extra record to the
> > log.
> > > > >>> > > > > > > > >> > > > > > > > Is the concern a tool to describe
> > > > >>> transactions
> > > > >>> > > > won't
> > > > >>> > > > > > > work
> > > > >>> > > > > > > > >> (ie,
> > > > >>> > > > > > > > >> > > the
> > > > >>> > > > > > > > >> > > > > > > complete
> > > > >>> > > > > > > > >> > > > > > > > state is needed to calculate the
> > time
> > > > >>> since
> > > > >>> > the
> > > > >>> > > > > > > > transaction
> > > > >>> > > > > > > > >> > > > > completed?)
> > > > >>> > > > > > > > >> > > > > > > > If we have a reason like this, it
> is
> > > > >>> enough to
> > > > >>> > > > > > convince
> > > > >>> > > > > > > me
> > > > >>> > > > > > > > >> we
> > > > >>> > > > > > > > >> > > need
> > > > >>> > > > > > > > >> > > > > such
> > > > >>> > > > > > > > >> > > > > > > an
> > > > >>> > > > > > > > >> > > > > > > > extra record. It seems like it
> would
> > > be
> > > > >>> > > replacing
> > > > >>> > > > > the
> > > > >>> > > > > > > > record
> > > > >>> > > > > > > > >> > > > written
> > > > >>> > > > > > > > >> > > > > on
> > > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is this correct?
> > > > >>> > > > > > > > >> > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > > >>> > > > > > > > >> > > > > > > > Justine
> > > > >>> > > > > > > > >> > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024 at 5:14 PM
> Jun
> > > Rao
> > > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > > >>> > > > > > > > >> > > >
> > > > >>> > > > > > > > >> > > > > > > wrote:
> > > > >>> > > > > > > > >> > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > > >>> > > > > > > > >> > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > Thanks for the explanation. I
> > > > >>> understand the
> > > > >>> > > > > > intention
> > > > >>> > > > > > > > >> now.
> > > > >>> > > > > > > > >> > In
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > > > > overflow
> > > > >>> > > > > > > > >> > > > > > > > > case, we set the non-tagged
> field
> > to
> > > > >>> the old
> > > > >>> > > pid
> > > > >>> > > > > > (and
> > > > >>> > > > > > > > the
> > > > >>> > > > > > > > >> max
> > > > >>> > > > > > > > >> > > > > epoch)
> > > > >>> > > > > > > > >> > > > > > in
> > > > >>> > > > > > > > >> > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > prepare marker so that we could
> > > > >>> correctly
> > > > >>> > > write
> > > > >>> > > > > the
> > > > >>> > > > > > > > >> marker to
> > > > >>> > > > > > > > >> > > the
> > > > >>> > > > > > > > >> > > > > > data
> > > > >>> > > > > > > > >> > > > > > > > > partition if the broker
> > downgrades.
> > > > When
> > > > >>> > > writing
> > > > >>> > > > > the
> > > > >>> > > > > > > > >> complete
> > > > >>> > > > > > > > >> > > > > marker,
> > > > >>> > > > > > > > >> > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > know the marker has already been
> > > > >>> written to
> > > > >>> > > the
> > > > >>> > > > > data
> > > > >>> > > > > > > > >> > partition.
> > > > >>> > > > > > > > >> > > > We
> > > > >>> > > > > > > > >> > > > > > set
> > > > >>> > > > > > > > >> > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > non-tagged field to the new pid
> to
> > > > avoid
> > > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > > >>> > > > > > > > >> > > > > > in
> > > > >>> > > > > > > > >> > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > client if the broker downgrades.
> > > > >>> > > > > > > > >> > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > The above seems to work. It's
> > just a
> > > > bit
> > > > >>> > > > > > inconsistent
> > > > >>> > > > > > > > for
> > > > >>> > > > > > > > >> a
> > > > >>> > > > > > > > >> > > > prepare
> > > > >>> > > > > > > > >> > > > > > > > marker
> > > > >>> > > > > > > > >> > > > > > > > > and a complete marker to use
> > > different
> > > > >>> pids
> > > > >>> > in
> > > > >>> > > > > this
> > > > >>> > > > > > > > >> special
> > > > >>> > > > > > > > >> > > case.
> > > > >>> > > > > > > > >> > > > > If
> > > > >>> > > > > > > > >> > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > downgrade with the complete
> > marker,
> > > it
> > > > >>> seems
> > > > >>> > > > that
> > > > >>> > > > > we
> > > > >>> > > > > > > > will
> > > > >>> > > > > > > > >> > never
> > > > >>> > > > > > > > >> > > > be
> > > > >>> > > > > > > > >> > > > > > able
> > > > >>> > > > > > > > >> > > > > > > > to
> > > > >>> > > > > > > > >> > > > > > > > > write the complete marker with
> the
> > > old
> > > > >>> pid.
> > > > >>> > > Not
> > > > >>> > > > > sure
> > > > >>> > > > > > > if
> > > > >>> > > > > > > > it
> > > > >>> > > > > > > > >> > > causes
> > > > >>> > > > > > > > >> > > > > any
> > > > >>> > > > > > > > >> > > > > > > > > issue, but it seems a bit weird.
> > > > >>> Instead of
> > > > >>> > > > > writing
> > > > >>> > > > > > > the
> > > > >>> > > > > > > > >> > > complete
> > > > >>> > > > > > > > >> > > > > > marker
> > > > >>> > > > > > > > >> > > > > > > > > with the new pid, could we write
> > two
> > > > >>> > records:
> > > > >>> > > a
> > > > >>> > > > > > > complete
> > > > >>> > > > > > > > >> > marker
> > > > >>> > > > > > > > >> > > > > with
> > > > >>> > > > > > > > >> > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > old pid followed by a
> > > > >>> TransactionLogValue
> > > > >>> > with
> > > > >>> > > > the
> > > > >>> > > > > > new
> > > > >>> > > > > > > > pid
> > > > >>> > > > > > > > >> > and
> > > > >>> > > > > > > > >> > > an
> > > > >>> > > > > > > > >> > > > > > empty
> > > > >>> > > > > > > > >> > > > > > > > > state? We could make the two
> > records
> > > > in
> > > > >>> the
> > > > >>> > > same
> > > > >>> > > > > > batch
> > > > >>> > > > > > > > so
> > > > >>> > > > > > > > >> > that
> > > > >>> > > > > > > > >> > > > they
> > > > >>> > > > > > > > >> > > > > > > will
> > > > >>> > > > > > > > >> > > > > > > > be
> > > > >>> > > > > > > > >> > > > > > > > > added to the log atomically.
> > > > >>> > > > > > > > >> > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > > >>> > > > > > > > >> > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > Jun
> > > > >>> > > > > > > > >> > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024 at 5:40 PM
> > > > Justine
> > > > >>> > Olshan
> > > > >>> > > > > > > > >> > > > > > > > > <jo...@confluent.io.invalid>
> > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > >>> > > > > > > > >> > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare marker is
> > written,
> > > > >>> but the
> > > > >>> > > > > endTxn
> > > > >>> > > > > > > > >> response
> > > > >>> > > > > > > > >> > is
> > > > >>> > > > > > > > >> > > > not
> > > > >>> > > > > > > > >> > > > > > > > > received
> > > > >>> > > > > > > > >> > > > > > > > > > by the client when the server
> > > > >>> downgrades
> > > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare marker is
> > > written,
> > > > >>> the
> > > > >>> > > endTxn
> > > > >>> > > > > > > > response
> > > > >>> > > > > > > > >> is
> > > > >>> > > > > > > > >> > > > > received
> > > > >>> > > > > > > > >> > > > > > > by
> > > > >>> > > > > > > > >> > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > client when the server
> > downgrades.
> > > > >>> > > > > > > > >> > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > I think I am still a little
> > > > confused.
> > > > >>> In
> > > > >>> > > both
> > > > >>> > > > of
> > > > >>> > > > > > > these
> > > > >>> > > > > > > > >> > cases,
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > transaction log has the old
> > > producer
> > > > >>> ID.
> > > > >>> > We
> > > > >>> > > > > don't
> > > > >>> > > > > > > > write
> > > > >>> > > > > > > > >> the
> > > > >>> > > > > > > > >> > > new
> > > > >>> > > > > > > > >> > > > > > > > producer
> > > > >>> > > > > > > > >> > > > > > > > > ID
> > > > >>> > > > > > > > >> > > > > > > > > > in the prepare marker's non
> > tagged
> > > > >>> fields.
> > > > >>> > > > > > > > >> > > > > > > > > > If the server downgrades now,
> it
> > > > would
> > > > >>> > read
> > > > >>> > > > the
> > > > >>> > > > > > > > records
> > > > >>> > > > > > > > >> not
> > > > >>> > > > > > > > >> > > in
> > > > >>> > > > > > > > >> > > > > > tagged
> > > > >>> > > > > > > > >> > > > > > > > > > fields and the complete marker
> > > will
> > > > >>> also
> > > > >>> > > have
> > > > >>> > > > > the
> > > > >>> > > > > > > old
> > > > >>> > > > > > > > >> > > producer
> > > > >>> > > > > > > > >> > > > > ID.
> > > > >>> > > > > > > > >> > > > > > > > > > (If we had used the new
> producer
> > > ID,
> > > > >>> we
> > > > >>> > > would
> > > > >>> > > > > not
> > > > >>> > > > > > > have
> > > > >>> > > > > > > > >> > > > > > transactional
> > > > >>> > > > > > > > >> > > > > > > > > > correctness since the producer
> > id
> > > > >>> doesn't
> > > > >>> > > > match
> > > > >>> > > > > > the
> > > > >>> > > > > > > > >> > > transaction
> > > > >>> > > > > > > > >> > > > > and
> > > > >>> > > > > > > > >> > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > state would not be correct on
> > the
> > > > data
> > > > >>> > > > > partition.)
> > > > >>> > > > > > > > >> > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > In the overflow case, I'd
> expect
> > > the
> > > > >>> > > following
> > > > >>> > > > > to
> > > > >>> > > > > > > > >> happen on
> > > > >>> > > > > > > > >> > > the
> > > > >>> > > > > > > > >> > > > > > > client
> > > > >>> > > > > > > > >> > > > > > > > > side
> > > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we retry EndTxn --
> it
> > > is
> > > > >>> the
> > > > >>> > same
> > > > >>> > > > > > > producer
> > > > >>> > > > > > > > ID
> > > > >>> > > > > > > > >> > and
> > > > >>> > > > > > > > >> > > > > epoch
> > > > >>> > > > > > > > >> > > > > > -
> > > > >>> > > > > > > > >> > > > > > > 1
> > > > >>> > > > > > > > >> > > > > > > > > this
> > > > >>> > > > > > > > >> > > > > > > > > > would fence the producer
> > > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't retry
> EndTxn
> > > and
> > > > >>> use
> > > > >>> > the
> > > > >>> > > > new
> > > > >>> > > > > > > > >> producer id
> > > > >>> > > > > > > > >> > > > which
> > > > >>> > > > > > > > >> > > > > > > would
> > > > >>> > > > > > > > >> > > > > > > > > > result in
> > > InvalidPidMappingException
> > > > >>> > > > > > > > >> > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have special
> > handling
> > > > for
> > > > >>> > when
> > > > >>> > > a
> > > > >>> > > > > > server
> > > > >>> > > > > > > > >> > > > downgrades.
> > > > >>> > > > > > > > >> > > > > > When
> > > > >>> > > > > > > > >> > > > > > > > it
> > > > >>> > > > > > > > >> > > > > > > > > > reconnects we could get an API
> > > > version
> > > > >>> > > request
> > > > >>> > > > > > > showing
> > > > >>> > > > > > > > >> > > KIP-890
> > > > >>> > > > > > > > >> > > > > > part 2
> > > > >>> > > > > > > > >> > > > > > > > is
> > > > >>> > > > > > > > >> > > > > > > > > > not supported. In that case,
> we
> > > can
> > > > >>> call
> > > > >>> > > > > > > > initProducerId
> > > > >>> > > > > > > > >> to
> > > > >>> > > > > > > > >> > > > abort
> > > > >>> > > > > > > > >> > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > transaction. (In the overflow
> > > case,
> > > > >>> this
> > > > >>> > > > > correctly
> > > > >>> > > > > > > > gives
> > > > >>> > > > > > > > >> > us a
> > > > >>> > > > > > > > >> > > > new
> > > > >>> > > > > > > > >> > > > > > > > > producer
> > > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > > >>> > > > > > > > >> > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > I guess the corresponding case
> > > would
> > > > >>> be
> > > > >>> > > where
> > > > >>> > > > > the
> > > > >>> > > > > > > > >> *complete
> > > > >>> > > > > > > > >> > > > > marker
> > > > >>> > > > > > > > >> > > > > > > *is
> > > > >>> > > > > > > > >> > > > > > > > > > written but the endTxn is not
> > > > >>> received by
> > > > >>> > > the
> > > > >>> > > > > > client
> > > > >>> > > > > > > > and
> > > > >>> > > > > > > > >> > the
> > > > >>> > > > > > > > >> > > > > server
> > > > >>> > > > > > > > >> > > > > > > > > > downgrades? This would result
> in
> > > the
> > > > >>> > > > transaction
> > > > >>> > > > > > > > >> > coordinator
> > > > >>> > > > > > > > >> > > > > having
> > > > >>> > > > > > > > >> > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > new
> > > > >>> > > > > > > > >> > > > > > > > > > ID and not the old one.  If
> the
> > > > client
> > > > >>> > > > retries,
> > > > >>> > > > > it
> > > > >>> > > > > > > > will
> > > > >>> > > > > > > > >> > > receive
> > > > >>> > > > > > > > >> > > > > an
> > > > >>> > > > > > > > >> > > > > > > > > > InvalidPidMappingException.
> The
> > > > >>> > > InitProducerId
> > > > >>> > > > > > > > scenario
> > > > >>> > > > > > > > >> > above
> > > > >>> > > > > > > > >> > > > > would
> > > > >>> > > > > > > > >> > > > > > > > help
> > > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > > >>> > > > > > > > >> > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > To be clear, my compatibility
> > > story
> > > > is
> > > > >>> > meant
> > > > >>> > > > to
> > > > >>> > > > > > > > support
> > > > >>> > > > > > > > >> > > > > downgrades
> > > > >>> > > > > > > > >> > > > > > > > server
> > > > >>> > > > > > > > >> > > > > > > > > > side in keeping the
> > transactional
> > > > >>> > > correctness.
> > > > >>> > > > > > > Keeping
> > > > >>> > > > > > > > >> the
> > > > >>> > > > > > > > >> > > > client
> > > > >>> > > > > > > > >> > > > > > > from
> > > > >>> > > > > > > > >> > > > > > > > > > fencing itself is not the
> > > priority.
> > > > >>> > > > > > > > >> > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I can also
> add
> > > text
> > > > >>> in
> > > > >>> > the
> > > > >>> > > > KIP
> > > > >>> > > > > > > about
> > > > >>> > > > > > > > >> > > > > > InitProducerId
> > > > >>> > > > > > > > >> > > > > > > if
> > > > >>> > > > > > > > >> > > > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > think that fixes some edge
> > cases.
> > > > >>> > > > > > > > >> > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > > >>> > > > > > > > >> > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12, 2024 at
> 4:10 PM
> > > Jun
> > > > >>> Rao
> > > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > > >>> > > > > > > > >> > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > > >>> > > > > > > > >> > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the reply.
> > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > I agree that we don't need
> to
> > > > >>> optimize
> > > > >>> > for
> > > > >>> > > > > > fencing
> > > > >>> > > > > > > > >> during
> > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > >>> > > > > > > > >> > > > > > > > > > > Regarding consistency, there
> > are
> > > > two
> > > > >>> > > > possible
> > > > >>> > > > > > > cases:
> > > > >>> > > > > > > > >> (1)
> > > > >>> > > > > > > > >> > > the
> > > > >>> > > > > > > > >> > > > > > > prepare
> > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > >>> > > > > > > > >> > > > > > > > > > > is written, but the endTxn
> > > > response
> > > > >>> is
> > > > >>> > not
> > > > >>> > > > > > > received
> > > > >>> > > > > > > > by
> > > > >>> > > > > > > > >> > the
> > > > >>> > > > > > > > >> > > > > client
> > > > >>> > > > > > > > >> > > > > > > > when
> > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > server downgrades; (2)  the
> > > > prepare
> > > > >>> > marker
> > > > >>> > > > is
> > > > >>> > > > > > > > written,
> > > > >>> > > > > > > > >> > the
> > > > >>> > > > > > > > >> > > > > endTxn
> > > > >>> > > > > > > > >> > > > > > > > > > response
> > > > >>> > > > > > > > >> > > > > > > > > > > is received by the client
> when
> > > the
> > > > >>> > server
> > > > >>> > > > > > > > downgrades.
> > > > >>> > > > > > > > >> In
> > > > >>> > > > > > > > >> > > (1),
> > > > >>> > > > > > > > >> > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > client
> > > > >>> > > > > > > > >> > > > > > > > > > > will have the old produce Id
> > and
> > > > in
> > > > >>> (2),
> > > > >>> > > the
> > > > >>> > > > > > > client
> > > > >>> > > > > > > > >> will
> > > > >>> > > > > > > > >> > > have
> > > > >>> > > > > > > > >> > > > > the
> > > > >>> > > > > > > > >> > > > > > > new
> > > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If we downgrade
> > > right
> > > > >>> after
> > > > >>> > > the
> > > > >>> > > > > > > prepare
> > > > >>> > > > > > > > >> > marker,
> > > > >>> > > > > > > > >> > > > we
> > > > >>> > > > > > > > >> > > > > > > can't
> > > > >>> > > > > > > > >> > > > > > > > be
> > > > >>> > > > > > > > >> > > > > > > > > > > consistent to both (1) and
> (2)
> > > > >>> since we
> > > > >>> > > can
> > > > >>> > > > > only
> > > > >>> > > > > > > put
> > > > >>> > > > > > > > >> one
> > > > >>> > > > > > > > >> > > > value
> > > > >>> > > > > > > > >> > > > > in
> > > > >>> > > > > > > > >> > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > existing produce Id field.
> > It's
> > > > >>> also not
> > > > >>> > > > clear
> > > > >>> > > > > > > which
> > > > >>> > > > > > > > >> case
> > > > >>> > > > > > > > >> > > is
> > > > >>> > > > > > > > >> > > > > more
> > > > >>> > > > > > > > >> > > > > > > > > likely.
> > > > >>> > > > > > > > >> > > > > > > > > > > So we could probably be
> > > consistent
> > > > >>> with
> > > > >>> > > > either
> > > > >>> > > > > > > case.
> > > > >>> > > > > > > > >> By
> > > > >>> > > > > > > > >> > > > putting
> > > > >>> > > > > > > > >> > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > new
> > > > >>> > > > > > > > >> > > > > > > > > > > producer Id in the prepare
> > > marker,
> > > > >>> we
> > > > >>> > are
> > > > >>> > > > > > > consistent
> > > > >>> > > > > > > > >> with
> > > > >>> > > > > > > > >> > > > case
> > > > >>> > > > > > > > >> > > > > > (2)
> > > > >>> > > > > > > > >> > > > > > > > and
> > > > >>> > > > > > > > >> > > > > > > > > it
> > > > >>> > > > > > > > >> > > > > > > > > > > also has the slight benefit
> > that
> > > > the
> > > > >>> > > produce
> > > > >>> > > > > > field
> > > > >>> > > > > > > > in
> > > > >>> > > > > > > > >> the
> > > > >>> > > > > > > > >> > > > > prepare
> > > > >>> > > > > > > > >> > > > > > > and
> > > > >>> > > > > > > > >> > > > > > > > > > > complete marker are
> consistent
> > > in
> > > > >>> the
> > > > >>> > > > overflow
> > > > >>> > > > > > > case.
> > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12, 2024 at
> > 3:11 PM
> > > > >>> Justine
> > > > >>> > > > Olshan
> > > > >>> > > > > > > > >> > > > > > > > > > >
> <jolshan@confluent.io.invalid
> > >
> > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > In the case you describe,
> we
> > > > would
> > > > >>> > need
> > > > >>> > > to
> > > > >>> > > > > > have
> > > > >>> > > > > > > a
> > > > >>> > > > > > > > >> > delayed
> > > > >>> > > > > > > > >> > > > > > > request,
> > > > >>> > > > > > > > >> > > > > > > > > > send a
> > > > >>> > > > > > > > >> > > > > > > > > > > > successful EndTxn, and a
> > > > >>> successful
> > > > >>> > > > > > > > >> AddPartitionsToTxn
> > > > >>> > > > > > > > >> > > and
> > > > >>> > > > > > > > >> > > > > then
> > > > >>> > > > > > > > >> > > > > > > > have
> > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn request go
> > > > through
> > > > >>> for
> > > > >>> > a
> > > > >>> > > > > given
> > > > >>> > > > > > > > >> producer.
> > > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to figure out
> if
> > it
> > > > is
> > > > >>> > > possible
> > > > >>> > > > > for
> > > > >>> > > > > > > the
> > > > >>> > > > > > > > >> > client
> > > > >>> > > > > > > > >> > > > to
> > > > >>> > > > > > > > >> > > > > > > > > transition
> > > > >>> > > > > > > > >> > > > > > > > > > > if
> > > > >>> > > > > > > > >> > > > > > > > > > > > a previous request is
> > delayed
> > > > >>> > somewhere.
> > > > >>> > > > But
> > > > >>> > > > > > > yes,
> > > > >>> > > > > > > > in
> > > > >>> > > > > > > > >> > this
> > > > >>> > > > > > > > >> > > > > case
> > > > >>> > > > > > > > >> > > > > > I
> > > > >>> > > > > > > > >> > > > > > > > > think
> > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > > > would fence the client.
> > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > Not for the overflow case.
> > In
> > > > the
> > > > >>> > > overflow
> > > > >>> > > > > > case,
> > > > >>> > > > > > > > the
> > > > >>> > > > > > > > >> > > > producer
> > > > >>> > > > > > > > >> > > > > > ID
> > > > >>> > > > > > > > >> > > > > > > > and
> > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > epoch are different on the
> > > > marker
> > > > >>> and
> > > > >>> > on
> > > > >>> > > > the
> > > > >>> > > > > > new
> > > > >>> > > > > > > > >> > > > transaction.
> > > > >>> > > > > > > > >> > > > > > So
> > > > >>> > > > > > > > >> > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > want
> > > > >>> > > > > > > > >> > > > > > > > > > > > the marker to use the max
> > > epoch
> > > > >>> but
> > > > >>> > the
> > > > >>> > > > new
> > > > >>> > > > > > > > >> > transaction
> > > > >>> > > > > > > > >> > > > > should
> > > > >>> > > > > > > > >> > > > > > > > start
> > > > >>> > > > > > > > >> > > > > > > > > > > with
> > > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and epoch 0 in
> > the
> > > > >>> > > > transactional
> > > > >>> > > > > > > state.
> > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > In the server downgrade
> > case,
> > > we
> > > > >>> want
> > > > >>> > to
> > > > >>> > > > see
> > > > >>> > > > > > the
> > > > >>> > > > > > > > >> > producer
> > > > >>> > > > > > > > >> > > > ID
> > > > >>> > > > > > > > >> > > > > as
> > > > >>> > > > > > > > >> > > > > > > > that
> > > > >>> > > > > > > > >> > > > > > > > > is
> > > > >>> > > > > > > > >> > > > > > > > > > > > what the client will have.
> > If
> > > we
> > > > >>> > > complete
> > > > >>> > > > > the
> > > > >>> > > > > > > > >> commit,
> > > > >>> > > > > > > > >> > and
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > > >>> > > > > > > > >> > > > > > > > > > > > state is reloaded, we need
> > the
> > > > new
> > > > >>> > > > producer
> > > > >>> > > > > ID
> > > > >>> > > > > > > in
> > > > >>> > > > > > > > >> the
> > > > >>> > > > > > > > >> > > state
> > > > >>> > > > > > > > >> > > > > so
> > > > >>> > > > > > > > >> > > > > > > > there
> > > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > > >>> > > > > > > > >> > > > > > > > > > > > an invalid producer ID
> > > mapping.
> > > > >>> > > > > > > > >> > > > > > > > > > > > The server downgrade cases
> > are
> > > > >>> > > considering
> > > > >>> > > > > > > > >> > transactional
> > > > >>> > > > > > > > >> > > > > > > > correctness
> > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > >>> > > > > > > > >> > > > > > > > > > > > not regressing from
> previous
> > > > >>> behavior
> > > > >>> > --
> > > > >>> > > > and
> > > > >>> > > > > > are
> > > > >>> > > > > > > > not
> > > > >>> > > > > > > > >> > > > > concerned
> > > > >>> > > > > > > > >> > > > > > > > about
> > > > >>> > > > > > > > >> > > > > > > > > > > > supporting the safety from
> > > > fencing
> > > > >>> > > retries
> > > > >>> > > > > (as
> > > > >>> > > > > > > we
> > > > >>> > > > > > > > >> have
> > > > >>> > > > > > > > >> > > > > > downgraded
> > > > >>> > > > > > > > >> > > > > > > > so
> > > > >>> > > > > > > > >> > > > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > > > don't need to support).
> > > Perhaps
> > > > >>> this
> > > > >>> > is
> > > > >>> > > a
> > > > >>> > > > > > trade
> > > > >>> > > > > > > > off,
> > > > >>> > > > > > > > >> > but
> > > > >>> > > > > > > > >> > > I
> > > > >>> > > > > > > > >> > > > > > think
> > > > >>> > > > > > > > >> > > > > > > it
> > > > >>> > > > > > > > >> > > > > > > > > is
> > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > (If the client downgrades,
> > it
> > > > will
> > > > >>> > have
> > > > >>> > > > > > > restarted
> > > > >>> > > > > > > > >> and
> > > > >>> > > > > > > > >> > it
> > > > >>> > > > > > > > >> > > is
> > > > >>> > > > > > > > >> > > > > ok
> > > > >>> > > > > > > > >> > > > > > > for
> > > > >>> > > > > > > > >> > > > > > > > it
> > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > >>> > > > > > > > >> > > > > > > > > > > > have a new producer ID
> too).
> > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan 12, 2024 at
> > > 11:42 AM
> > > > >>> Jun
> > > > >>> > Rao
> > > > >>> > > > > > > > >> > > > > > > <jun@confluent.io.invalid
> > > > >>> > > > > > > > >> > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for the reply.
> > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If the marker is
> > > > written
> > > > >>> by
> > > > >>> > the
> > > > >>> > > > new
> > > > >>> > > > > > > > >> client, we
> > > > >>> > > > > > > > >> > > can
> > > > >>> > > > > > > > >> > > > > as
> > > > >>> > > > > > > > >> > > > > > I
> > > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > >>> > > > > > > > >> > > > > > > > > > > > > the last email guarantee
> > > that
> > > > >>> any
> > > > >>> > > EndTxn
> > > > >>> > > > > > > > requests
> > > > >>> > > > > > > > >> > with
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > > same
> > > > >>> > > > > > > > >> > > > > > > > > epoch
> > > > >>> > > > > > > > >> > > > > > > > > > > are
> > > > >>> > > > > > > > >> > > > > > > > > > > > > from the same producer
> and
> > > the
> > > > >>> same
> > > > >>> > > > > > > transaction.
> > > > >>> > > > > > > > >> Then
> > > > >>> > > > > > > > >> > > we
> > > > >>> > > > > > > > >> > > > > > don't
> > > > >>> > > > > > > > >> > > > > > > > have
> > > > >>> > > > > > > > >> > > > > > > > > > to
> > > > >>> > > > > > > > >> > > > > > > > > > > > > return a fenced error
> but
> > > can
> > > > >>> handle
> > > > >>> > > > > > > gracefully
> > > > >>> > > > > > > > as
> > > > >>> > > > > > > > >> > > > > described
> > > > >>> > > > > > > > >> > > > > > in
> > > > >>> > > > > > > > >> > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > > >>> > > > > > > > >> > > > > > > > > > > > > When a delayed EndTnx
> > > request
> > > > is
> > > > >>> > > > > processed,
> > > > >>> > > > > > > the
> > > > >>> > > > > > > > >> txn
> > > > >>> > > > > > > > >> > > state
> > > > >>> > > > > > > > >> > > > > > could
> > > > >>> > > > > > > > >> > > > > > > > be
> > > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > > >>> > > > > > > > >> > > > > > > > > > > > > for the next txn. I
> guess
> > in
> > > > >>> this
> > > > >>> > case
> > > > >>> > > > we
> > > > >>> > > > > > > still
> > > > >>> > > > > > > > >> > return
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > > > fenced
> > > > >>> > > > > > > > >> > > > > > > > > > error
> > > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed request?
> > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry, my question
> > was
> > > > >>> > > inaccurate.
> > > > >>> > > > > What
> > > > >>> > > > > > > you
> > > > >>> > > > > > > > >> > > > described
> > > > >>> > > > > > > > >> > > > > is
> > > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > "The downgrade
> > > compatibility I
> > > > >>> > mention
> > > > >>> > > > is
> > > > >>> > > > > > that
> > > > >>> > > > > > > > we
> > > > >>> > > > > > > > >> > keep
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > > same
> > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in the main
> > > > >>> (non-tagged)
> > > > >>> > > > fields
> > > > >>> > > > > as
> > > > >>> > > > > > > we
> > > > >>> > > > > > > > >> did
> > > > >>> > > > > > > > >> > > > before
> > > > >>> > > > > > > > >> > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > code
> > > > >>> > > > > > > > >> > > > > > > > > > on
> > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > server side." If we want
> > to
> > > do
> > > > >>> this,
> > > > >>> > > it
> > > > >>> > > > > > seems
> > > > >>> > > > > > > > >> that we
> > > > >>> > > > > > > > >> > > > > should
> > > > >>> > > > > > > > >> > > > > > > use
> > > > >>> > > > > > > > >> > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > current produce Id and
> max
> > > > >>> epoch in
> > > > >>> > > the
> > > > >>> > > > > > > existing
> > > > >>> > > > > > > > >> > > > producerId
> > > > >>> > > > > > > > >> > > > > > and
> > > > >>> > > > > > > > >> > > > > > > > > > > > > producerEpoch fields for
> > > both
> > > > >>> the
> > > > >>> > > > prepare
> > > > >>> > > > > > and
> > > > >>> > > > > > > > the
> > > > >>> > > > > > > > >> > > > complete
> > > > >>> > > > > > > > >> > > > > > > > marker,
> > > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > > >>> > > > > > > > >> > > > > > > > > > > > > The downgrade can happen
> > > after
> > > > >>> the
> > > > >>> > > > > complete
> > > > >>> > > > > > > > >> marker is
> > > > >>> > > > > > > > >> > > > > > written.
> > > > >>> > > > > > > > >> > > > > > > > With
> > > > >>> > > > > > > > >> > > > > > > > > > > what
> > > > >>> > > > > > > > >> > > > > > > > > > > > > you described, the
> > > downgraded
> > > > >>> > > > coordinator
> > > > >>> > > > > > will
> > > > >>> > > > > > > > see
> > > > >>> > > > > > > > >> > the
> > > > >>> > > > > > > > >> > > > new
> > > > >>> > > > > > > > >> > > > > > > > produce
> > > > >>> > > > > > > > >> > > > > > > > > Id
> > > > >>> > > > > > > > >> > > > > > > > > > > > > instead of the old one.
> > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan 12, 2024 at
> > > > 10:44 AM
> > > > >>> > > Justine
> > > > >>> > > > > > > Olshan
> > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > <jolshan@confluent.io.invalid
> > > > >
> > > > >>> > wrote:
> > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > I can update the
> > > > description.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe your second
> > > point
> > > > is
> > > > >>> > > > mentioned
> > > > >>> > > > > > in
> > > > >>> > > > > > > > the
> > > > >>> > > > > > > > >> > KIP.
> > > > >>> > > > > > > > >> > > I
> > > > >>> > > > > > > > >> > > > > can
> > > > >>> > > > > > > > >> > > > > > > add
> > > > >>> > > > > > > > >> > > > > > > > > more
> > > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it is helpful.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The delayed message
> > case
> > > > can
> > > > >>> > also
> > > > >>> > > > > > violate
> > > > >>> > > > > > > > EOS
> > > > >>> > > > > > > > >> if
> > > > >>> > > > > > > > >> > > the
> > > > >>> > > > > > > > >> > > > > > > delayed
> > > > >>> > > > > > > > >> > > > > > > > > > > message
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in after the
> next
> > > > >>> > > > > addPartitionsToTxn
> > > > >>> > > > > > > > >> request
> > > > >>> > > > > > > > >> > > > comes
> > > > >>> > > > > > > > >> > > > > > in.
> > > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a message
> from a
> > > > >>> previous
> > > > >>> > > > > > (aborted)
> > > > >>> > > > > > > > >> > > transaction
> > > > >>> > > > > > > > >> > > > > > > become
> > > > >>> > > > > > > > >> > > > > > > > > part
> > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > next transaction.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > If the marker is
> written
> > > by
> > > > >>> the
> > > > >>> > new
> > > > >>> > > > > > client,
> > > > >>> > > > > > > we
> > > > >>> > > > > > > > >> can
> > > > >>> > > > > > > > >> > > as I
> > > > >>> > > > > > > > >> > > > > > > > mentioned
> > > > >>> > > > > > > > >> > > > > > > > > > in
> > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > last email guarantee
> > that
> > > > any
> > > > >>> > EndTxn
> > > > >>> > > > > > > requests
> > > > >>> > > > > > > > >> with
> > > > >>> > > > > > > > >> > > the
> > > > >>> > > > > > > > >> > > > > same
> > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > >>> > > > > > > > >> > > > > > > > > > are
> > > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > the same producer and
> > the
> > > > same
> > > > >>> > > > > > transaction.
> > > > >>> > > > > > > > >> Then we
> > > > >>> > > > > > > > >> > > > don't
> > > > >>> > > > > > > > >> > > > > > > have
> > > > >>> > > > > > > > >> > > > > > > > to
> > > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced error but can
> > > handle
> > > > >>> > > gracefully
> > > > >>> > > > > as
> > > > >>> > > > > > > > >> described
> > > > >>> > > > > > > > >> > > in
> > > > >>> > > > > > > > >> > > > > the
> > > > >>> > > > > > > > >> > > > > > > KIP.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't think a
> boolean
> > is
> > > > >>> useful
> > > > >>> > > > since
> > > > >>> > > > > it
> > > > >>> > > > > > > is
> > > > >>> > > > > > > > >> > > directly
> > > > >>> > > > > > > > >> > > > > > > encoded
> > > > >>> > > > > > > > >> > > > > > > > by
> > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > existence or lack of
> the
> > > > >>> tagged
> > > > >>> > > field
> > > > >>> > > > > > being
> > > > >>> > > > > > > > >> > written.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the prepare marker
> we
> > > > will
> > > > >>> have
> > > > >>> > > the
> > > > >>> > > > > > same
> > > > >>> > > > > > > > >> > producer
> > > > >>> > > > > > > > >> > > ID
> > > > >>> > > > > > > > >> > > > > in
> > > > >>> > > > > > > > >> > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In the Complete
> > > state
> > > > >>> we
> > > > >>> > may
> > > > >>> > > > not.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure why the
> > > ongoing
> > > > >>> state
> > > > >>> > > > > matters
> > > > >>> > > > > > > for
> > > > >>> > > > > > > > >> this
> > > > >>> > > > > > > > >> > > > KIP.
> > > > >>> > > > > > > > >> > > > > It
> > > > >>> > > > > > > > >> > > > > > > > does
> > > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure what you
> > are
> > > > >>> > referring
> > > > >>> > > to
> > > > >>> > > > > > about
> > > > >>> > > > > > > > >> > writing
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > > > > previous
> > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the prepare
> > marker.
> > > > >>> This is
> > > > >>> > > not
> > > > >>> > > > in
> > > > >>> > > > > > the
> > > > >>> > > > > > > > >> KIP.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > In the overflow case,
> we
> > > > >>> write the
> > > > >>> > > > > > > > >> nextProducerId
> > > > >>> > > > > > > > >> > in
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > > > > prepare
> > > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so we know
> what
> > we
> > > > >>> > assigned
> > > > >>> > > > when
> > > > >>> > > > > > we
> > > > >>> > > > > > > > >> reload
> > > > >>> > > > > > > > >> > > the
> > > > >>> > > > > > > > >> > > > > > > > > transaction
> > > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we complete, we
> > > > >>> transition
> > > > >>> > this
> > > > >>> > > > ID
> > > > >>> > > > > to
> > > > >>> > > > > > > the
> > > > >>> > > > > > > > >> main
> > > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > have the previous
> > producer
> > > > ID
> > > > >>> > field
> > > > >>> > > > > filled
> > > > >>> > > > > > > in.
> > > > >>> > > > > > > > >> This
> > > > >>> > > > > > > > >> > > is
> > > > >>> > > > > > > > >> > > > so
> > > > >>> > > > > > > > >> > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > can
> > > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry case the
> > > > operation
> > > > >>> > > > completed
> > > > >>> > > > > > > > >> > successfully
> > > > >>> > > > > > > > >> > > > and
> > > > >>> > > > > > > > >> > > > > we
> > > > >>> > > > > > > > >> > > > > > > > don't
> > > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer. The
> downgrade
> > > > >>> > > compatibility
> > > > >>> > > > I
> > > > >>> > > > > > > > mention
> > > > >>> > > > > > > > >> is
> > > > >>> > > > > > > > >> > > that
> > > > >>> > > > > > > > >> > > > > we
> > > > >>> > > > > > > > >> > > > > > > keep
> > > > >>> > > > > > > > >> > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer ID and epoch
> in
> > > the
> > > > >>> main
> > > > >>> > > > > > > (non-tagged)
> > > > >>> > > > > > > > >> > fields
> > > > >>> > > > > > > > >> > > > as
> > > > >>> > > > > > > > >> > > > > we
> > > > >>> > > > > > > > >> > > > > > > did
> > > > >>> > > > > > > > >> > > > > > > > > > > before
> > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > code on the server
> side.
> > > If
> > > > >>> the
> > > > >>> > > server
> > > > >>> > > > > > > > >> downgrades,
> > > > >>> > > > > > > > >> > we
> > > > >>> > > > > > > > >> > > > are
> > > > >>> > > > > > > > >> > > > > > > still
> > > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > This addresses both
> the
> > > > >>> prepare
> > > > >>> > and
> > > > >>> > > > > > complete
> > > > >>> > > > > > > > >> state
> > > > >>> > > > > > > > >> > > > > > > downgrades.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri, Jan 12, 2024
> at
> > > > >>> 10:21 AM
> > > > >>> > Jun
> > > > >>> > > > Rao
> > > > >>> > > > > > > > >> > > > > > > > > <jun@confluent.io.invalid
> > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi, Justine,
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks for the
> reply.
> > > > Sorry
> > > > >>> for
> > > > >>> > > the
> > > > >>> > > > > > > delay. I
> > > > >>> > > > > > > > >> > have a
> > > > >>> > > > > > > > >> > > > few
> > > > >>> > > > > > > > >> > > > > > > more
> > > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I think the
> > > > motivation
> > > > >>> > > section
> > > > >>> > > > > > could
> > > > >>> > > > > > > be
> > > > >>> > > > > > > > >> > > > improved.
> > > > >>> > > > > > > > >> > > > > > One
> > > > >>> > > > > > > > >> > > > > > > of
> > > > >>> > > > > > > > >> > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > motivations listed
> by
> > > the
> > > > >>> KIP is
> > > > >>> > > > "This
> > > > >>> > > > > > can
> > > > >>> > > > > > > > >> happen
> > > > >>> > > > > > > > >> > > > when
> > > > >>> > > > > > > > >> > > > > a
> > > > >>> > > > > > > > >> > > > > > > > > message
> > > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > or delayed due to
> > > > networking
> > > > >>> > > issues
> > > > >>> > > > > or a
> > > > >>> > > > > > > > >> network
> > > > >>> > > > > > > > >> > > > > > partition,
> > > > >>> > > > > > > > >> > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > transaction
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts, and then the
> > > > delayed
> > > > >>> > > message
> > > > >>> > > > > > > finally
> > > > >>> > > > > > > > >> > comes
> > > > >>> > > > > > > > >> > > > > in.".
> > > > >>> > > > > > > > >> > > > > > > This
> > > > >>> > > > > > > > >> > > > > > > > > > seems
> > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > very accurate.
> Without
> > > > >>> KIP-890,
> > > > >>> > > > > > currently,
> > > > >>> > > > > > > > if
> > > > >>> > > > > > > > >> the
> > > > >>> > > > > > > > >> > > > > > > coordinator
> > > > >>> > > > > > > > >> > > > > > > > > > times
> > > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts an ongoing
> > > > >>> transaction,
> > > > >>> > it
> > > > >>> > > > > > already
> > > > >>> > > > > > > > >> bumps
> > > > >>> > > > > > > > >> > up
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > > > epoch
> > > > >>> > > > > > > > >> > > > > > > > in
> > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > which prevents the
> > > delayed
> > > > >>> > produce
> > > > >>> > > > > > message
> > > > >>> > > > > > > > >> from
> > > > >>> > > > > > > > >> > > being
> > > > >>> > > > > > > > >> > > > > > added
> > > > >>> > > > > > > > >> > > > > > > > to
> > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > partition. What can
> > > cause
> > > > a
> > > > >>> > > hanging
> > > > >>> > > > > > > > >> transaction
> > > > >>> > > > > > > > >> > is
> > > > >>> > > > > > > > >> > > > that
> > > > >>> > > > > > > > >> > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completes (either
> > aborts
> > > > or
> > > > >>> > > > commits) a
> > > > >>> > > > > > > > >> > transaction
> > > > >>> > > > > > > > >> > > > > before
> > > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > successful ack on
> > > messages
> > > > >>> > > published
> > > > >>> > > > > in
> > > > >>> > > > > > > the
> > > > >>> > > > > > > > >> same
> > > > >>> > > > > > > > >> > > txn.
> > > > >>> > > > > > > > >> > > > > In
> > > > >>> > > > > > > > >> > > > > > > this
> > > > >>> > > > > > > > >> > > > > > > > > > case,
> > > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible for the
> > delayed
> > > > >>> message
> > > > >>> > > to
> > > > >>> > > > be
> > > > >>> > > > > > > > >> appended
> > > > >>> > > > > > > > >> > to
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > > > > > partition
> > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker, causing a
> > > > >>> transaction to
> > > > >>> > > > hang.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > A similar issue (not
> > > > >>> mentioned
> > > > >>> > in
> > > > >>> > > > the
> > > > >>> > > > > > > > >> motivation)
> > > > >>> > > > > > > > >> > > > could
> > > > >>> > > > > > > > >> > > > > > > > happen
> > > > >>> > > > > > > > >> > > > > > > > > on
> > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker in the
> > > > coordinator's
> > > > >>> log.
> > > > >>> > > For
> > > > >>> > > > > > > > example,
> > > > >>> > > > > > > > >> > it's
> > > > >>> > > > > > > > >> > > > > > possible
> > > > >>> > > > > > > > >> > > > > > > > for
> > > > >>> > > > > > > > >> > > > > > > > > > an
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest to be
> > > > delayed
> > > > >>> on
> > > > >>> > the
> > > > >>> > > > > > > > >> coordinator.
> > > > >>> > > > > > > > >> > By
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > > time
> > > > >>> > > > > > > > >> > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest is
> > > > processed,
> > > > >>> it's
> > > > >>> > > > > > possible
> > > > >>> > > > > > > > that
> > > > >>> > > > > > > > >> > the
> > > > >>> > > > > > > > >> > > > > > previous
> > > > >>> > > > > > > > >> > > > > > > > txn
> > > > >>> > > > > > > > >> > > > > > > > > > has
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > completed and a new
> > txn
> > > > has
> > > > >>> > > started.
> > > > >>> > > > > > > > >> Currently,
> > > > >>> > > > > > > > >> > > since
> > > > >>> > > > > > > > >> > > > > the
> > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > >>> > > > > > > > >> > > > > > > > > > is
> > > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped on every txn,
> > the
> > > > >>> delayed
> > > > >>> > > > > > > > EndTxnRequest
> > > > >>> > > > > > > > >> > will
> > > > >>> > > > > > > > >> > > > add
> > > > >>> > > > > > > > >> > > > > > an
> > > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare marker (and
> > > > >>> eventually a
> > > > >>> > > > > > complete
> > > > >>> > > > > > > > >> marker)
> > > > >>> > > > > > > > >> > > to
> > > > >>> > > > > > > > >> > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't cause the
> > > > transaction
> > > > >>> to
> > > > >>> > > hang,
> > > > >>> > > > > but
> > > > >>> > > > > > > it
> > > > >>> > > > > > > > >> will
> > > > >>> > > > > > > > >> > > > break
> > > > >>> > > > > > > > >> > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > EoS
> > > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > The proposal in this
> > KIP
> > > > >>> will
> > > > >>> > > > address
> > > > >>> > > > > > this
> > > > >>> > > > > > > > >> issue
> > > > >>> > > > > > > > >> > > too.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101. "However, I was
> > > > >>> writing it
> > > > >>> > so
> > > > >>> > > > > that
> > > > >>> > > > > > we
> > > > >>> > > > > > > > can
> > > > >>> > > > > > > > >> > > > > > distinguish
> > > > >>> > > > > > > > >> > > > > > > > > > between
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > old clients where we
> > > don't
> > > > >>> have
> > > > >>> > > the
> > > > >>> > > > > > > ability
> > > > >>> > > > > > > > do
> > > > >>> > > > > > > > >> > this
> > > > >>> > > > > > > > >> > > > > > > operation
> > > > >>> > > > > > > > >> > > > > > > > > and
> > > > >>> > > > > > > > >> > > > > > > > > > > new
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients that can.
> (Old
> > > > >>> clients
> > > > >>> > > don't
> > > > >>> > > > > > bump
> > > > >>> > > > > > > > the
> > > > >>> > > > > > > > >> > epoch
> > > > >>> > > > > > > > >> > > > on
> > > > >>> > > > > > > > >> > > > > > > > commit,
> > > > >>> > > > > > > > >> > > > > > > > > so
> > > > >>> > > > > > > > >> > > > > > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for sure the
> write
> > > > >>> belongs
> > > > >>> > to
> > > > >>> > > > the
> > > > >>> > > > > > > given
> > > > >>> > > > > > > > >> > > > > > transaction)."
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I am wondering
> > why
> > > > we
> > > > >>> need
> > > > >>> > > to
> > > > >>> > > > > > > > >> distinguish
> > > > >>> > > > > > > > >> > > > whether
> > > > >>> > > > > > > > >> > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > marker
> > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > written by the old
> and
> > > the
> > > > >>> new
> > > > >>> > > > client.
> > > > >>> > > > > > > Could
> > > > >>> > > > > > > > >> you
> > > > >>> > > > > > > > >> > > > > describe
> > > > >>> > > > > > > > >> > > > > > > > what
> > > > >>> > > > > > > > >> > > > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > > do
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > differently if we
> know
> > > the
> > > > >>> > marker
> > > > >>> > > is
> > > > >>> > > > > > > written
> > > > >>> > > > > > > > >> by
> > > > >>> > > > > > > > >> > the
> > > > >>> > > > > > > > >> > > > new
> > > > >>> > > > > > > > >> > > > > > > > client?
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If we do need
> a
> > > way
> > > > to
> > > > >>> > > > > distinguish
> > > > >>> > > > > > > > >> whether
> > > > >>> > > > > > > > >> > > the
> > > > >>> > > > > > > > >> > > > > > marker
> > > > >>> > > > > > > > >> > > > > > > > is
> > > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old and the new
> > > > client.
> > > > >>> > Would
> > > > >>> > > it
> > > > >>> > > > > be
> > > > >>> > > > > > > > >> simpler
> > > > >>> > > > > > > > >> > to
> > > > >>> > > > > > > > >> > > > just
> > > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > field instead of
> > > > indirectly
> > > > >>> > > through
> > > > >>> > > > > the
> > > > >>> > > > > > > > >> previous
> > > > >>> > > > > > > > >> > > > > produce
> > > > >>> > > > > > > > >> > > > > > ID
> > > > >>> > > > > > > > >> > > > > > > > > > field?
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3 It's not clear
> > to
> > > me
> > > > >>> why
> > > > >>> > we
> > > > >>> > > > only
> > > > >>> > > > > > add
> > > > >>> > > > > > > > the
> > > > >>> > > > > > > > >> > > > previous
> > > > >>> > > > > > > > >> > > > > > > > produce
> > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the complete marker,
> > but
> > > > >>> not in
> > > > >>> > > the
> > > > >>> > > > > > > prepare
> > > > >>> > > > > > > > >> > marker.
> > > > >>> > > > > > > > >> > > > If
> > > > >>> > > > > > > > >> > > > > we
> > > > >>> > > > > > > > >> > > > > > > > want
> > > > >>> > > > > > > > >> > > > > > > > > to
> > > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether a marker is
> > > > written
> > > > >>> by
> > > > >>> > the
> > > > >>> > > > new
> > > > >>> > > > > > > > client
> > > > >>> > > > > > > > >> or
> > > > >>> > > > > > > > >> > > not,
> > > > >>> > > > > > > > >> > > > > it
> > > > >>> > > > > > > > >> > > > > > > > seems
> > > > >>> > > > > > > > >> > > > > > > > > > that
> > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do this
> > consistently
> > > > for
> > > > >>> all
> > > > >>> > > > > markers.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4 What about the
> > > > >>> > > > > TransactionLogValue
> > > > >>> > > > > > > > >> record
> > > > >>> > > > > > > > >> > > > > > > representing
> > > > >>> > > > > > > > >> > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > state? Should we
> also
> > > > >>> > distinguish
> > > > >>> > > > > > whether
> > > > >>> > > > > > > > it's
> > > > >>> > > > > > > > >> > > > written
> > > > >>> > > > > > > > >> > > > > by
> > > > >>> > > > > > > > >> > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > old
> > > > >>> > > > > > > > >> > > > > > > > > > > or
> > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > new client?
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In the overflow
> > > case,
> > > > >>> it's
> > > > >>> > > > still
> > > > >>> > > > > > not
> > > > >>> > > > > > > > >> clear
> > > > >>> > > > > > > > >> > to
> > > > >>> > > > > > > > >> > > me
> > > > >>> > > > > > > > >> > > > > why
> > > > >>> > > > > > > > >> > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > write
> > > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous produce Id
> in
> > > the
> > > > >>> > prepare
> > > > >>> > > > > > marker
> > > > >>> > > > > > > > >> while
> > > > >>> > > > > > > > >> > > > writing
> > > > >>> > > > > > > > >> > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > next
> > > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the complete
> > marker.
> > > > You
> > > > >>> > > > mentioned
> > > > >>> > > > > > that
> > > > >>> > > > > > > > >> it's
> > > > >>> > > > > > > > >> > for
> > > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could downgrade
> > with
> > > > >>> either
> > > > >>> > the
> > > > >>> > > > > > prepare
> > > > >>> > > > > > > > >> marker
> > > > >>> > > > > > > > >> > > or
> > > > >>> > > > > > > > >> > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > complete
> > > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > In either case, the
> > > > >>> downgraded
> > > > >>> > > > > > coordinator
> > > > >>> > > > > > > > >> should
> > > > >>> > > > > > > > >> > > see
> > > > >>> > > > > > > > >> > > > > the
> > > > >>> > > > > > > > >> > > > > > > > same
> > > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > (probably the
> previous
> > > > >>> produce
> > > > >>> > > Id),
> > > > >>> > > > > > right?
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed, Dec 20, 2023
> > at
> > > > >>> 6:00 PM
> > > > >>> > > > > Justine
> > > > >>> > > > > > > > Olshan
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > >>> <jo...@confluent.io.invalid>
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey Jun,
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks for taking
> a
> > > look
> > > > >>> at
> > > > >>> > the
> > > > >>> > > > KIP
> > > > >>> > > > > > > again.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100. For the epoch
> > > > >>> overflow
> > > > >>> > > case,
> > > > >>> > > > > only
> > > > >>> > > > > > > the
> > > > >>> > > > > > > > >> > marker
> > > > >>> > > > > > > > >> > > > > will
> > > > >>> > > > > > > > >> > > > > > > have
> > > > >>> > > > > > > > >> > > > > > > > > max
> > > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps the behavior
> > of
> > > > the
> > > > >>> rest
> > > > >>> > > of
> > > > >>> > > > > the
> > > > >>> > > > > > > > >> markers
> > > > >>> > > > > > > > >> > > where
> > > > >>> > > > > > > > >> > > > > the
> > > > >>> > > > > > > > >> > > > > > > > last
> > > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch of the
> > > transaction
> > > > >>> > > records +
> > > > >>> > > > > 1.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101. You are
> correct
> > > > that
> > > > >>> we
> > > > >>> > > don't
> > > > >>> > > > > > need
> > > > >>> > > > > > > to
> > > > >>> > > > > > > > >> > write
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > > > > producer
> > > > >>> > > > > > > > >> > > > > > > > > > ID
> > > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the same.
> > However,
> > > I
> > > > >>> was
> > > > >>> > > > writing
> > > > >>> > > > > it
> > > > >>> > > > > > > so
> > > > >>> > > > > > > > >> that
> > > > >>> > > > > > > > >> > we
> > > > >>> > > > > > > > >> > > > can
> > > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old clients where
> we
> > > > don't
> > > > >>> > have
> > > > >>> > > > the
> > > > >>> > > > > > > > ability
> > > > >>> > > > > > > > >> do
> > > > >>> > > > > > > > >> > > this
> > > > >>> > > > > > > > >> > > > > > > > operation
> > > > >>> > > > > > > > >> > > > > > > > > > and
> > > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > clients that can.
> > (Old
> > > > >>> clients
> > > > >>> > > > don't
> > > > >>> > > > > > > bump
> > > > >>> > > > > > > > >> the
> > > > >>> > > > > > > > >> > > epoch
> > > > >>> > > > > > > > >> > > > > on
> > > > >>> > > > > > > > >> > > > > > > > > commit,
> > > > >>> > > > > > > > >> > > > > > > > > > so
> > > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say for sure the
> > write
> > > > >>> belongs
> > > > >>> > > to
> > > > >>> > > > > the
> > > > >>> > > > > > > > given
> > > > >>> > > > > > > > >> > > > > > transaction).
> > > > >>> > > > > > > > >> > > > > > > > If
> > > > >>> > > > > > > > >> > > > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn request
> from
> > a
> > > > new
> > > > >>> > > client,
> > > > >>> > > > we
> > > > >>> > > > > > > will
> > > > >>> > > > > > > > >> fill
> > > > >>> > > > > > > > >> > > this
> > > > >>> > > > > > > > >> > > > > > > field.
> > > > >>> > > > > > > > >> > > > > > > > We
> > > > >>> > > > > > > > >> > > > > > > > > > can
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > guarantee
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that any EndTxn
> > > requests
> > > > >>> with
> > > > >>> > > the
> > > > >>> > > > > same
> > > > >>> > > > > > > > epoch
> > > > >>> > > > > > > > >> > are
> > > > >>> > > > > > > > >> > > > from
> > > > >>> > > > > > > > >> > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > same
> > > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the same
> > transaction.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102. In prepare
> > phase,
> > > > we
> > > > >>> have
> > > > >>> > > the
> > > > >>> > > > > > same
> > > > >>> > > > > > > > >> > producer
> > > > >>> > > > > > > > >> > > ID
> > > > >>> > > > > > > > >> > > > > and
> > > > >>> > > > > > > > >> > > > > > > > epoch
> > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had. It is the
> > > producer
> > > > >>> ID and
> > > > >>> > > > epoch
> > > > >>> > > > > > > that
> > > > >>> > > > > > > > >> are
> > > > >>> > > > > > > > >> > on
> > > > >>> > > > > > > > >> > > > the
> > > > >>> > > > > > > > >> > > > > > > > marker.
> > > > >>> > > > > > > > >> > > > > > > > > In
> > > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase, we stay the
> > > same
> > > > >>> unless
> > > > >>> > > it
> > > > >>> > > > is
> > > > >>> > > > > > the
> > > > >>> > > > > > > > >> > overflow
> > > > >>> > > > > > > > >> > > > > case.
> > > > >>> > > > > > > > >> > > > > > > In
> > > > >>> > > > > > > > >> > > > > > > > > that
> > > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set the producer
> ID
> > to
> > > > >>> the new
> > > > >>> > > one
> > > > >>> > > > > we
> > > > >>> > > > > > > > >> generated
> > > > >>> > > > > > > > >> > > and
> > > > >>> > > > > > > > >> > > > > > epoch
> > > > >>> > > > > > > > >> > > > > > > > to
> > > > >>> > > > > > > > >> > > > > > > > > 0
> > > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > complete. This is
> > for
> > > > >>> > downgrade
> > > > >>> > > > > > > > >> compatibility.
> > > > >>> > > > > > > > >> > > The
> > > > >>> > > > > > > > >> > > > > > tagged
> > > > >>> > > > > > > > >> > > > > > > > > > fields
> > > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety guards for
> > > > retries
> > > > >>> and
> > > > >>> > > > > > failovers.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In prepare phase
> for
> > > > epoch
> > > > >>> > > > overflow
> > > > >>> > > > > > case
> > > > >>> > > > > > > > >> only
> > > > >>> > > > > > > > >> > we
> > > > >>> > > > > > > > >> > > > > store
> > > > >>> > > > > > > > >> > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > next
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID. This is for
> the
> > > case
> > > > >>> where
> > > > >>> > > we
> > > > >>> > > > > > reload
> > > > >>> > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > transaction
> > > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > prepare state.
> Once
> > > the
> > > > >>> > > > transaction
> > > > >>> > > > > is
> > > > >>> > > > > > > > >> > committed,
> > > > >>> > > > > > > > >> > > > we
> > > > >>> > > > > > > > >> > > > > > can
> > > > >>> > > > > > > > >> > > > > > > > use
> > > > >>> > > > > > > > >> > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the client
> > already
> > > is
> > > > >>> > using.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In commit phase,
> we
> > > > store
> > > > >>> the
> > > > >>> > > > > previous
> > > > >>> > > > > > > > >> producer
> > > > >>> > > > > > > > >> > > ID
> > > > >>> > > > > > > > >> > > > in
> > > > >>> > > > > > > > >> > > > > > > case
> > > > >>> > > > > > > > >> > > > > > > > of
> > > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I think it is
> easier
> > > to
> > > > >>> think
> > > > >>> > of
> > > > >>> > > > it
> > > > >>> > > > > as
> > > > >>> > > > > > > > just
> > > > >>> > > > > > > > >> how
> > > > >>> > > > > > > > >> > > we
> > > > >>> > > > > > > > >> > > > > were
> > > > >>> > > > > > > > >> > > > > > > > > storing
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and epoch before,
> > with
> > > > >>> some
> > > > >>> > > extra
> > > > >>> > > > > > > > bookeeping
> > > > >>> > > > > > > > >> > and
> > > > >>> > > > > > > > >> > > > edge
> > > > >>> > > > > > > > >> > > > > > > case
> > > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged fields. We
> > have
> > > > to
> > > > >>> do
> > > > >>> > it
> > > > >>> > > > this
> > > > >>> > > > > > way
> > > > >>> > > > > > > > for
> > > > >>> > > > > > > > >> > > > > > > compatibility
> > > > >>> > > > > > > > >> > > > > > > > > with
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > downgrades.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103. Next producer
> > ID
> > > is
> > > > >>> for
> > > > >>> > > > prepare
> > > > >>> > > > > > > > status
> > > > >>> > > > > > > > >> and
> > > > >>> > > > > > > > >> > > > > > previous
> > > > >>> > > > > > > > >> > > > > > > > > > producer
> > > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after complete.
> The
> > > > >>> reason why
> > > > >>> > > we
> > > > >>> > > > > need
> > > > >>> > > > > > > two
> > > > >>> > > > > > > > >> > > separate
> > > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > backwards
> > > compatibility.
> > > > >>> We
> > > > >>> > need
> > > > >>> > > > to
> > > > >>> > > > > > keep
> > > > >>> > > > > > > > the
> > > > >>> > > > > > > > >> > same
> > > > >>> > > > > > > > >> > > > > > > semantics
> > > > >>> > > > > > > > >> > > > > > > > > for
> > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged field
> in
> > > case
> > > > >>> we
> > > > >>> > > > > downgrade.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104. We set the
> > fields
> > > > as
> > > > >>> we
> > > > >>> > do
> > > > >>> > > in
> > > > >>> > > > > the
> > > > >>> > > > > > > > >> > > > transactional
> > > > >>> > > > > > > > >> > > > > > > state
> > > > >>> > > > > > > > >> > > > > > > > > (as
> > > > >>> > > > > > > > >> > > > > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do this for
> > > > compatibility
> > > > >>> --
> > > > >>> > if
> > > > >>> > > we
> > > > >>> > > > > > > > >> downgrade,
> > > > >>> > > > > > > > >> > we
> > > > >>> > > > > > > > >> > > > will
> > > > >>> > > > > > > > >> > > > > > > only
> > > > >>> > > > > > > > >> > > > > > > > > have
> > > > >>> > > > > > > > >> > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged fields)
> > It
> > > > >>> will be
> > > > >>> > > the
> > > > >>> > > > > old
> > > > >>> > > > > > > > >> producer
> > > > >>> > > > > > > > >> > ID
> > > > >>> > > > > > > > >> > > > and
> > > > >>> > > > > > > > >> > > > > > max
> > > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope this helps.
> Let
> > > me
> > > > >>> know
> > > > >>> > if
> > > > >>> > > > you
> > > > >>> > > > > > have
> > > > >>> > > > > > > > >> > further
> > > > >>> > > > > > > > >> > > > > > > questions.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Justine
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On Wed, Dec 20,
> 2023
> > > at
> > > > >>> > 3:33 PM
> > > > >>> > > > Jun
> > > > >>> > > > > > Rao
> > > > >>> > > > > > > > >> > > > > > > > > > <jun@confluent.io.invalid
> > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi, Justine,
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It seems that
> you
> > > have
> > > > >>> made
> > > > >>> > > some
> > > > >>> > > > > > > changes
> > > > >>> > > > > > > > >> to
> > > > >>> > > > > > > > >> > > > KIP-890
> > > > >>> > > > > > > > >> > > > > > > since
> > > > >>> > > > > > > > >> > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > particular, we
> are
> > > > >>> changing
> > > > >>> > > the
> > > > >>> > > > > > format
> > > > >>> > > > > > > > of
> > > > >>> > > > > > > > >> > > > > > > > > > TransactionLogValue.
> > > > >>> > > > > > > > >> > > > > > > > > > > A
> > > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > comments related
> > to
> > > > >>> that.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100. Just to be
> > > clear.
> > > > >>> The
> > > > >>> > > > > overflow
> > > > >>> > > > > > > case
> > > > >>> > > > > > > > >> > (i.e.
> > > > >>> > > > > > > > >> > > > > when a
> > > > >>> > > > > > > > >> > > > > > > new
> > > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > generated) is
> when
> > > the
> > > > >>> > current
> > > > >>> > > > > epoch
> > > > >>> > > > > > > > >> equals
> > > > >>> > > > > > > > >> > to
> > > > >>> > > > > > > > >> > > > max
> > > > >>> > > > > > > > >> > > > > -
> > > > >>> > > > > > > > >> > > > > > 1
> > > > >>> > > > > > > > >> > > > > > > > and
> > > > >>> > > > > > > > >> > > > > > > > > > not
> > > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101. For the
> "not
> > > > epoch
> > > > >>> > > > overflow"
> > > > >>> > > > > > > case,
> > > > >>> > > > > > > > we
> > > > >>> > > > > > > > >> > > write
> > > > >>> > > > > > > > >> > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > previous
> > > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > tagged field in
> > the
> > > > >>> complete
> > > > >>> > > > > phase.
> > > > >>> > > > > > Do
> > > > >>> > > > > > > > we
> > > > >>> > > > > > > > >> > need
> > > > >>> > > > > > > > >> > > to
> > > > >>> > > > > > > > >> > > > > do
> > > > >>> > > > > > > > >> > > > > > > that
> > > > >>> > > > > > > > >> > > > > > > > > > since
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > doesn't change
> in
> > > this
> > > > >>> case?
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102. It seems
> that
> > > the
> > > > >>> > meaning
> > > > >>> > > > for
> > > > >>> > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > ProducerId/ProducerEpoch
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > TransactionLogValue
> > > > >>> changes
> > > > >>> > > > > > depending
> > > > >>> > > > > > > on
> > > > >>> > > > > > > > >> the
> > > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > > TransactionStatus
> > > > is
> > > > >>> > > > ongoing,
> > > > >>> > > > > > they
> > > > >>> > > > > > > > >> > > represent
> > > > >>> > > > > > > > >> > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > current
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the current
> > > > >>> ProducerEpoch.
> > > > >>> > > When
> > > > >>> > > > > the
> > > > >>> > > > > > > > >> > > > > TransactionStatus
> > > > >>> > > > > > > > >> > > > > > > is
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > >>> PrepareCommit/PrepareAbort,
> > > > >>> > > they
> > > > >>> > > > > > > > represent
> > > > >>> > > > > > > > >> > the
> > > > >>> > > > > > > > >> > > > > > current
> > > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> > ProducerEpoch.
> > > > >>> When the
> > > > >>> > > > > > > > >> > TransactionStatus
> > > > >>> > > > > > > > >> > > is
> > > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > further depend
> on
> > > > >>> whether
> > > > >>> > the
> > > > >>> > > > > epoch
> > > > >>> > > > > > > > >> overflows
> > > > >>> > > > > > > > >> > > or
> > > > >>> > > > > > > > >> > > > > not.
> > > > >>> > > > > > > > >> > > > > > > If
> > > > >>> > > > > > > > >> > > > > > > > > > there
> > > > >>> > > > > > > > >> > > > > > > > > > > is
> > > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > overflow, they
> > > > represent
> > > > >>> > the
> > > > >>> > > > > > current
> > > > >>> > > > > > > > >> > > ProducerId
> > > > >>> > > > > > > > >> > > > > and
> > > > >>> > > > > > > > >> > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > next
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ProducerEpoch
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > (max).
> Otherwise,
> > > they
> > > > >>> > > represent
> > > > >>> > > > > the
> > > > >>> > > > > > > > newly
> > > > >>> > > > > > > > >> > > > > generated
> > > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > ProducerEpoch of
> > 0.
> > > Is
> > > > >>> that
> > > > >>> > > > right?
> > > > >>> > > > > > > This
> > > > >>> > > > > > > > >> seems
> > > > >>> > > > > > > > >> > > not
> > > > >>> > > > > > > > >> > > > > > easy
> > > > >>> > > > > > > > >> > > > > > > to
> > > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we provide some
> > > > examples
> > > > >>> > like
> > > > >>> > > > what
> > > > >>> > > > > > > Artem
> > > > >>> > > > > > > > >> has
> > > > >>> > > > > > > > >> > > done
> > > > >>> > > > > > > > >> > > > > in
> > > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > considered a
> > simpler
> > > > >>> design
> > > > >>> > > > where
> > > > >>> > > > > > > > >> > > > > > > > ProducerId/ProducerEpoch
> > > > >>> > > > > > > > >> > > > > > > > > > > always
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > represent
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the same value
> > (e.g.
> > > > >>> for the
> > > > >>> > > > > current
> > > > >>> > > > > > > > >> > > transaction)
> > > > >>> > > > > > > > >> > > > > > > > > independent
> > > > >>> > > > > > > > >> > > > > > > > > > > of
> > > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> TransactionStatus
> > > and
> > > > >>> epoch
> > > > >>> > > > > > overflow?
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103. It's not
> > clear
> > > to
> > > > >>> me
> > > > >>> > why
> > > > >>> > > we
> > > > >>> > > > > > need
> > > > >>> > > > > > > 3
> > > > >>> > > > > > > > >> > fields:
> > > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > PrevProducerId,
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > NextProducerId.
> > > Could
> > > > we
> > > > >>> > just
> > > > >>> > > > have
> > > > >>> > > > > > > > >> ProducerId
> > > > >>> > > > > > > > >> > > and
> > > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104. For
> > > > >>> > > WriteTxnMarkerRequests,
> > > > >>> > > > > if
> > > > >>> > > > > > > the
> > > > >>> > > > > > > > >> > > producer
> > > > >>> > > > > > > > >> > > > > > epoch
> > > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we set the
> > > producerId
> > > > >>> and
> > > > >>> > the
> > > > >>> > > > > > > > >> producerEpoch?
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Thanks,
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > > >
> > > > >>> > > > > > > > >> > > > > > > >
> > > > >>> > > > > > > > >> > > > > > >
> > > > >>> > > > > > > > >> > > > > >
> > > > >>> > > > > > > > >> > > > >
> > > > >>> > > > > > > > >> > > >
> > > > >>> > > > > > > > >> > >
> > > > >>> > > > > > > > >> >
> > > > >>> > > > > > > > >>
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > >
> > > > >>> > > > > > >
> > > > >>> > > > > >
> > > > >>> > > > >
> > > > >>> > > >
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Justine,

Thanks for the reply.

120. If the broker sends TV Y for the finalized version in
ApiVersionResponse, but the client doesn't support Y, how does the broker
know the TV that the client supports?

Jun

On Wed, Jul 10, 2024 at 2:29 PM Justine Olshan <jo...@confluent.io.invalid>
wrote:

> Hey Jun,
>
> No worries. Work on this KIP has been blocked for a bit anyways -- catching
> up and rereading what I wrote :)
>
> 120. ClientTransactionProtocolVersion is the transaction version as defined
> by the highest transaction version (feature version value) supported by the
> client and the server. This works by the broker sending an
> ApiVersionsRequest to the client with the finalized version. Assuming
> kip-890 part 2 is enabled by transaction version Y, if this request
> contains finalized version Y and the client has the logic to set this
> field, it will set Y. If the server has Y - 1 (kip 890 part 2 not enable)
> the client will send Y - 1, even though the client has the ability to
> support kip-890 part 2.
>
> 121. You are correct that this is not needed. However, currently that field
> is already being set in memory -- just not written to disk. I think it is
> ok to write it to disk though. Let me know if you think otherwise.
>
> Justine
>
> On Wed, Jul 10, 2024 at 2:16 PM Jun Rao <ju...@confluent.io.invalid> wrote:
>
> > Hi, Justine,
> >
> > Thanks for the update and sorry for the late reply.
> >
> > 120. I am wondering what value is used for
> > ClientTransactionProtocolVersion. Is it the version of the EndTxnRequest?
> >
> > 121. Earlier, you made the change to set lastProducerId in PREPARE to
> > indicate that the marker is written for the new client. With the new
> > ClientTransactionProtocolVersion field, it seems this is no longer
> > necessary.
> >
> > Jun
> >
> > On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> > <jo...@confluent.io.invalid>
> > wrote:
> >
> > > Hi there -- another update!
> > >
> > > When looking into the implementation for the safe epoch bumps I
> realized
> > > that we are already populating previousProducerID in memory as part of
> > > KIP-360.
> > > If we are to start using flexible fields, it is better to always use
> this
> > > information and have an explicit (tagged) field to indicate whether the
> > > client supports KIP-890 part 2.
> > >
> > > I've included the extra field and how it is set in the KIP. I've also
> > > updated the KIP to explain that we will be setting the tagged fields
> when
> > > they are available for all transitions.
> > >
> > > Finally, I added clearer text about the transaction protocol versions
> > > included with this KIP. 1 for flexible transaction state records and 2
> > for
> > > KIP-890 part 2 enablement.
> > >
> > > Justine
> > >
> > > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <jo...@confluent.io>
> > > wrote:
> > >
> > > > Hey there -- small update to the KIP,
> > > >
> > > > The KIP mentioned introducing ABORTABLE_ERROR and bumping
> > TxnOffsetCommit
> > > > and Produce requests. I've changed the name in the KIP to
> > > > ABORTABLE_TRANSACTION and the corresponding exception
> > > > AbortableTransactionException to match the pattern we had for other
> > > errors.
> > > > I also mentioned bumping all 6 transactional APIs so we can future
> > > > proof/support the error on the client going forward. If a future
> change
> > > > wants to have an error scenario that requires us to abort the
> > > transaction,
> > > > we can rely on the 3.8+ clients to support it. We ran into issues
> > finding
> > > > good/generic error codes that older clients could support while
> working
> > > on
> > > > this KIP, so this should help in the future.
> > > >
> > > > The features discussion is still ongoing in KIP-1022. Will update
> again
> > > > here when that concludes.
> > > >
> > > > Justine
> > > >
> > > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <jo...@confluent.io>
> > > > wrote:
> > > >
> > > >> I don't think AddPartitions is a good example since we currenly
> don't
> > > >> gate the version on TV or MV. (We only set a different flag
> depending
> > on
> > > >> the TV)
> > > >>
> > > >> Even if we did want to gate it on TV, I think the idea is to move
> away
> > > >> from MV gating inter broker protocols. Ideally we can get to a state
> > > where
> > > >> MV is just used for metadata changes.
> > > >>
> > > >> I think some of this discussion might fit more with the feature
> > version
> > > >> KIP, so I can try to open that up soon. Until we settle that, some
> of
> > > the
> > > >> work in KIP-890 is blocked.
> > > >>
> > > >> Justine
> > > >>
> > > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao <ju...@confluent.io.invalid>
> > > wrote:
> > > >>
> > > >>> Hi, Justine,
> > > >>>
> > > >>> Thanks for the reply.
> > > >>>
> > > >>> Since AddPartitions is an inter broker request, will its version be
> > > gated
> > > >>> only by TV or other features like MV too? For example, if we need
> to
> > > >>> change
> > > >>> the protocol for AddPartitions for reasons other than txn
> > verification
> > > in
> > > >>> the future, will the new version be gated by a new MV? If so, does
> > > >>> downgrading a TV imply potential downgrade of MV too?
> > > >>>
> > > >>> Jun
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > > >>> <jo...@confluent.io.invalid>
> > > >>> wrote:
> > > >>>
> > > >>> > One TV gates the flexible feature version (no rpcs involved, only
> > the
> > > >>> > transactional records that should only be gated by TV)
> > > >>> > Another TV gates the ability to turn on kip-890 part 2. This
> would
> > > >>> gate the
> > > >>> > version of Produce and EndTxn (likely only used by transactions),
> > and
> > > >>> > specifies a flag in AddPartitionsToTxn though the version is
> > already
> > > >>> used
> > > >>> > without TV.
> > > >>> >
> > > >>> > I think the only concern is the Produce request and we could
> > consider
> > > >>> work
> > > >>> > arounds similar to the AddPartitionsToTxn call.
> > > >>> >
> > > >>> > Justine
> > > >>> >
> > > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao <jun@confluent.io.invalid
> >
> > > >>> wrote:
> > > >>> >
> > > >>> > > Hi, Justine,
> > > >>> > >
> > > >>> > > Which PRC/record protocols will TV guard? Going forward, will
> > those
> > > >>> > > PRC/record protocols only be guarded by TV and not by other
> > > features
> > > >>> like
> > > >>> > > MV?
> > > >>> > >
> > > >>> > > Thanks,
> > > >>> > >
> > > >>> > > Jun
> > > >>> > >
> > > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > > >>> > <jolshan@confluent.io.invalid
> > > >>> > > >
> > > >>> > > wrote:
> > > >>> > >
> > > >>> > > > Hi Jun,
> > > >>> > > >
> > > >>> > > > Sorry I think I misunderstood your question or answered
> > > >>> incorrectly.
> > > >>> > The
> > > >>> > > TV
> > > >>> > > > version should ideally be fully independent from MV.
> > > >>> > > > At least for the changes I proposed, TV should not affect MV
> > and
> > > MV
> > > >>> > > should
> > > >>> > > > not affect TV/
> > > >>> > > >
> > > >>> > > > I think if we downgrade TV, only that feature should
> downgrade.
> > > >>> > Likewise
> > > >>> > > > the same with MV. The finalizedFeatures should just reflect
> the
> > > >>> feature
> > > >>> > > > downgrade we made.
> > > >>> > > >
> > > >>> > > > I also plan to write a new KIP for managing the disk format
> and
> > > >>> upgrade
> > > >>> > > > tool as we will need new flags to support these features.
> That
> > > >>> should
> > > >>> > > help
> > > >>> > > > clarify some things.
> > > >>> > > >
> > > >>> > > > Justine
> > > >>> > > >
> > > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> > <jun@confluent.io.invalid
> > > >
> > > >>> > > wrote:
> > > >>> > > >
> > > >>> > > > > Hi, Justine,
> > > >>> > > > >
> > > >>> > > > > Thanks for the reply.
> > > >>> > > > >
> > > >>> > > > > So, if we downgrade TV, we could implicitly downgrade
> another
> > > >>> feature
> > > >>> > > > (say
> > > >>> > > > > MV) that has dependency (e.g. RPC). What would we return
> for
> > > >>> > > > > FinalizedFeatures for MV in ApiVersionsResponse in that
> case?
> > > >>> > > > >
> > > >>> > > > > Thanks,
> > > >>> > > > >
> > > >>> > > > > Jun
> > > >>> > > > >
> > > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > > >>> > > > <jolshan@confluent.io.invalid
> > > >>> > > > > >
> > > >>> > > > > wrote:
> > > >>> > > > >
> > > >>> > > > > > Hey Jun,
> > > >>> > > > > >
> > > >>> > > > > > Yes, the idea is that if we downgrade TV (transaction
> > > version)
> > > >>> we
> > > >>> > > will
> > > >>> > > > > stop
> > > >>> > > > > > using the add partitions to txn optimization and stop
> > writing
> > > >>> the
> > > >>> > > > > flexible
> > > >>> > > > > > feature version of the log.
> > > >>> > > > > > In the compatibility section I included some explanations
> > on
> > > >>> how
> > > >>> > this
> > > >>> > > > is
> > > >>> > > > > > done.
> > > >>> > > > > >
> > > >>> > > > > > Thanks,
> > > >>> > > > > > Justine
> > > >>> > > > > >
> > > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > > >>> <ju...@confluent.io.invalid>
> > > >>> > > > > wrote:
> > > >>> > > > > >
> > > >>> > > > > > > Hi, Justine,
> > > >>> > > > > > >
> > > >>> > > > > > > Thanks for the update.
> > > >>> > > > > > >
> > > >>> > > > > > > If we ever downgrade the transaction feature, any
> feature
> > > >>> > depending
> > > >>> > > > on
> > > >>> > > > > > > changes on top of those RPC/record
> > > >>> > > > > > > (AddPartitionsToTxnRequest/TransactionLogValue) changes
> > > made
> > > >>> in
> > > >>> > > > KIP-890
> > > >>> > > > > > > will be automatically downgraded too?
> > > >>> > > > > > >
> > > >>> > > > > > > Jun
> > > >>> > > > > > >
> > > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine Olshan
> > > >>> > > > > > > <jo...@confluent.io.invalid>
> > > >>> > > > > > > wrote:
> > > >>> > > > > > >
> > > >>> > > > > > > > Hey Jun,
> > > >>> > > > > > > >
> > > >>> > > > > > > > I wanted to get back to you about your questions
> about
> > > >>> MV/IBP.
> > > >>> > > > > > > >
> > > >>> > > > > > > > Looking at the options, I think it makes the most
> sense
> > > to
> > > >>> > > create a
> > > >>> > > > > > > > separate feature for transactions and use that to
> > version
> > > >>> gate
> > > >>> > > the
> > > >>> > > > > > > features
> > > >>> > > > > > > > we need to version gate (flexible transactional state
> > > >>> records
> > > >>> > and
> > > >>> > > > > using
> > > >>> > > > > > > the
> > > >>> > > > > > > > new protocol)
> > > >>> > > > > > > > I've updated the KIP to include this change.
> Hopefully
> > > >>> that's
> > > >>> > > > > > everything
> > > >>> > > > > > > we
> > > >>> > > > > > > > need for this KIP :)
> > > >>> > > > > > > >
> > > >>> > > > > > > > Justine
> > > >>> > > > > > > >
> > > >>> > > > > > > >
> > > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine Olshan <
> > > >>> > > > jolshan@confluent.io
> > > >>> > > > > >
> > > >>> > > > > > > > wrote:
> > > >>> > > > > > > >
> > > >>> > > > > > > > > Thanks Jun,
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > I will update the KIP with the prev field for
> prepare
> > > as
> > > >>> > well.
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > PREPARE
> > > >>> > > > > > > > > producerId: x
> > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > >>> > > > > > > > > nextProducerId (tagged field): empty or z if y will
> > > >>> overflow
> > > >>> > > > > > > > > producerEpoch: y + 1
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > COMPLETE
> > > >>> > > > > > > > > producerId: x or z if y overflowed
> > > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > > >>> > > > > > > > > nextProducerId (tagged field): empty
> > > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we overflowed
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > Thanks again,
> > > >>> > > > > > > > > Justine
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun Rao
> > > >>> > > <jun@confluent.io.invalid
> > > >>> > > > >
> > > >>> > > > > > > > wrote:
> > > >>> > > > > > > > >
> > > >>> > > > > > > > >> Hi, Justine,
> > > >>> > > > > > > > >>
> > > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > > >>> > > > > > > > >> (1) My point was that the coordinator could fail
> > right
> > > >>> after
> > > >>> > > > > writing
> > > >>> > > > > > > the
> > > >>> > > > > > > > >> prepare marker. When the new txn coordinator
> > generates
> > > >>> the
> > > >>> > > > > complete
> > > >>> > > > > > > > marker
> > > >>> > > > > > > > >> after the failover, it needs some field from the
> > > prepare
> > > >>> > > marker
> > > >>> > > > to
> > > >>> > > > > > > > >> determine whether it's written by the new client.
> > > >>> > > > > > > > >>
> > > >>> > > > > > > > >> (2) The changing of the behavior sounds good to
> me.
> > We
> > > >>> only
> > > >>> > > want
> > > >>> > > > > to
> > > >>> > > > > > > > return
> > > >>> > > > > > > > >> success if the prepare state is written by the new
> > > >>> client.
> > > >>> > So,
> > > >>> > > > in
> > > >>> > > > > > the
> > > >>> > > > > > > > >> non-overflow case, it seems that we also need sth
> in
> > > the
> > > >>> > > prepare
> > > >>> > > > > > > marker
> > > >>> > > > > > > > to
> > > >>> > > > > > > > >> tell us whether it's written by the new client.
> > > >>> > > > > > > > >>
> > > >>> > > > > > > > >> 112. Thanks for the explanation. That sounds good
> to
> > > me.
> > > >>> > > > > > > > >>
> > > >>> > > > > > > > >> Jun
> > > >>> > > > > > > > >>
> > > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM Justine Olshan
> > > >>> > > > > > > > >> <jo...@confluent.io.invalid> wrote:
> > > >>> > > > > > > > >>
> > > >>> > > > > > > > >> > 101.3 I realized that I actually have two
> > questions.
> > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need to write
> > the
> > > >>> > > previous
> > > >>> > > > > > > produce
> > > >>> > > > > > > > Id
> > > >>> > > > > > > > >> > tagged field in the end maker so that we know if
> > the
> > > >>> > marker
> > > >>> > > is
> > > >>> > > > > > from
> > > >>> > > > > > > > the
> > > >>> > > > > > > > >> new
> > > >>> > > > > > > > >> > client. Since the end maker is derived from the
> > > >>> prepare
> > > >>> > > > marker,
> > > >>> > > > > > > should
> > > >>> > > > > > > > >> we
> > > >>> > > > > > > > >> > write the previous produce Id in the prepare
> > marker
> > > >>> field
> > > >>> > > too?
> > > >>> > > > > > > > >> Otherwise,
> > > >>> > > > > > > > >> > we will lose this information when deriving the
> > end
> > > >>> > marker.
> > > >>> > > > > > > > >> >
> > > >>> > > > > > > > >> > The "previous" producer ID is in the normal
> > producer
> > > >>> ID
> > > >>> > > field.
> > > >>> > > > > So
> > > >>> > > > > > > yes,
> > > >>> > > > > > > > >> we
> > > >>> > > > > > > > >> > need it in prepare and that was always the plan.
> > > >>> > > > > > > > >> >
> > > >>> > > > > > > > >> > Maybe it is a bit unclear so I will enumerate
> the
> > > >>> fields
> > > >>> > and
> > > >>> > > > add
> > > >>> > > > > > > them
> > > >>> > > > > > > > to
> > > >>> > > > > > > > >> > the KIP if that helps.
> > > >>> > > > > > > > >> > Say we have producer ID x and epoch y. When we
> > > >>> overflow
> > > >>> > > epoch
> > > >>> > > > y
> > > >>> > > > > we
> > > >>> > > > > > > get
> > > >>> > > > > > > > >> > producer ID Z.
> > > >>> > > > > > > > >> >
> > > >>> > > > > > > > >> > PREPARE
> > > >>> > > > > > > > >> > producerId: x
> > > >>> > > > > > > > >> > previous/lastProducerId (tagged field): empty
> > > >>> > > > > > > > >> > nextProducerId (tagged field): empty or z if y
> > will
> > > >>> > overflow
> > > >>> > > > > > > > >> > producerEpoch: y + 1
> > > >>> > > > > > > > >> >
> > > >>> > > > > > > > >> > COMPLETE
> > > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > > >>> > > > > > > > >> > previous/lastProducerId (tagged field): x
> > > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we overflowed
> > > >>> > > > > > > > >> >
> > > >>> > > > > > > > >> > (2) In the prepare phase, if we retry and see
> > epoch
> > > -
> > > >>> 1 +
> > > >>> > ID
> > > >>> > > > in
> > > >>> > > > > > last
> > > >>> > > > > > > > >> seen
> > > >>> > > > > > > > >> > fields and are issuing the same command (ie
> commit
> > > not
> > > >>> > > abort),
> > > >>> > > > > we
> > > >>> > > > > > > > return
> > > >>> > > > > > > > >> > success. The logic before KIP-890 seems to
> return
> > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > >>> > > > > > > > >> > in this case. Are we intentionally making this
> > > change?
> > > >>> > > > > > > > >> >
> > > >>> > > > > > > > >> > Hmm -- we would fence the producer if the epoch
> is
> > > >>> bumped
> > > >>> > > and
> > > >>> > > > we
> > > >>> > > > > > > get a
> > > >>> > > > > > > > >> > lower epoch. Yes -- we are intentionally adding
> > this
> > > >>> to
> > > >>> > > > prevent
> > > >>> > > > > > > > fencing.
> > > >>> > > > > > > > >> >
> > > >>> > > > > > > > >> >
> > > >>> > > > > > > > >> > 112. We already merged the code that adds the
> > > >>> VerifyOnly
> > > >>> > > field
> > > >>> > > > > in
> > > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which is an inter
> > broker
> > > >>> > request.
> > > >>> > > > It
> > > >>> > > > > > > seems
> > > >>> > > > > > > > >> that
> > > >>> > > > > > > > >> > we didn't bump up the IBP for that. Do you know
> > why?
> > > >>> > > > > > > > >> >
> > > >>> > > > > > > > >> > We no longer need IBP for all interbroker
> requests
> > > as
> > > >>> > > > > ApiVersions
> > > >>> > > > > > > > should
> > > >>> > > > > > > > >> > correctly gate versioning.
> > > >>> > > > > > > > >> > We also handle unsupported version errors
> > correctly
> > > >>> if we
> > > >>> > > > > receive
> > > >>> > > > > > > them
> > > >>> > > > > > > > >> in
> > > >>> > > > > > > > >> > edge cases like upgrades/downgrades.
> > > >>> > > > > > > > >> >
> > > >>> > > > > > > > >> > Justine
> > > >>> > > > > > > > >> >
> > > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM Jun Rao
> > > >>> > > > > <jun@confluent.io.invalid
> > > >>> > > > > > >
> > > >>> > > > > > > > >> wrote:
> > > >>> > > > > > > > >> >
> > > >>> > > > > > > > >> > > Hi, Justine,
> > > >>> > > > > > > > >> > >
> > > >>> > > > > > > > >> > > Thanks for the reply.
> > > >>> > > > > > > > >> > >
> > > >>> > > > > > > > >> > > 101.3 I realized that I actually have two
> > > questions.
> > > >>> > > > > > > > >> > > (1) In the non-overflow case, we need to write
> > the
> > > >>> > > previous
> > > >>> > > > > > > produce
> > > >>> > > > > > > > Id
> > > >>> > > > > > > > >> > > tagged field in the end maker so that we know
> if
> > > the
> > > >>> > > marker
> > > >>> > > > is
> > > >>> > > > > > > from
> > > >>> > > > > > > > >> the
> > > >>> > > > > > > > >> > new
> > > >>> > > > > > > > >> > > client. Since the end maker is derived from
> the
> > > >>> prepare
> > > >>> > > > > marker,
> > > >>> > > > > > > > >> should we
> > > >>> > > > > > > > >> > > write the previous produce Id in the prepare
> > > marker
> > > >>> > field
> > > >>> > > > too?
> > > >>> > > > > > > > >> Otherwise,
> > > >>> > > > > > > > >> > > we will lose this information when deriving
> the
> > > end
> > > >>> > > marker.
> > > >>> > > > > > > > >> > > (2) In the prepare phase, if we retry and see
> > > epoch
> > > >>> - 1
> > > >>> > +
> > > >>> > > ID
> > > >>> > > > > in
> > > >>> > > > > > > last
> > > >>> > > > > > > > >> seen
> > > >>> > > > > > > > >> > > fields and are issuing the same command (ie
> > commit
> > > >>> not
> > > >>> > > > abort),
> > > >>> > > > > > we
> > > >>> > > > > > > > >> return
> > > >>> > > > > > > > >> > > success. The logic before KIP-890 seems to
> > return
> > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > >>> > > > > > > > >> > > in this case. Are we intentionally making this
> > > >>> change?
> > > >>> > > > > > > > >> > >
> > > >>> > > > > > > > >> > > 112. We already merged the code that adds the
> > > >>> VerifyOnly
> > > >>> > > > field
> > > >>> > > > > > in
> > > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which is an inter
> > > broker
> > > >>> > > request.
> > > >>> > > > > It
> > > >>> > > > > > > > seems
> > > >>> > > > > > > > >> > that
> > > >>> > > > > > > > >> > > we didn't bump up the IBP for that. Do you
> know
> > > why?
> > > >>> > > > > > > > >> > >
> > > >>> > > > > > > > >> > > Jun
> > > >>> > > > > > > > >> > >
> > > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM Justine Olshan
> > > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > > >>> > > > > > > > >> > > wrote:
> > > >>> > > > > > > > >> > >
> > > >>> > > > > > > > >> > > > Hi Jun,
> > > >>> > > > > > > > >> > > >
> > > >>> > > > > > > > >> > > > 101.3 I can change "last seen" to "current
> > > >>> producer id
> > > >>> > > and
> > > >>> > > > > > > epoch"
> > > >>> > > > > > > > if
> > > >>> > > > > > > > >> > that
> > > >>> > > > > > > > >> > > > was the part that was confusing
> > > >>> > > > > > > > >> > > > 110 I can mention this
> > > >>> > > > > > > > >> > > > 111 I can do that
> > > >>> > > > > > > > >> > > > 112 We still need it. But I am still
> > finalizing
> > > >>> the
> > > >>> > > > design.
> > > >>> > > > > I
> > > >>> > > > > > > will
> > > >>> > > > > > > > >> > update
> > > >>> > > > > > > > >> > > > the KIP once I get the information
> finalized.
> > > >>> Sorry
> > > >>> > for
> > > >>> > > > the
> > > >>> > > > > > > > delays.
> > > >>> > > > > > > > >> > > >
> > > >>> > > > > > > > >> > > > Justine
> > > >>> > > > > > > > >> > > >
> > > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at 10:50 AM Jun Rao
> > > >>> > > > > > > <jun@confluent.io.invalid
> > > >>> > > > > > > > >
> > > >>> > > > > > > > >> > > wrote:
> > > >>> > > > > > > > >> > > >
> > > >>> > > > > > > > >> > > > > Hi, Justine,
> > > >>> > > > > > > > >> > > > >
> > > >>> > > > > > > > >> > > > > Thanks for the reply.
> > > >>> > > > > > > > >> > > > >
> > > >>> > > > > > > > >> > > > > 101.3 In the non-overflow case, the
> previous
> > > ID
> > > >>> is
> > > >>> > the
> > > >>> > > > > same
> > > >>> > > > > > as
> > > >>> > > > > > > > the
> > > >>> > > > > > > > >> > > > produce
> > > >>> > > > > > > > >> > > > > ID for the complete marker too, but we set
> > the
> > > >>> > > previous
> > > >>> > > > ID
> > > >>> > > > > > in
> > > >>> > > > > > > > the
> > > >>> > > > > > > > >> > > > complete
> > > >>> > > > > > > > >> > > > > marker. Earlier you mentioned that this is
> > to
> > > >>> know
> > > >>> > > that
> > > >>> > > > > the
> > > >>> > > > > > > > >> marker is
> > > >>> > > > > > > > >> > > > > written by the new client so that we could
> > > >>> return
> > > >>> > > > success
> > > >>> > > > > on
> > > >>> > > > > > > > >> retried
> > > >>> > > > > > > > >> > > > > endMarker requests. I was trying to
> > understand
> > > >>> why
> > > >>> > > this
> > > >>> > > > is
> > > >>> > > > > > not
> > > >>> > > > > > > > >> needed
> > > >>> > > > > > > > >> > > for
> > > >>> > > > > > > > >> > > > > the prepare marker since retry can happen
> in
> > > the
> > > >>> > > prepare
> > > >>> > > > > > state
> > > >>> > > > > > > > >> too.
> > > >>> > > > > > > > >> > Is
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > reason that in the prepare state, we
> return
> > > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > > >>> > > > > > > > >> > > > instead
> > > >>> > > > > > > > >> > > > > of success on retried endMaker requests?
> If
> > > so,
> > > >>> > should
> > > >>> > > > we
> > > >>> > > > > > > change
> > > >>> > > > > > > > >> "If
> > > >>> > > > > > > > >> > we
> > > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID in last seen
> > > >>> fields and
> > > >>> > > are
> > > >>> > > > > > > issuing
> > > >>> > > > > > > > >> the
> > > >>> > > > > > > > >> > > same
> > > >>> > > > > > > > >> > > > > command (ie commit not abort) we can
> return
> > > >>> (with
> > > >>> > the
> > > >>> > > > new
> > > >>> > > > > > > > epoch)"
> > > >>> > > > > > > > >> > > > > accordingly?
> > > >>> > > > > > > > >> > > > >
> > > >>> > > > > > > > >> > > > > 110. Yes, without this KIP, a delayed
> > endMaker
> > > >>> > request
> > > >>> > > > > > carries
> > > >>> > > > > > > > the
> > > >>> > > > > > > > >> > same
> > > >>> > > > > > > > >> > > > > epoch and won't be fenced. This can
> > > >>> commit/abort a
> > > >>> > > > future
> > > >>> > > > > > > > >> transaction
> > > >>> > > > > > > > >> > > > > unexpectedly. I am not sure if we have
> seen
> > > >>> this in
> > > >>> > > > > practice
> > > >>> > > > > > > > >> though.
> > > >>> > > > > > > > >> > > > >
> > > >>> > > > > > > > >> > > > > 111. Sounds good. It would be useful to
> make
> > > it
> > > >>> > clear
> > > >>> > > > that
> > > >>> > > > > > we
> > > >>> > > > > > > > can
> > > >>> > > > > > > > >> now
> > > >>> > > > > > > > >> > > > > populate the lastSeen field from the log
> > > >>> reliably.
> > > >>> > > > > > > > >> > > > >
> > > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > > >>> > AddPartitionsToTxnRequest
> > > >>> > > > > since
> > > >>> > > > > > > > it's
> > > >>> > > > > > > > >> > > called
> > > >>> > > > > > > > >> > > > > across brokers and we are changing its
> > schema.
> > > >>> Are
> > > >>> > you
> > > >>> > > > > > saying
> > > >>> > > > > > > we
> > > >>> > > > > > > > >> > don't
> > > >>> > > > > > > > >> > > > need
> > > >>> > > > > > > > >> > > > > it any more? I thought that we already
> > > >>> implemented
> > > >>> > the
> > > >>> > > > > > server
> > > >>> > > > > > > > side
> > > >>> > > > > > > > >> > > > > verification logic based on
> > > >>> > AddPartitionsToTxnRequest
> > > >>> > > > > across
> > > >>> > > > > > > > >> brokers.
> > > >>> > > > > > > > >> > > > >
> > > >>> > > > > > > > >> > > > > Jun
> > > >>> > > > > > > > >> > > > >
> > > >>> > > > > > > > >> > > > >
> > > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at 5:05 PM Justine
> > Olshan
> > > >>> > > > > > > > >> > > > > <jo...@confluent.io.invalid>
> > > >>> > > > > > > > >> > > > > wrote:
> > > >>> > > > > > > > >> > > > >
> > > >>> > > > > > > > >> > > > > > Hey Jun,
> > > >>> > > > > > > > >> > > > > >
> > > >>> > > > > > > > >> > > > > > 101.3 We don't set the previous ID in
> the
> > > >>> Prepare
> > > >>> > > > field
> > > >>> > > > > > > since
> > > >>> > > > > > > > we
> > > >>> > > > > > > > >> > > don't
> > > >>> > > > > > > > >> > > > > need
> > > >>> > > > > > > > >> > > > > > it. It is the same producer ID as the
> main
> > > >>> > producer
> > > >>> > > ID
> > > >>> > > > > > > field.
> > > >>> > > > > > > > >> > > > > >
> > > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to reread your
> > > message
> > > >>> > about
> > > >>> > > > > > delayed
> > > >>> > > > > > > > >> > markers.
> > > >>> > > > > > > > >> > > > If
> > > >>> > > > > > > > >> > > > > we
> > > >>> > > > > > > > >> > > > > > receive a delayed endTxn marker after
> the
> > > >>> > > transaction
> > > >>> > > > is
> > > >>> > > > > > > > already
> > > >>> > > > > > > > >> > > > > complete?
> > > >>> > > > > > > > >> > > > > > So we will commit the next transaction
> > early
> > > >>> > without
> > > >>> > > > the
> > > >>> > > > > > > fixes
> > > >>> > > > > > > > >> in
> > > >>> > > > > > > > >> > > part
> > > >>> > > > > > > > >> > > > 2?
> > > >>> > > > > > > > >> > > > > >
> > > >>> > > > > > > > >> > > > > > 111 Yes -- this terminology was used in
> a
> > > >>> previous
> > > >>> > > KIP
> > > >>> > > > > and
> > > >>> > > > > > > > never
> > > >>> > > > > > > > >> > > > > > implemented it in the log -- only in
> > memory
> > > >>> > > > > > > > >> > > > > >
> > > >>> > > > > > > > >> > > > > > 112 Hmm -- which interbroker protocol
> are
> > > you
> > > >>> > > > referring
> > > >>> > > > > > to?
> > > >>> > > > > > > I
> > > >>> > > > > > > > am
> > > >>> > > > > > > > >> > > > working
> > > >>> > > > > > > > >> > > > > on
> > > >>> > > > > > > > >> > > > > > the design for the work to remove the
> > extra
> > > >>> add
> > > >>> > > > > partitions
> > > >>> > > > > > > > call
> > > >>> > > > > > > > >> > and I
> > > >>> > > > > > > > >> > > > > right
> > > >>> > > > > > > > >> > > > > > now the design bumps MV. I have yet to
> > > update
> > > >>> that
> > > >>> > > > > section
> > > >>> > > > > > > as
> > > >>> > > > > > > > I
> > > >>> > > > > > > > >> > > > finalize
> > > >>> > > > > > > > >> > > > > > the design so please stay tuned. Was
> there
> > > >>> > anything
> > > >>> > > > else
> > > >>> > > > > > you
> > > >>> > > > > > > > >> > thought
> > > >>> > > > > > > > >> > > > > needed
> > > >>> > > > > > > > >> > > > > > MV bump?
> > > >>> > > > > > > > >> > > > > >
> > > >>> > > > > > > > >> > > > > > Justine
> > > >>> > > > > > > > >> > > > > >
> > > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at 3:07 PM Jun Rao
> > > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > > >>> > > > > > > > >> > > > > wrote:
> > > >>> > > > > > > > >> > > > > >
> > > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > > >>> > > > > > > > >> > > > > > >
> > > >>> > > > > > > > >> > > > > > > I don't see this create any issue. It
> > just
> > > >>> makes
> > > >>> > > it
> > > >>> > > > a
> > > >>> > > > > > bit
> > > >>> > > > > > > > >> hard to
> > > >>> > > > > > > > >> > > > > explain
> > > >>> > > > > > > > >> > > > > > > what this non-tagged produce id field
> > > >>> means. We
> > > >>> > > are
> > > >>> > > > > > > > >> essentially
> > > >>> > > > > > > > >> > > > trying
> > > >>> > > > > > > > >> > > > > to
> > > >>> > > > > > > > >> > > > > > > combine two actions (completing a txn
> > and
> > > >>> init a
> > > >>> > > new
> > > >>> > > > > > > produce
> > > >>> > > > > > > > >> Id)
> > > >>> > > > > > > > >> > > in a
> > > >>> > > > > > > > >> > > > > > > single record. But, this may be fine
> > too.
> > > >>> > > > > > > > >> > > > > > >
> > > >>> > > > > > > > >> > > > > > > A few other follow up comments.
> > > >>> > > > > > > > >> > > > > > >
> > > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason that we only
> > set
> > > >>> the
> > > >>> > > > previous
> > > >>> > > > > > > > >> produce id
> > > >>> > > > > > > > >> > > > > tagged
> > > >>> > > > > > > > >> > > > > > > field in the complete marker, but not
> in
> > > the
> > > >>> > > prepare
> > > >>> > > > > > > marker,
> > > >>> > > > > > > > >> is
> > > >>> > > > > > > > >> > > that
> > > >>> > > > > > > > >> > > > in
> > > >>> > > > > > > > >> > > > > > the
> > > >>> > > > > > > > >> > > > > > > prepare state, we always return
> > > >>> > > > > CONCURRENT_TRANSACTIONS
> > > >>> > > > > > on
> > > >>> > > > > > > > >> > retried
> > > >>> > > > > > > > >> > > > > > endMaker
> > > >>> > > > > > > > >> > > > > > > requests?
> > > >>> > > > > > > > >> > > > > > >
> > > >>> > > > > > > > >> > > > > > > 110. "I believe your second point is
> > > >>> mentioned
> > > >>> > in
> > > >>> > > > the
> > > >>> > > > > > > KIP. I
> > > >>> > > > > > > > >> can
> > > >>> > > > > > > > >> > > add
> > > >>> > > > > > > > >> > > > > more
> > > >>> > > > > > > > >> > > > > > > text on
> > > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > > >>> > > > > > > > >> > > > > > > > The delayed message case can also
> > > violate
> > > >>> EOS
> > > >>> > if
> > > >>> > > > the
> > > >>> > > > > > > > delayed
> > > >>> > > > > > > > >> > > > message
> > > >>> > > > > > > > >> > > > > > > comes in after the next
> > addPartitionsToTxn
> > > >>> > request
> > > >>> > > > > comes
> > > >>> > > > > > > in.
> > > >>> > > > > > > > >> > > > > Effectively
> > > >>> > > > > > > > >> > > > > > we
> > > >>> > > > > > > > >> > > > > > > may see a message from a previous
> > > (aborted)
> > > >>> > > > > transaction
> > > >>> > > > > > > > become
> > > >>> > > > > > > > >> > part
> > > >>> > > > > > > > >> > > > of
> > > >>> > > > > > > > >> > > > > > the
> > > >>> > > > > > > > >> > > > > > > next transaction."
> > > >>> > > > > > > > >> > > > > > >
> > > >>> > > > > > > > >> > > > > > > The above is the case when a delayed
> > > >>> message is
> > > >>> > > > > appended
> > > >>> > > > > > > to
> > > >>> > > > > > > > >> the
> > > >>> > > > > > > > >> > > data
> > > >>> > > > > > > > >> > > > > > > partition. What I mentioned is a
> > slightly
> > > >>> > > different
> > > >>> > > > > case
> > > >>> > > > > > > > when
> > > >>> > > > > > > > >> a
> > > >>> > > > > > > > >> > > > delayed
> > > >>> > > > > > > > >> > > > > > > marker is appended to the transaction
> > log
> > > >>> > > partition.
> > > >>> > > > > > > > >> > > > > > >
> > > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once we move past
> the
> > > >>> Prepare
> > > >>> > > and
> > > >>> > > > > > > > Complete
> > > >>> > > > > > > > >> > > states,
> > > >>> > > > > > > > >> > > > > we
> > > >>> > > > > > > > >> > > > > > > don’t need to worry about lastSeen
> > fields
> > > >>> and
> > > >>> > > clear
> > > >>> > > > > > them,
> > > >>> > > > > > > > just
> > > >>> > > > > > > > >> > > handle
> > > >>> > > > > > > > >> > > > > > state
> > > >>> > > > > > > > >> > > > > > > transitions as normal.". Is the
> lastSeen
> > > >>> field
> > > >>> > the
> > > >>> > > > > same
> > > >>> > > > > > as
> > > >>> > > > > > > > the
> > > >>> > > > > > > > >> > > > previous
> > > >>> > > > > > > > >> > > > > > > Produce Id tagged field in
> > > >>> TransactionLogValue?
> > > >>> > > > > > > > >> > > > > > >
> > > >>> > > > > > > > >> > > > > > > 112. Since the kip changes the
> > > inter-broker
> > > >>> > > > protocol,
> > > >>> > > > > > > should
> > > >>> > > > > > > > >> we
> > > >>> > > > > > > > >> > > bump
> > > >>> > > > > > > > >> > > > up
> > > >>> > > > > > > > >> > > > > > the
> > > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this feature only
> for
> > > the
> > > >>> > KRaft
> > > >>> > > > > mode?
> > > >>> > > > > > > > >> > > > > > >
> > > >>> > > > > > > > >> > > > > > > Thanks,
> > > >>> > > > > > > > >> > > > > > >
> > > >>> > > > > > > > >> > > > > > > Jun
> > > >>> > > > > > > > >> > > > > > >
> > > >>> > > > > > > > >> > > > > > >
> > > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at 11:13 AM
> Justine
> > > >>> Olshan
> > > >>> > > > > > > > >> > > > > > > <jo...@confluent.io.invalid> wrote:
> > > >>> > > > > > > > >> > > > > > >
> > > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > > >>> > > > > > > > >> > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > I'm glad we are getting to
> convergence
> > > on
> > > >>> the
> > > >>> > > > > design.
> > > >>> > > > > > :)
> > > >>> > > > > > > > >> > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > While I understand it seems a little
> > > >>> "weird".
> > > >>> > > I'm
> > > >>> > > > > not
> > > >>> > > > > > > sure
> > > >>> > > > > > > > >> what
> > > >>> > > > > > > > >> > > the
> > > >>> > > > > > > > >> > > > > > > benefit
> > > >>> > > > > > > > >> > > > > > > > of writing an extra record to the
> log.
> > > >>> > > > > > > > >> > > > > > > > Is the concern a tool to describe
> > > >>> transactions
> > > >>> > > > won't
> > > >>> > > > > > > work
> > > >>> > > > > > > > >> (ie,
> > > >>> > > > > > > > >> > > the
> > > >>> > > > > > > > >> > > > > > > complete
> > > >>> > > > > > > > >> > > > > > > > state is needed to calculate the
> time
> > > >>> since
> > > >>> > the
> > > >>> > > > > > > > transaction
> > > >>> > > > > > > > >> > > > > completed?)
> > > >>> > > > > > > > >> > > > > > > > If we have a reason like this, it is
> > > >>> enough to
> > > >>> > > > > > convince
> > > >>> > > > > > > me
> > > >>> > > > > > > > >> we
> > > >>> > > > > > > > >> > > need
> > > >>> > > > > > > > >> > > > > such
> > > >>> > > > > > > > >> > > > > > > an
> > > >>> > > > > > > > >> > > > > > > > extra record. It seems like it would
> > be
> > > >>> > > replacing
> > > >>> > > > > the
> > > >>> > > > > > > > record
> > > >>> > > > > > > > >> > > > written
> > > >>> > > > > > > > >> > > > > on
> > > >>> > > > > > > > >> > > > > > > > InitProducerId. Is this correct?
> > > >>> > > > > > > > >> > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > Thanks,
> > > >>> > > > > > > > >> > > > > > > > Justine
> > > >>> > > > > > > > >> > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024 at 5:14 PM Jun
> > Rao
> > > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > > >>> > > > > > > > >> > > >
> > > >>> > > > > > > > >> > > > > > > wrote:
> > > >>> > > > > > > > >> > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > > >>> > > > > > > > >> > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > Thanks for the explanation. I
> > > >>> understand the
> > > >>> > > > > > intention
> > > >>> > > > > > > > >> now.
> > > >>> > > > > > > > >> > In
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > > > > overflow
> > > >>> > > > > > > > >> > > > > > > > > case, we set the non-tagged field
> to
> > > >>> the old
> > > >>> > > pid
> > > >>> > > > > > (and
> > > >>> > > > > > > > the
> > > >>> > > > > > > > >> max
> > > >>> > > > > > > > >> > > > > epoch)
> > > >>> > > > > > > > >> > > > > > in
> > > >>> > > > > > > > >> > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > prepare marker so that we could
> > > >>> correctly
> > > >>> > > write
> > > >>> > > > > the
> > > >>> > > > > > > > >> marker to
> > > >>> > > > > > > > >> > > the
> > > >>> > > > > > > > >> > > > > > data
> > > >>> > > > > > > > >> > > > > > > > > partition if the broker
> downgrades.
> > > When
> > > >>> > > writing
> > > >>> > > > > the
> > > >>> > > > > > > > >> complete
> > > >>> > > > > > > > >> > > > > marker,
> > > >>> > > > > > > > >> > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > know the marker has already been
> > > >>> written to
> > > >>> > > the
> > > >>> > > > > data
> > > >>> > > > > > > > >> > partition.
> > > >>> > > > > > > > >> > > > We
> > > >>> > > > > > > > >> > > > > > set
> > > >>> > > > > > > > >> > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > non-tagged field to the new pid to
> > > avoid
> > > >>> > > > > > > > >> > > > InvalidPidMappingException
> > > >>> > > > > > > > >> > > > > > in
> > > >>> > > > > > > > >> > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > client if the broker downgrades.
> > > >>> > > > > > > > >> > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > The above seems to work. It's
> just a
> > > bit
> > > >>> > > > > > inconsistent
> > > >>> > > > > > > > for
> > > >>> > > > > > > > >> a
> > > >>> > > > > > > > >> > > > prepare
> > > >>> > > > > > > > >> > > > > > > > marker
> > > >>> > > > > > > > >> > > > > > > > > and a complete marker to use
> > different
> > > >>> pids
> > > >>> > in
> > > >>> > > > > this
> > > >>> > > > > > > > >> special
> > > >>> > > > > > > > >> > > case.
> > > >>> > > > > > > > >> > > > > If
> > > >>> > > > > > > > >> > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > downgrade with the complete
> marker,
> > it
> > > >>> seems
> > > >>> > > > that
> > > >>> > > > > we
> > > >>> > > > > > > > will
> > > >>> > > > > > > > >> > never
> > > >>> > > > > > > > >> > > > be
> > > >>> > > > > > > > >> > > > > > able
> > > >>> > > > > > > > >> > > > > > > > to
> > > >>> > > > > > > > >> > > > > > > > > write the complete marker with the
> > old
> > > >>> pid.
> > > >>> > > Not
> > > >>> > > > > sure
> > > >>> > > > > > > if
> > > >>> > > > > > > > it
> > > >>> > > > > > > > >> > > causes
> > > >>> > > > > > > > >> > > > > any
> > > >>> > > > > > > > >> > > > > > > > > issue, but it seems a bit weird.
> > > >>> Instead of
> > > >>> > > > > writing
> > > >>> > > > > > > the
> > > >>> > > > > > > > >> > > complete
> > > >>> > > > > > > > >> > > > > > marker
> > > >>> > > > > > > > >> > > > > > > > > with the new pid, could we write
> two
> > > >>> > records:
> > > >>> > > a
> > > >>> > > > > > > complete
> > > >>> > > > > > > > >> > marker
> > > >>> > > > > > > > >> > > > > with
> > > >>> > > > > > > > >> > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > old pid followed by a
> > > >>> TransactionLogValue
> > > >>> > with
> > > >>> > > > the
> > > >>> > > > > > new
> > > >>> > > > > > > > pid
> > > >>> > > > > > > > >> > and
> > > >>> > > > > > > > >> > > an
> > > >>> > > > > > > > >> > > > > > empty
> > > >>> > > > > > > > >> > > > > > > > > state? We could make the two
> records
> > > in
> > > >>> the
> > > >>> > > same
> > > >>> > > > > > batch
> > > >>> > > > > > > > so
> > > >>> > > > > > > > >> > that
> > > >>> > > > > > > > >> > > > they
> > > >>> > > > > > > > >> > > > > > > will
> > > >>> > > > > > > > >> > > > > > > > be
> > > >>> > > > > > > > >> > > > > > > > > added to the log atomically.
> > > >>> > > > > > > > >> > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > Thanks,
> > > >>> > > > > > > > >> > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > Jun
> > > >>> > > > > > > > >> > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024 at 5:40 PM
> > > Justine
> > > >>> > Olshan
> > > >>> > > > > > > > >> > > > > > > > > <jo...@confluent.io.invalid>
> > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > >>> > > > > > > > >> > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > (1) the prepare marker is
> written,
> > > >>> but the
> > > >>> > > > > endTxn
> > > >>> > > > > > > > >> response
> > > >>> > > > > > > > >> > is
> > > >>> > > > > > > > >> > > > not
> > > >>> > > > > > > > >> > > > > > > > > received
> > > >>> > > > > > > > >> > > > > > > > > > by the client when the server
> > > >>> downgrades
> > > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare marker is
> > written,
> > > >>> the
> > > >>> > > endTxn
> > > >>> > > > > > > > response
> > > >>> > > > > > > > >> is
> > > >>> > > > > > > > >> > > > > received
> > > >>> > > > > > > > >> > > > > > > by
> > > >>> > > > > > > > >> > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > client when the server
> downgrades.
> > > >>> > > > > > > > >> > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > I think I am still a little
> > > confused.
> > > >>> In
> > > >>> > > both
> > > >>> > > > of
> > > >>> > > > > > > these
> > > >>> > > > > > > > >> > cases,
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > > > > > > transaction log has the old
> > producer
> > > >>> ID.
> > > >>> > We
> > > >>> > > > > don't
> > > >>> > > > > > > > write
> > > >>> > > > > > > > >> the
> > > >>> > > > > > > > >> > > new
> > > >>> > > > > > > > >> > > > > > > > producer
> > > >>> > > > > > > > >> > > > > > > > > ID
> > > >>> > > > > > > > >> > > > > > > > > > in the prepare marker's non
> tagged
> > > >>> fields.
> > > >>> > > > > > > > >> > > > > > > > > > If the server downgrades now, it
> > > would
> > > >>> > read
> > > >>> > > > the
> > > >>> > > > > > > > records
> > > >>> > > > > > > > >> not
> > > >>> > > > > > > > >> > > in
> > > >>> > > > > > > > >> > > > > > tagged
> > > >>> > > > > > > > >> > > > > > > > > > fields and the complete marker
> > will
> > > >>> also
> > > >>> > > have
> > > >>> > > > > the
> > > >>> > > > > > > old
> > > >>> > > > > > > > >> > > producer
> > > >>> > > > > > > > >> > > > > ID.
> > > >>> > > > > > > > >> > > > > > > > > > (If we had used the new producer
> > ID,
> > > >>> we
> > > >>> > > would
> > > >>> > > > > not
> > > >>> > > > > > > have
> > > >>> > > > > > > > >> > > > > > transactional
> > > >>> > > > > > > > >> > > > > > > > > > correctness since the producer
> id
> > > >>> doesn't
> > > >>> > > > match
> > > >>> > > > > > the
> > > >>> > > > > > > > >> > > transaction
> > > >>> > > > > > > > >> > > > > and
> > > >>> > > > > > > > >> > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > state would not be correct on
> the
> > > data
> > > >>> > > > > partition.)
> > > >>> > > > > > > > >> > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > In the overflow case, I'd expect
> > the
> > > >>> > > following
> > > >>> > > > > to
> > > >>> > > > > > > > >> happen on
> > > >>> > > > > > > > >> > > the
> > > >>> > > > > > > > >> > > > > > > client
> > > >>> > > > > > > > >> > > > > > > > > side
> > > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we retry EndTxn -- it
> > is
> > > >>> the
> > > >>> > same
> > > >>> > > > > > > producer
> > > >>> > > > > > > > ID
> > > >>> > > > > > > > >> > and
> > > >>> > > > > > > > >> > > > > epoch
> > > >>> > > > > > > > >> > > > > > -
> > > >>> > > > > > > > >> > > > > > > 1
> > > >>> > > > > > > > >> > > > > > > > > this
> > > >>> > > > > > > > >> > > > > > > > > > would fence the producer
> > > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't retry EndTxn
> > and
> > > >>> use
> > > >>> > the
> > > >>> > > > new
> > > >>> > > > > > > > >> producer id
> > > >>> > > > > > > > >> > > > which
> > > >>> > > > > > > > >> > > > > > > would
> > > >>> > > > > > > > >> > > > > > > > > > result in
> > InvalidPidMappingException
> > > >>> > > > > > > > >> > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > Maybe we can have special
> handling
> > > for
> > > >>> > when
> > > >>> > > a
> > > >>> > > > > > server
> > > >>> > > > > > > > >> > > > downgrades.
> > > >>> > > > > > > > >> > > > > > When
> > > >>> > > > > > > > >> > > > > > > > it
> > > >>> > > > > > > > >> > > > > > > > > > reconnects we could get an API
> > > version
> > > >>> > > request
> > > >>> > > > > > > showing
> > > >>> > > > > > > > >> > > KIP-890
> > > >>> > > > > > > > >> > > > > > part 2
> > > >>> > > > > > > > >> > > > > > > > is
> > > >>> > > > > > > > >> > > > > > > > > > not supported. In that case, we
> > can
> > > >>> call
> > > >>> > > > > > > > initProducerId
> > > >>> > > > > > > > >> to
> > > >>> > > > > > > > >> > > > abort
> > > >>> > > > > > > > >> > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > transaction. (In the overflow
> > case,
> > > >>> this
> > > >>> > > > > correctly
> > > >>> > > > > > > > gives
> > > >>> > > > > > > > >> > us a
> > > >>> > > > > > > > >> > > > new
> > > >>> > > > > > > > >> > > > > > > > > producer
> > > >>> > > > > > > > >> > > > > > > > > > ID)
> > > >>> > > > > > > > >> > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > I guess the corresponding case
> > would
> > > >>> be
> > > >>> > > where
> > > >>> > > > > the
> > > >>> > > > > > > > >> *complete
> > > >>> > > > > > > > >> > > > > marker
> > > >>> > > > > > > > >> > > > > > > *is
> > > >>> > > > > > > > >> > > > > > > > > > written but the endTxn is not
> > > >>> received by
> > > >>> > > the
> > > >>> > > > > > client
> > > >>> > > > > > > > and
> > > >>> > > > > > > > >> > the
> > > >>> > > > > > > > >> > > > > server
> > > >>> > > > > > > > >> > > > > > > > > > downgrades? This would result in
> > the
> > > >>> > > > transaction
> > > >>> > > > > > > > >> > coordinator
> > > >>> > > > > > > > >> > > > > having
> > > >>> > > > > > > > >> > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > new
> > > >>> > > > > > > > >> > > > > > > > > > ID and not the old one.  If the
> > > client
> > > >>> > > > retries,
> > > >>> > > > > it
> > > >>> > > > > > > > will
> > > >>> > > > > > > > >> > > receive
> > > >>> > > > > > > > >> > > > > an
> > > >>> > > > > > > > >> > > > > > > > > > InvalidPidMappingException. The
> > > >>> > > InitProducerId
> > > >>> > > > > > > > scenario
> > > >>> > > > > > > > >> > above
> > > >>> > > > > > > > >> > > > > would
> > > >>> > > > > > > > >> > > > > > > > help
> > > >>> > > > > > > > >> > > > > > > > > > here too.
> > > >>> > > > > > > > >> > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > To be clear, my compatibility
> > story
> > > is
> > > >>> > meant
> > > >>> > > > to
> > > >>> > > > > > > > support
> > > >>> > > > > > > > >> > > > > downgrades
> > > >>> > > > > > > > >> > > > > > > > server
> > > >>> > > > > > > > >> > > > > > > > > > side in keeping the
> transactional
> > > >>> > > correctness.
> > > >>> > > > > > > Keeping
> > > >>> > > > > > > > >> the
> > > >>> > > > > > > > >> > > > client
> > > >>> > > > > > > > >> > > > > > > from
> > > >>> > > > > > > > >> > > > > > > > > > fencing itself is not the
> > priority.
> > > >>> > > > > > > > >> > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I can also add
> > text
> > > >>> in
> > > >>> > the
> > > >>> > > > KIP
> > > >>> > > > > > > about
> > > >>> > > > > > > > >> > > > > > InitProducerId
> > > >>> > > > > > > > >> > > > > > > if
> > > >>> > > > > > > > >> > > > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > think that fixes some edge
> cases.
> > > >>> > > > > > > > >> > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > Justine
> > > >>> > > > > > > > >> > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12, 2024 at 4:10 PM
> > Jun
> > > >>> Rao
> > > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > > >>> > > > > > > > >> > > > > >
> > > >>> > > > > > > > >> > > > > > > > > wrote:
> > > >>> > > > > > > > >> > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > > >>> > > > > > > > >> > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > Thanks for the reply.
> > > >>> > > > > > > > >> > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > I agree that we don't need to
> > > >>> optimize
> > > >>> > for
> > > >>> > > > > > fencing
> > > >>> > > > > > > > >> during
> > > >>> > > > > > > > >> > > > > > > downgrades.
> > > >>> > > > > > > > >> > > > > > > > > > > Regarding consistency, there
> are
> > > two
> > > >>> > > > possible
> > > >>> > > > > > > cases:
> > > >>> > > > > > > > >> (1)
> > > >>> > > > > > > > >> > > the
> > > >>> > > > > > > > >> > > > > > > prepare
> > > >>> > > > > > > > >> > > > > > > > > > marker
> > > >>> > > > > > > > >> > > > > > > > > > > is written, but the endTxn
> > > response
> > > >>> is
> > > >>> > not
> > > >>> > > > > > > received
> > > >>> > > > > > > > by
> > > >>> > > > > > > > >> > the
> > > >>> > > > > > > > >> > > > > client
> > > >>> > > > > > > > >> > > > > > > > when
> > > >>> > > > > > > > >> > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > server downgrades; (2)  the
> > > prepare
> > > >>> > marker
> > > >>> > > > is
> > > >>> > > > > > > > written,
> > > >>> > > > > > > > >> > the
> > > >>> > > > > > > > >> > > > > endTxn
> > > >>> > > > > > > > >> > > > > > > > > > response
> > > >>> > > > > > > > >> > > > > > > > > > > is received by the client when
> > the
> > > >>> > server
> > > >>> > > > > > > > downgrades.
> > > >>> > > > > > > > >> In
> > > >>> > > > > > > > >> > > (1),
> > > >>> > > > > > > > >> > > > > the
> > > >>> > > > > > > > >> > > > > > > > > client
> > > >>> > > > > > > > >> > > > > > > > > > > will have the old produce Id
> and
> > > in
> > > >>> (2),
> > > >>> > > the
> > > >>> > > > > > > client
> > > >>> > > > > > > > >> will
> > > >>> > > > > > > > >> > > have
> > > >>> > > > > > > > >> > > > > the
> > > >>> > > > > > > > >> > > > > > > new
> > > >>> > > > > > > > >> > > > > > > > > > > produce Id. If we downgrade
> > right
> > > >>> after
> > > >>> > > the
> > > >>> > > > > > > prepare
> > > >>> > > > > > > > >> > marker,
> > > >>> > > > > > > > >> > > > we
> > > >>> > > > > > > > >> > > > > > > can't
> > > >>> > > > > > > > >> > > > > > > > be
> > > >>> > > > > > > > >> > > > > > > > > > > consistent to both (1) and (2)
> > > >>> since we
> > > >>> > > can
> > > >>> > > > > only
> > > >>> > > > > > > put
> > > >>> > > > > > > > >> one
> > > >>> > > > > > > > >> > > > value
> > > >>> > > > > > > > >> > > > > in
> > > >>> > > > > > > > >> > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > existing produce Id field.
> It's
> > > >>> also not
> > > >>> > > > clear
> > > >>> > > > > > > which
> > > >>> > > > > > > > >> case
> > > >>> > > > > > > > >> > > is
> > > >>> > > > > > > > >> > > > > more
> > > >>> > > > > > > > >> > > > > > > > > likely.
> > > >>> > > > > > > > >> > > > > > > > > > > So we could probably be
> > consistent
> > > >>> with
> > > >>> > > > either
> > > >>> > > > > > > case.
> > > >>> > > > > > > > >> By
> > > >>> > > > > > > > >> > > > putting
> > > >>> > > > > > > > >> > > > > > the
> > > >>> > > > > > > > >> > > > > > > > new
> > > >>> > > > > > > > >> > > > > > > > > > > producer Id in the prepare
> > marker,
> > > >>> we
> > > >>> > are
> > > >>> > > > > > > consistent
> > > >>> > > > > > > > >> with
> > > >>> > > > > > > > >> > > > case
> > > >>> > > > > > > > >> > > > > > (2)
> > > >>> > > > > > > > >> > > > > > > > and
> > > >>> > > > > > > > >> > > > > > > > > it
> > > >>> > > > > > > > >> > > > > > > > > > > also has the slight benefit
> that
> > > the
> > > >>> > > produce
> > > >>> > > > > > field
> > > >>> > > > > > > > in
> > > >>> > > > > > > > >> the
> > > >>> > > > > > > > >> > > > > prepare
> > > >>> > > > > > > > >> > > > > > > and
> > > >>> > > > > > > > >> > > > > > > > > > > complete marker are consistent
> > in
> > > >>> the
> > > >>> > > > overflow
> > > >>> > > > > > > case.
> > > >>> > > > > > > > >> > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > Jun
> > > >>> > > > > > > > >> > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12, 2024 at
> 3:11 PM
> > > >>> Justine
> > > >>> > > > Olshan
> > > >>> > > > > > > > >> > > > > > > > > > > <jolshan@confluent.io.invalid
> >
> > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > >>> > > > > > > > >> > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > > >>> > > > > > > > >> > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > In the case you describe, we
> > > would
> > > >>> > need
> > > >>> > > to
> > > >>> > > > > > have
> > > >>> > > > > > > a
> > > >>> > > > > > > > >> > delayed
> > > >>> > > > > > > > >> > > > > > > request,
> > > >>> > > > > > > > >> > > > > > > > > > send a
> > > >>> > > > > > > > >> > > > > > > > > > > > successful EndTxn, and a
> > > >>> successful
> > > >>> > > > > > > > >> AddPartitionsToTxn
> > > >>> > > > > > > > >> > > and
> > > >>> > > > > > > > >> > > > > then
> > > >>> > > > > > > > >> > > > > > > > have
> > > >>> > > > > > > > >> > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn request go
> > > through
> > > >>> for
> > > >>> > a
> > > >>> > > > > given
> > > >>> > > > > > > > >> producer.
> > > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to figure out if
> it
> > > is
> > > >>> > > possible
> > > >>> > > > > for
> > > >>> > > > > > > the
> > > >>> > > > > > > > >> > client
> > > >>> > > > > > > > >> > > > to
> > > >>> > > > > > > > >> > > > > > > > > transition
> > > >>> > > > > > > > >> > > > > > > > > > > if
> > > >>> > > > > > > > >> > > > > > > > > > > > a previous request is
> delayed
> > > >>> > somewhere.
> > > >>> > > > But
> > > >>> > > > > > > yes,
> > > >>> > > > > > > > in
> > > >>> > > > > > > > >> > this
> > > >>> > > > > > > > >> > > > > case
> > > >>> > > > > > > > >> > > > > > I
> > > >>> > > > > > > > >> > > > > > > > > think
> > > >>> > > > > > > > >> > > > > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > > > would fence the client.
> > > >>> > > > > > > > >> > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > Not for the overflow case.
> In
> > > the
> > > >>> > > overflow
> > > >>> > > > > > case,
> > > >>> > > > > > > > the
> > > >>> > > > > > > > >> > > > producer
> > > >>> > > > > > > > >> > > > > > ID
> > > >>> > > > > > > > >> > > > > > > > and
> > > >>> > > > > > > > >> > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > epoch are different on the
> > > marker
> > > >>> and
> > > >>> > on
> > > >>> > > > the
> > > >>> > > > > > new
> > > >>> > > > > > > > >> > > > transaction.
> > > >>> > > > > > > > >> > > > > > So
> > > >>> > > > > > > > >> > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > want
> > > >>> > > > > > > > >> > > > > > > > > > > > the marker to use the max
> > epoch
> > > >>> but
> > > >>> > the
> > > >>> > > > new
> > > >>> > > > > > > > >> > transaction
> > > >>> > > > > > > > >> > > > > should
> > > >>> > > > > > > > >> > > > > > > > start
> > > >>> > > > > > > > >> > > > > > > > > > > with
> > > >>> > > > > > > > >> > > > > > > > > > > > the new ID and epoch 0 in
> the
> > > >>> > > > transactional
> > > >>> > > > > > > state.
> > > >>> > > > > > > > >> > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > In the server downgrade
> case,
> > we
> > > >>> want
> > > >>> > to
> > > >>> > > > see
> > > >>> > > > > > the
> > > >>> > > > > > > > >> > producer
> > > >>> > > > > > > > >> > > > ID
> > > >>> > > > > > > > >> > > > > as
> > > >>> > > > > > > > >> > > > > > > > that
> > > >>> > > > > > > > >> > > > > > > > > is
> > > >>> > > > > > > > >> > > > > > > > > > > > what the client will have.
> If
> > we
> > > >>> > > complete
> > > >>> > > > > the
> > > >>> > > > > > > > >> commit,
> > > >>> > > > > > > > >> > and
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > > > > > > transaction
> > > >>> > > > > > > > >> > > > > > > > > > > > state is reloaded, we need
> the
> > > new
> > > >>> > > > producer
> > > >>> > > > > ID
> > > >>> > > > > > > in
> > > >>> > > > > > > > >> the
> > > >>> > > > > > > > >> > > state
> > > >>> > > > > > > > >> > > > > so
> > > >>> > > > > > > > >> > > > > > > > there
> > > >>> > > > > > > > >> > > > > > > > > > > isn't
> > > >>> > > > > > > > >> > > > > > > > > > > > an invalid producer ID
> > mapping.
> > > >>> > > > > > > > >> > > > > > > > > > > > The server downgrade cases
> are
> > > >>> > > considering
> > > >>> > > > > > > > >> > transactional
> > > >>> > > > > > > > >> > > > > > > > correctness
> > > >>> > > > > > > > >> > > > > > > > > > and
> > > >>> > > > > > > > >> > > > > > > > > > > > not regressing from previous
> > > >>> behavior
> > > >>> > --
> > > >>> > > > and
> > > >>> > > > > > are
> > > >>> > > > > > > > not
> > > >>> > > > > > > > >> > > > > concerned
> > > >>> > > > > > > > >> > > > > > > > about
> > > >>> > > > > > > > >> > > > > > > > > > > > supporting the safety from
> > > fencing
> > > >>> > > retries
> > > >>> > > > > (as
> > > >>> > > > > > > we
> > > >>> > > > > > > > >> have
> > > >>> > > > > > > > >> > > > > > downgraded
> > > >>> > > > > > > > >> > > > > > > > so
> > > >>> > > > > > > > >> > > > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > > > don't need to support).
> > Perhaps
> > > >>> this
> > > >>> > is
> > > >>> > > a
> > > >>> > > > > > trade
> > > >>> > > > > > > > off,
> > > >>> > > > > > > > >> > but
> > > >>> > > > > > > > >> > > I
> > > >>> > > > > > > > >> > > > > > think
> > > >>> > > > > > > > >> > > > > > > it
> > > >>> > > > > > > > >> > > > > > > > > is
> > > >>> > > > > > > > >> > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > > >>> > > > > > > > >> > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > (If the client downgrades,
> it
> > > will
> > > >>> > have
> > > >>> > > > > > > restarted
> > > >>> > > > > > > > >> and
> > > >>> > > > > > > > >> > it
> > > >>> > > > > > > > >> > > is
> > > >>> > > > > > > > >> > > > > ok
> > > >>> > > > > > > > >> > > > > > > for
> > > >>> > > > > > > > >> > > > > > > > it
> > > >>> > > > > > > > >> > > > > > > > > > to
> > > >>> > > > > > > > >> > > > > > > > > > > > have a new producer ID too).
> > > >>> > > > > > > > >> > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > > >>> > > > > > > > >> > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan 12, 2024 at
> > 11:42 AM
> > > >>> Jun
> > > >>> > Rao
> > > >>> > > > > > > > >> > > > > > > <jun@confluent.io.invalid
> > > >>> > > > > > > > >> > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > > >>> > > > > > > > >> > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for the reply.
> > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If the marker is
> > > written
> > > >>> by
> > > >>> > the
> > > >>> > > > new
> > > >>> > > > > > > > >> client, we
> > > >>> > > > > > > > >> > > can
> > > >>> > > > > > > > >> > > > > as
> > > >>> > > > > > > > >> > > > > > I
> > > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > >>> > > > > > > > >> > > > > > > > > > > > > the last email guarantee
> > that
> > > >>> any
> > > >>> > > EndTxn
> > > >>> > > > > > > > requests
> > > >>> > > > > > > > >> > with
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > > same
> > > >>> > > > > > > > >> > > > > > > > > epoch
> > > >>> > > > > > > > >> > > > > > > > > > > are
> > > >>> > > > > > > > >> > > > > > > > > > > > > from the same producer and
> > the
> > > >>> same
> > > >>> > > > > > > transaction.
> > > >>> > > > > > > > >> Then
> > > >>> > > > > > > > >> > > we
> > > >>> > > > > > > > >> > > > > > don't
> > > >>> > > > > > > > >> > > > > > > > have
> > > >>> > > > > > > > >> > > > > > > > > > to
> > > >>> > > > > > > > >> > > > > > > > > > > > > return a fenced error but
> > can
> > > >>> handle
> > > >>> > > > > > > gracefully
> > > >>> > > > > > > > as
> > > >>> > > > > > > > >> > > > > described
> > > >>> > > > > > > > >> > > > > > in
> > > >>> > > > > > > > >> > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > > >>> > > > > > > > >> > > > > > > > > > > > > When a delayed EndTnx
> > request
> > > is
> > > >>> > > > > processed,
> > > >>> > > > > > > the
> > > >>> > > > > > > > >> txn
> > > >>> > > > > > > > >> > > state
> > > >>> > > > > > > > >> > > > > > could
> > > >>> > > > > > > > >> > > > > > > > be
> > > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > > >>> > > > > > > > >> > > > > > > > > > > > > for the next txn. I guess
> in
> > > >>> this
> > > >>> > case
> > > >>> > > > we
> > > >>> > > > > > > still
> > > >>> > > > > > > > >> > return
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > > > fenced
> > > >>> > > > > > > > >> > > > > > > > > > error
> > > >>> > > > > > > > >> > > > > > > > > > > > for
> > > >>> > > > > > > > >> > > > > > > > > > > > > the delayed request?
> > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry, my question
> was
> > > >>> > > inaccurate.
> > > >>> > > > > What
> > > >>> > > > > > > you
> > > >>> > > > > > > > >> > > > described
> > > >>> > > > > > > > >> > > > > is
> > > >>> > > > > > > > >> > > > > > > > > > accurate.
> > > >>> > > > > > > > >> > > > > > > > > > > > > "The downgrade
> > compatibility I
> > > >>> > mention
> > > >>> > > > is
> > > >>> > > > > > that
> > > >>> > > > > > > > we
> > > >>> > > > > > > > >> > keep
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > > same
> > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in the main
> > > >>> (non-tagged)
> > > >>> > > > fields
> > > >>> > > > > as
> > > >>> > > > > > > we
> > > >>> > > > > > > > >> did
> > > >>> > > > > > > > >> > > > before
> > > >>> > > > > > > > >> > > > > > the
> > > >>> > > > > > > > >> > > > > > > > code
> > > >>> > > > > > > > >> > > > > > > > > > on
> > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > server side." If we want
> to
> > do
> > > >>> this,
> > > >>> > > it
> > > >>> > > > > > seems
> > > >>> > > > > > > > >> that we
> > > >>> > > > > > > > >> > > > > should
> > > >>> > > > > > > > >> > > > > > > use
> > > >>> > > > > > > > >> > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > current produce Id and max
> > > >>> epoch in
> > > >>> > > the
> > > >>> > > > > > > existing
> > > >>> > > > > > > > >> > > > producerId
> > > >>> > > > > > > > >> > > > > > and
> > > >>> > > > > > > > >> > > > > > > > > > > > > producerEpoch fields for
> > both
> > > >>> the
> > > >>> > > > prepare
> > > >>> > > > > > and
> > > >>> > > > > > > > the
> > > >>> > > > > > > > >> > > > complete
> > > >>> > > > > > > > >> > > > > > > > marker,
> > > >>> > > > > > > > >> > > > > > > > > > > right?
> > > >>> > > > > > > > >> > > > > > > > > > > > > The downgrade can happen
> > after
> > > >>> the
> > > >>> > > > > complete
> > > >>> > > > > > > > >> marker is
> > > >>> > > > > > > > >> > > > > > written.
> > > >>> > > > > > > > >> > > > > > > > With
> > > >>> > > > > > > > >> > > > > > > > > > > what
> > > >>> > > > > > > > >> > > > > > > > > > > > > you described, the
> > downgraded
> > > >>> > > > coordinator
> > > >>> > > > > > will
> > > >>> > > > > > > > see
> > > >>> > > > > > > > >> > the
> > > >>> > > > > > > > >> > > > new
> > > >>> > > > > > > > >> > > > > > > > produce
> > > >>> > > > > > > > >> > > > > > > > > Id
> > > >>> > > > > > > > >> > > > > > > > > > > > > instead of the old one.
> > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan 12, 2024 at
> > > 10:44 AM
> > > >>> > > Justine
> > > >>> > > > > > > Olshan
> > > >>> > > > > > > > >> > > > > > > > > > > > >
> > <jolshan@confluent.io.invalid
> > > >
> > > >>> > wrote:
> > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > I can update the
> > > description.
> > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > I believe your second
> > point
> > > is
> > > >>> > > > mentioned
> > > >>> > > > > > in
> > > >>> > > > > > > > the
> > > >>> > > > > > > > >> > KIP.
> > > >>> > > > > > > > >> > > I
> > > >>> > > > > > > > >> > > > > can
> > > >>> > > > > > > > >> > > > > > > add
> > > >>> > > > > > > > >> > > > > > > > > more
> > > >>> > > > > > > > >> > > > > > > > > > > > text
> > > >>> > > > > > > > >> > > > > > > > > > > > > on
> > > >>> > > > > > > > >> > > > > > > > > > > > > > this if it is helpful.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > The delayed message
> case
> > > can
> > > >>> > also
> > > >>> > > > > > violate
> > > >>> > > > > > > > EOS
> > > >>> > > > > > > > >> if
> > > >>> > > > > > > > >> > > the
> > > >>> > > > > > > > >> > > > > > > delayed
> > > >>> > > > > > > > >> > > > > > > > > > > message
> > > >>> > > > > > > > >> > > > > > > > > > > > > > comes in after the next
> > > >>> > > > > addPartitionsToTxn
> > > >>> > > > > > > > >> request
> > > >>> > > > > > > > >> > > > comes
> > > >>> > > > > > > > >> > > > > > in.
> > > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > > > > > may see a message from a
> > > >>> previous
> > > >>> > > > > > (aborted)
> > > >>> > > > > > > > >> > > transaction
> > > >>> > > > > > > > >> > > > > > > become
> > > >>> > > > > > > > >> > > > > > > > > part
> > > >>> > > > > > > > >> > > > > > > > > > > of
> > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > next transaction.
> > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > If the marker is written
> > by
> > > >>> the
> > > >>> > new
> > > >>> > > > > > client,
> > > >>> > > > > > > we
> > > >>> > > > > > > > >> can
> > > >>> > > > > > > > >> > > as I
> > > >>> > > > > > > > >> > > > > > > > mentioned
> > > >>> > > > > > > > >> > > > > > > > > > in
> > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > last email guarantee
> that
> > > any
> > > >>> > EndTxn
> > > >>> > > > > > > requests
> > > >>> > > > > > > > >> with
> > > >>> > > > > > > > >> > > the
> > > >>> > > > > > > > >> > > > > same
> > > >>> > > > > > > > >> > > > > > > > epoch
> > > >>> > > > > > > > >> > > > > > > > > > are
> > > >>> > > > > > > > >> > > > > > > > > > > > > from
> > > >>> > > > > > > > >> > > > > > > > > > > > > > the same producer and
> the
> > > same
> > > >>> > > > > > transaction.
> > > >>> > > > > > > > >> Then we
> > > >>> > > > > > > > >> > > > don't
> > > >>> > > > > > > > >> > > > > > > have
> > > >>> > > > > > > > >> > > > > > > > to
> > > >>> > > > > > > > >> > > > > > > > > > > > return
> > > >>> > > > > > > > >> > > > > > > > > > > > > a
> > > >>> > > > > > > > >> > > > > > > > > > > > > > fenced error but can
> > handle
> > > >>> > > gracefully
> > > >>> > > > > as
> > > >>> > > > > > > > >> described
> > > >>> > > > > > > > >> > > in
> > > >>> > > > > > > > >> > > > > the
> > > >>> > > > > > > > >> > > > > > > KIP.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > I don't think a boolean
> is
> > > >>> useful
> > > >>> > > > since
> > > >>> > > > > it
> > > >>> > > > > > > is
> > > >>> > > > > > > > >> > > directly
> > > >>> > > > > > > > >> > > > > > > encoded
> > > >>> > > > > > > > >> > > > > > > > by
> > > >>> > > > > > > > >> > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > existence or lack of the
> > > >>> tagged
> > > >>> > > field
> > > >>> > > > > > being
> > > >>> > > > > > > > >> > written.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > In the prepare marker we
> > > will
> > > >>> have
> > > >>> > > the
> > > >>> > > > > > same
> > > >>> > > > > > > > >> > producer
> > > >>> > > > > > > > >> > > ID
> > > >>> > > > > > > > >> > > > > in
> > > >>> > > > > > > > >> > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > > >>> > > > > > > > >> > > > > > > > > > > > > > field. In the Complete
> > state
> > > >>> we
> > > >>> > may
> > > >>> > > > not.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure why the
> > ongoing
> > > >>> state
> > > >>> > > > > matters
> > > >>> > > > > > > for
> > > >>> > > > > > > > >> this
> > > >>> > > > > > > > >> > > > KIP.
> > > >>> > > > > > > > >> > > > > It
> > > >>> > > > > > > > >> > > > > > > > does
> > > >>> > > > > > > > >> > > > > > > > > > > matter
> > > >>> > > > > > > > >> > > > > > > > > > > > > for
> > > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure what you
> are
> > > >>> > referring
> > > >>> > > to
> > > >>> > > > > > about
> > > >>> > > > > > > > >> > writing
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > > > > previous
> > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the prepare
> marker.
> > > >>> This is
> > > >>> > > not
> > > >>> > > > in
> > > >>> > > > > > the
> > > >>> > > > > > > > >> KIP.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > In the overflow case, we
> > > >>> write the
> > > >>> > > > > > > > >> nextProducerId
> > > >>> > > > > > > > >> > in
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > > > > prepare
> > > >>> > > > > > > > >> > > > > > > > > > > state.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > This is so we know what
> we
> > > >>> > assigned
> > > >>> > > > when
> > > >>> > > > > > we
> > > >>> > > > > > > > >> reload
> > > >>> > > > > > > > >> > > the
> > > >>> > > > > > > > >> > > > > > > > > transaction
> > > >>> > > > > > > > >> > > > > > > > > > > log.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > Once we complete, we
> > > >>> transition
> > > >>> > this
> > > >>> > > > ID
> > > >>> > > > > to
> > > >>> > > > > > > the
> > > >>> > > > > > > > >> main
> > > >>> > > > > > > > >> > > > > > > (non-tagged
> > > >>> > > > > > > > >> > > > > > > > > > > field)
> > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > >>> > > > > > > > >> > > > > > > > > > > > > > have the previous
> producer
> > > ID
> > > >>> > field
> > > >>> > > > > filled
> > > >>> > > > > > > in.
> > > >>> > > > > > > > >> This
> > > >>> > > > > > > > >> > > is
> > > >>> > > > > > > > >> > > > so
> > > >>> > > > > > > > >> > > > > > we
> > > >>> > > > > > > > >> > > > > > > > can
> > > >>> > > > > > > > >> > > > > > > > > > > > identify
> > > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry case the
> > > operation
> > > >>> > > > completed
> > > >>> > > > > > > > >> > successfully
> > > >>> > > > > > > > >> > > > and
> > > >>> > > > > > > > >> > > > > we
> > > >>> > > > > > > > >> > > > > > > > don't
> > > >>> > > > > > > > >> > > > > > > > > > > fence
> > > >>> > > > > > > > >> > > > > > > > > > > > > our
> > > >>> > > > > > > > >> > > > > > > > > > > > > > producer. The downgrade
> > > >>> > > compatibility
> > > >>> > > > I
> > > >>> > > > > > > > mention
> > > >>> > > > > > > > >> is
> > > >>> > > > > > > > >> > > that
> > > >>> > > > > > > > >> > > > > we
> > > >>> > > > > > > > >> > > > > > > keep
> > > >>> > > > > > > > >> > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > same
> > > >>> > > > > > > > >> > > > > > > > > > > > > > producer ID and epoch in
> > the
> > > >>> main
> > > >>> > > > > > > (non-tagged)
> > > >>> > > > > > > > >> > fields
> > > >>> > > > > > > > >> > > > as
> > > >>> > > > > > > > >> > > > > we
> > > >>> > > > > > > > >> > > > > > > did
> > > >>> > > > > > > > >> > > > > > > > > > > before
> > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > code on the server side.
> > If
> > > >>> the
> > > >>> > > server
> > > >>> > > > > > > > >> downgrades,
> > > >>> > > > > > > > >> > we
> > > >>> > > > > > > > >> > > > are
> > > >>> > > > > > > > >> > > > > > > still
> > > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > This addresses both the
> > > >>> prepare
> > > >>> > and
> > > >>> > > > > > complete
> > > >>> > > > > > > > >> state
> > > >>> > > > > > > > >> > > > > > > downgrades.
> > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri, Jan 12, 2024 at
> > > >>> 10:21 AM
> > > >>> > Jun
> > > >>> > > > Rao
> > > >>> > > > > > > > >> > > > > > > > > <jun@confluent.io.invalid
> > > >>> > > > > > > > >> > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi, Justine,
> > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks for the reply.
> > > Sorry
> > > >>> for
> > > >>> > > the
> > > >>> > > > > > > delay. I
> > > >>> > > > > > > > >> > have a
> > > >>> > > > > > > > >> > > > few
> > > >>> > > > > > > > >> > > > > > > more
> > > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I think the
> > > motivation
> > > >>> > > section
> > > >>> > > > > > could
> > > >>> > > > > > > be
> > > >>> > > > > > > > >> > > > improved.
> > > >>> > > > > > > > >> > > > > > One
> > > >>> > > > > > > > >> > > > > > > of
> > > >>> > > > > > > > >> > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > motivations listed by
> > the
> > > >>> KIP is
> > > >>> > > > "This
> > > >>> > > > > > can
> > > >>> > > > > > > > >> happen
> > > >>> > > > > > > > >> > > > when
> > > >>> > > > > > > > >> > > > > a
> > > >>> > > > > > > > >> > > > > > > > > message
> > > >>> > > > > > > > >> > > > > > > > > > > gets
> > > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > or delayed due to
> > > networking
> > > >>> > > issues
> > > >>> > > > > or a
> > > >>> > > > > > > > >> network
> > > >>> > > > > > > > >> > > > > > partition,
> > > >>> > > > > > > > >> > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > transaction
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts, and then the
> > > delayed
> > > >>> > > message
> > > >>> > > > > > > finally
> > > >>> > > > > > > > >> > comes
> > > >>> > > > > > > > >> > > > > in.".
> > > >>> > > > > > > > >> > > > > > > This
> > > >>> > > > > > > > >> > > > > > > > > > seems
> > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > very accurate. Without
> > > >>> KIP-890,
> > > >>> > > > > > currently,
> > > >>> > > > > > > > if
> > > >>> > > > > > > > >> the
> > > >>> > > > > > > > >> > > > > > > coordinator
> > > >>> > > > > > > > >> > > > > > > > > > times
> > > >>> > > > > > > > >> > > > > > > > > > > > out
> > > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts an ongoing
> > > >>> transaction,
> > > >>> > it
> > > >>> > > > > > already
> > > >>> > > > > > > > >> bumps
> > > >>> > > > > > > > >> > up
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > > > epoch
> > > >>> > > > > > > > >> > > > > > > > in
> > > >>> > > > > > > > >> > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > which prevents the
> > delayed
> > > >>> > produce
> > > >>> > > > > > message
> > > >>> > > > > > > > >> from
> > > >>> > > > > > > > >> > > being
> > > >>> > > > > > > > >> > > > > > added
> > > >>> > > > > > > > >> > > > > > > > to
> > > >>> > > > > > > > >> > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > user
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > partition. What can
> > cause
> > > a
> > > >>> > > hanging
> > > >>> > > > > > > > >> transaction
> > > >>> > > > > > > > >> > is
> > > >>> > > > > > > > >> > > > that
> > > >>> > > > > > > > >> > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > producer
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > completes (either
> aborts
> > > or
> > > >>> > > > commits) a
> > > >>> > > > > > > > >> > transaction
> > > >>> > > > > > > > >> > > > > before
> > > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > successful ack on
> > messages
> > > >>> > > published
> > > >>> > > > > in
> > > >>> > > > > > > the
> > > >>> > > > > > > > >> same
> > > >>> > > > > > > > >> > > txn.
> > > >>> > > > > > > > >> > > > > In
> > > >>> > > > > > > > >> > > > > > > this
> > > >>> > > > > > > > >> > > > > > > > > > case,
> > > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > possible for the
> delayed
> > > >>> message
> > > >>> > > to
> > > >>> > > > be
> > > >>> > > > > > > > >> appended
> > > >>> > > > > > > > >> > to
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > > > > > partition
> > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker, causing a
> > > >>> transaction to
> > > >>> > > > hang.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > A similar issue (not
> > > >>> mentioned
> > > >>> > in
> > > >>> > > > the
> > > >>> > > > > > > > >> motivation)
> > > >>> > > > > > > > >> > > > could
> > > >>> > > > > > > > >> > > > > > > > happen
> > > >>> > > > > > > > >> > > > > > > > > on
> > > >>> > > > > > > > >> > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > marker in the
> > > coordinator's
> > > >>> log.
> > > >>> > > For
> > > >>> > > > > > > > example,
> > > >>> > > > > > > > >> > it's
> > > >>> > > > > > > > >> > > > > > possible
> > > >>> > > > > > > > >> > > > > > > > for
> > > >>> > > > > > > > >> > > > > > > > > > an
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest to be
> > > delayed
> > > >>> on
> > > >>> > the
> > > >>> > > > > > > > >> coordinator.
> > > >>> > > > > > > > >> > By
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > > time
> > > >>> > > > > > > > >> > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest is
> > > processed,
> > > >>> it's
> > > >>> > > > > > possible
> > > >>> > > > > > > > that
> > > >>> > > > > > > > >> > the
> > > >>> > > > > > > > >> > > > > > previous
> > > >>> > > > > > > > >> > > > > > > > txn
> > > >>> > > > > > > > >> > > > > > > > > > has
> > > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > completed and a new
> txn
> > > has
> > > >>> > > started.
> > > >>> > > > > > > > >> Currently,
> > > >>> > > > > > > > >> > > since
> > > >>> > > > > > > > >> > > > > the
> > > >>> > > > > > > > >> > > > > > > > epoch
> > > >>> > > > > > > > >> > > > > > > > > > is
> > > >>> > > > > > > > >> > > > > > > > > > > > not
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped on every txn,
> the
> > > >>> delayed
> > > >>> > > > > > > > EndTxnRequest
> > > >>> > > > > > > > >> > will
> > > >>> > > > > > > > >> > > > add
> > > >>> > > > > > > > >> > > > > > an
> > > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare marker (and
> > > >>> eventually a
> > > >>> > > > > > complete
> > > >>> > > > > > > > >> marker)
> > > >>> > > > > > > > >> > > to
> > > >>> > > > > > > > >> > > > > the
> > > >>> > > > > > > > >> > > > > > > > > ongoing
> > > >>> > > > > > > > >> > > > > > > > > > > txn.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > won't cause the
> > > transaction
> > > >>> to
> > > >>> > > hang,
> > > >>> > > > > but
> > > >>> > > > > > > it
> > > >>> > > > > > > > >> will
> > > >>> > > > > > > > >> > > > break
> > > >>> > > > > > > > >> > > > > > the
> > > >>> > > > > > > > >> > > > > > > > EoS
> > > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > The proposal in this
> KIP
> > > >>> will
> > > >>> > > > address
> > > >>> > > > > > this
> > > >>> > > > > > > > >> issue
> > > >>> > > > > > > > >> > > too.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101. "However, I was
> > > >>> writing it
> > > >>> > so
> > > >>> > > > > that
> > > >>> > > > > > we
> > > >>> > > > > > > > can
> > > >>> > > > > > > > >> > > > > > distinguish
> > > >>> > > > > > > > >> > > > > > > > > > between
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > old clients where we
> > don't
> > > >>> have
> > > >>> > > the
> > > >>> > > > > > > ability
> > > >>> > > > > > > > do
> > > >>> > > > > > > > >> > this
> > > >>> > > > > > > > >> > > > > > > operation
> > > >>> > > > > > > > >> > > > > > > > > and
> > > >>> > > > > > > > >> > > > > > > > > > > new
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > clients that can. (Old
> > > >>> clients
> > > >>> > > don't
> > > >>> > > > > > bump
> > > >>> > > > > > > > the
> > > >>> > > > > > > > >> > epoch
> > > >>> > > > > > > > >> > > > on
> > > >>> > > > > > > > >> > > > > > > > commit,
> > > >>> > > > > > > > >> > > > > > > > > so
> > > >>> > > > > > > > >> > > > > > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > say for sure the write
> > > >>> belongs
> > > >>> > to
> > > >>> > > > the
> > > >>> > > > > > > given
> > > >>> > > > > > > > >> > > > > > transaction)."
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I am wondering
> why
> > > we
> > > >>> need
> > > >>> > > to
> > > >>> > > > > > > > >> distinguish
> > > >>> > > > > > > > >> > > > whether
> > > >>> > > > > > > > >> > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > marker
> > > >>> > > > > > > > >> > > > > > > > > > > is
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > written by the old and
> > the
> > > >>> new
> > > >>> > > > client.
> > > >>> > > > > > > Could
> > > >>> > > > > > > > >> you
> > > >>> > > > > > > > >> > > > > describe
> > > >>> > > > > > > > >> > > > > > > > what
> > > >>> > > > > > > > >> > > > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > > do
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > differently if we know
> > the
> > > >>> > marker
> > > >>> > > is
> > > >>> > > > > > > written
> > > >>> > > > > > > > >> by
> > > >>> > > > > > > > >> > the
> > > >>> > > > > > > > >> > > > new
> > > >>> > > > > > > > >> > > > > > > > client?
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If we do need a
> > way
> > > to
> > > >>> > > > > distinguish
> > > >>> > > > > > > > >> whether
> > > >>> > > > > > > > >> > > the
> > > >>> > > > > > > > >> > > > > > marker
> > > >>> > > > > > > > >> > > > > > > > is
> > > >>> > > > > > > > >> > > > > > > > > > > > written
> > > >>> > > > > > > > >> > > > > > > > > > > > > by
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > the old and the new
> > > client.
> > > >>> > Would
> > > >>> > > it
> > > >>> > > > > be
> > > >>> > > > > > > > >> simpler
> > > >>> > > > > > > > >> > to
> > > >>> > > > > > > > >> > > > just
> > > >>> > > > > > > > >> > > > > > > > > > introduce a
> > > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > field instead of
> > > indirectly
> > > >>> > > through
> > > >>> > > > > the
> > > >>> > > > > > > > >> previous
> > > >>> > > > > > > > >> > > > > produce
> > > >>> > > > > > > > >> > > > > > ID
> > > >>> > > > > > > > >> > > > > > > > > > field?
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3 It's not clear
> to
> > me
> > > >>> why
> > > >>> > we
> > > >>> > > > only
> > > >>> > > > > > add
> > > >>> > > > > > > > the
> > > >>> > > > > > > > >> > > > previous
> > > >>> > > > > > > > >> > > > > > > > produce
> > > >>> > > > > > > > >> > > > > > > > > > ID
> > > >>> > > > > > > > >> > > > > > > > > > > > > field
> > > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > the complete marker,
> but
> > > >>> not in
> > > >>> > > the
> > > >>> > > > > > > prepare
> > > >>> > > > > > > > >> > marker.
> > > >>> > > > > > > > >> > > > If
> > > >>> > > > > > > > >> > > > > we
> > > >>> > > > > > > > >> > > > > > > > want
> > > >>> > > > > > > > >> > > > > > > > > to
> > > >>> > > > > > > > >> > > > > > > > > > > > know
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > whether a marker is
> > > written
> > > >>> by
> > > >>> > the
> > > >>> > > > new
> > > >>> > > > > > > > client
> > > >>> > > > > > > > >> or
> > > >>> > > > > > > > >> > > not,
> > > >>> > > > > > > > >> > > > > it
> > > >>> > > > > > > > >> > > > > > > > seems
> > > >>> > > > > > > > >> > > > > > > > > > that
> > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > to do this
> consistently
> > > for
> > > >>> all
> > > >>> > > > > markers.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4 What about the
> > > >>> > > > > TransactionLogValue
> > > >>> > > > > > > > >> record
> > > >>> > > > > > > > >> > > > > > > representing
> > > >>> > > > > > > > >> > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > state? Should we also
> > > >>> > distinguish
> > > >>> > > > > > whether
> > > >>> > > > > > > > it's
> > > >>> > > > > > > > >> > > > written
> > > >>> > > > > > > > >> > > > > by
> > > >>> > > > > > > > >> > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > old
> > > >>> > > > > > > > >> > > > > > > > > > > or
> > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > new client?
> > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In the overflow
> > case,
> > > >>> it's
> > > >>> > > > still
> > > >>> > > > > > not
> > > >>> > > > > > > > >> clear
> > > >>> > > > > > > > >> > to
> > > >>> > > > > > > > >> > > me
> > > >>> > > > > > > > >> > > > > why
> > > >>> > > > > > > > >> > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > write
> > > >>> > > > > > > > >> > > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > previous produce Id in
> > the
> > > >>> > prepare
> > > >>> > > > > > marker
> > > >>> > > > > > > > >> while
> > > >>> > > > > > > > >> > > > writing
> > > >>> > > > > > > > >> > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > next
> > > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > in the complete
> marker.
> > > You
> > > >>> > > > mentioned
> > > >>> > > > > > that
> > > >>> > > > > > > > >> it's
> > > >>> > > > > > > > >> > for
> > > >>> > > > > > > > >> > > > > > > > > downgrading.
> > > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > we could downgrade
> with
> > > >>> either
> > > >>> > the
> > > >>> > > > > > prepare
> > > >>> > > > > > > > >> marker
> > > >>> > > > > > > > >> > > or
> > > >>> > > > > > > > >> > > > > the
> > > >>> > > > > > > > >> > > > > > > > > complete
> > > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > In either case, the
> > > >>> downgraded
> > > >>> > > > > > coordinator
> > > >>> > > > > > > > >> should
> > > >>> > > > > > > > >> > > see
> > > >>> > > > > > > > >> > > > > the
> > > >>> > > > > > > > >> > > > > > > > same
> > > >>> > > > > > > > >> > > > > > > > > > > > produce
> > > >>> > > > > > > > >> > > > > > > > > > > > > id
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > (probably the previous
> > > >>> produce
> > > >>> > > Id),
> > > >>> > > > > > right?
> > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed, Dec 20, 2023
> at
> > > >>> 6:00 PM
> > > >>> > > > > Justine
> > > >>> > > > > > > > Olshan
> > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > >>> <jo...@confluent.io.invalid>
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey Jun,
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks for taking a
> > look
> > > >>> at
> > > >>> > the
> > > >>> > > > KIP
> > > >>> > > > > > > again.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100. For the epoch
> > > >>> overflow
> > > >>> > > case,
> > > >>> > > > > only
> > > >>> > > > > > > the
> > > >>> > > > > > > > >> > marker
> > > >>> > > > > > > > >> > > > > will
> > > >>> > > > > > > > >> > > > > > > have
> > > >>> > > > > > > > >> > > > > > > > > max
> > > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps the behavior
> of
> > > the
> > > >>> rest
> > > >>> > > of
> > > >>> > > > > the
> > > >>> > > > > > > > >> markers
> > > >>> > > > > > > > >> > > where
> > > >>> > > > > > > > >> > > > > the
> > > >>> > > > > > > > >> > > > > > > > last
> > > >>> > > > > > > > >> > > > > > > > > > > marker
> > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch of the
> > transaction
> > > >>> > > records +
> > > >>> > > > > 1.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101. You are correct
> > > that
> > > >>> we
> > > >>> > > don't
> > > >>> > > > > > need
> > > >>> > > > > > > to
> > > >>> > > > > > > > >> > write
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > > > > producer
> > > >>> > > > > > > > >> > > > > > > > > > ID
> > > >>> > > > > > > > >> > > > > > > > > > > > > since
> > > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the same.
> However,
> > I
> > > >>> was
> > > >>> > > > writing
> > > >>> > > > > it
> > > >>> > > > > > > so
> > > >>> > > > > > > > >> that
> > > >>> > > > > > > > >> > we
> > > >>> > > > > > > > >> > > > can
> > > >>> > > > > > > > >> > > > > > > > > > distinguish
> > > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > old clients where we
> > > don't
> > > >>> > have
> > > >>> > > > the
> > > >>> > > > > > > > ability
> > > >>> > > > > > > > >> do
> > > >>> > > > > > > > >> > > this
> > > >>> > > > > > > > >> > > > > > > > operation
> > > >>> > > > > > > > >> > > > > > > > > > and
> > > >>> > > > > > > > >> > > > > > > > > > > > new
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > clients that can.
> (Old
> > > >>> clients
> > > >>> > > > don't
> > > >>> > > > > > > bump
> > > >>> > > > > > > > >> the
> > > >>> > > > > > > > >> > > epoch
> > > >>> > > > > > > > >> > > > > on
> > > >>> > > > > > > > >> > > > > > > > > commit,
> > > >>> > > > > > > > >> > > > > > > > > > so
> > > >>> > > > > > > > >> > > > > > > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > say for sure the
> write
> > > >>> belongs
> > > >>> > > to
> > > >>> > > > > the
> > > >>> > > > > > > > given
> > > >>> > > > > > > > >> > > > > > transaction).
> > > >>> > > > > > > > >> > > > > > > > If
> > > >>> > > > > > > > >> > > > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn request from
> a
> > > new
> > > >>> > > client,
> > > >>> > > > we
> > > >>> > > > > > > will
> > > >>> > > > > > > > >> fill
> > > >>> > > > > > > > >> > > this
> > > >>> > > > > > > > >> > > > > > > field.
> > > >>> > > > > > > > >> > > > > > > > We
> > > >>> > > > > > > > >> > > > > > > > > > can
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > guarantee
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > that any EndTxn
> > requests
> > > >>> with
> > > >>> > > the
> > > >>> > > > > same
> > > >>> > > > > > > > epoch
> > > >>> > > > > > > > >> > are
> > > >>> > > > > > > > >> > > > from
> > > >>> > > > > > > > >> > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > same
> > > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > the same
> transaction.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102. In prepare
> phase,
> > > we
> > > >>> have
> > > >>> > > the
> > > >>> > > > > > same
> > > >>> > > > > > > > >> > producer
> > > >>> > > > > > > > >> > > ID
> > > >>> > > > > > > > >> > > > > and
> > > >>> > > > > > > > >> > > > > > > > epoch
> > > >>> > > > > > > > >> > > > > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > > > > always
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > had. It is the
> > producer
> > > >>> ID and
> > > >>> > > > epoch
> > > >>> > > > > > > that
> > > >>> > > > > > > > >> are
> > > >>> > > > > > > > >> > on
> > > >>> > > > > > > > >> > > > the
> > > >>> > > > > > > > >> > > > > > > > marker.
> > > >>> > > > > > > > >> > > > > > > > > In
> > > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase, we stay the
> > same
> > > >>> unless
> > > >>> > > it
> > > >>> > > > is
> > > >>> > > > > > the
> > > >>> > > > > > > > >> > overflow
> > > >>> > > > > > > > >> > > > > case.
> > > >>> > > > > > > > >> > > > > > > In
> > > >>> > > > > > > > >> > > > > > > > > that
> > > >>> > > > > > > > >> > > > > > > > > > > > case,
> > > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > set the producer ID
> to
> > > >>> the new
> > > >>> > > one
> > > >>> > > > > we
> > > >>> > > > > > > > >> generated
> > > >>> > > > > > > > >> > > and
> > > >>> > > > > > > > >> > > > > > epoch
> > > >>> > > > > > > > >> > > > > > > > to
> > > >>> > > > > > > > >> > > > > > > > > 0
> > > >>> > > > > > > > >> > > > > > > > > > > > after
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > complete. This is
> for
> > > >>> > downgrade
> > > >>> > > > > > > > >> compatibility.
> > > >>> > > > > > > > >> > > The
> > > >>> > > > > > > > >> > > > > > tagged
> > > >>> > > > > > > > >> > > > > > > > > > fields
> > > >>> > > > > > > > >> > > > > > > > > > > > are
> > > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety guards for
> > > retries
> > > >>> and
> > > >>> > > > > > failovers.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In prepare phase for
> > > epoch
> > > >>> > > > overflow
> > > >>> > > > > > case
> > > >>> > > > > > > > >> only
> > > >>> > > > > > > > >> > we
> > > >>> > > > > > > > >> > > > > store
> > > >>> > > > > > > > >> > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > next
> > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID. This is for the
> > case
> > > >>> where
> > > >>> > > we
> > > >>> > > > > > reload
> > > >>> > > > > > > > the
> > > >>> > > > > > > > >> > > > > > transaction
> > > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > prepare state. Once
> > the
> > > >>> > > > transaction
> > > >>> > > > > is
> > > >>> > > > > > > > >> > committed,
> > > >>> > > > > > > > >> > > > we
> > > >>> > > > > > > > >> > > > > > can
> > > >>> > > > > > > > >> > > > > > > > use
> > > >>> > > > > > > > >> > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the client
> already
> > is
> > > >>> > using.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > In commit phase, we
> > > store
> > > >>> the
> > > >>> > > > > previous
> > > >>> > > > > > > > >> producer
> > > >>> > > > > > > > >> > > ID
> > > >>> > > > > > > > >> > > > in
> > > >>> > > > > > > > >> > > > > > > case
> > > >>> > > > > > > > >> > > > > > > > of
> > > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > I think it is easier
> > to
> > > >>> think
> > > >>> > of
> > > >>> > > > it
> > > >>> > > > > as
> > > >>> > > > > > > > just
> > > >>> > > > > > > > >> how
> > > >>> > > > > > > > >> > > we
> > > >>> > > > > > > > >> > > > > were
> > > >>> > > > > > > > >> > > > > > > > > storing
> > > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and epoch before,
> with
> > > >>> some
> > > >>> > > extra
> > > >>> > > > > > > > bookeeping
> > > >>> > > > > > > > >> > and
> > > >>> > > > > > > > >> > > > edge
> > > >>> > > > > > > > >> > > > > > > case
> > > >>> > > > > > > > >> > > > > > > > > > > handling
> > > >>> > > > > > > > >> > > > > > > > > > > > > in
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged fields. We
> have
> > > to
> > > >>> do
> > > >>> > it
> > > >>> > > > this
> > > >>> > > > > > way
> > > >>> > > > > > > > for
> > > >>> > > > > > > > >> > > > > > > compatibility
> > > >>> > > > > > > > >> > > > > > > > > with
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > downgrades.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103. Next producer
> ID
> > is
> > > >>> for
> > > >>> > > > prepare
> > > >>> > > > > > > > status
> > > >>> > > > > > > > >> and
> > > >>> > > > > > > > >> > > > > > previous
> > > >>> > > > > > > > >> > > > > > > > > > producer
> > > >>> > > > > > > > >> > > > > > > > > > > > ID
> > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > after complete. The
> > > >>> reason why
> > > >>> > > we
> > > >>> > > > > need
> > > >>> > > > > > > two
> > > >>> > > > > > > > >> > > separate
> > > >>> > > > > > > > >> > > > > > > > (tagged)
> > > >>> > > > > > > > >> > > > > > > > > > > fields
> > > >>> > > > > > > > >> > > > > > > > > > > > > is
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > backwards
> > compatibility.
> > > >>> We
> > > >>> > need
> > > >>> > > > to
> > > >>> > > > > > keep
> > > >>> > > > > > > > the
> > > >>> > > > > > > > >> > same
> > > >>> > > > > > > > >> > > > > > > semantics
> > > >>> > > > > > > > >> > > > > > > > > for
> > > >>> > > > > > > > >> > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged field in
> > case
> > > >>> we
> > > >>> > > > > downgrade.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104. We set the
> fields
> > > as
> > > >>> we
> > > >>> > do
> > > >>> > > in
> > > >>> > > > > the
> > > >>> > > > > > > > >> > > > transactional
> > > >>> > > > > > > > >> > > > > > > state
> > > >>> > > > > > > > >> > > > > > > > > (as
> > > >>> > > > > > > > >> > > > > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > > > > need
> > > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > do this for
> > > compatibility
> > > >>> --
> > > >>> > if
> > > >>> > > we
> > > >>> > > > > > > > >> downgrade,
> > > >>> > > > > > > > >> > we
> > > >>> > > > > > > > >> > > > will
> > > >>> > > > > > > > >> > > > > > > only
> > > >>> > > > > > > > >> > > > > > > > > have
> > > >>> > > > > > > > >> > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged fields)
> It
> > > >>> will be
> > > >>> > > the
> > > >>> > > > > old
> > > >>> > > > > > > > >> producer
> > > >>> > > > > > > > >> > ID
> > > >>> > > > > > > > >> > > > and
> > > >>> > > > > > > > >> > > > > > max
> > > >>> > > > > > > > >> > > > > > > > > > epoch.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope this helps. Let
> > me
> > > >>> know
> > > >>> > if
> > > >>> > > > you
> > > >>> > > > > > have
> > > >>> > > > > > > > >> > further
> > > >>> > > > > > > > >> > > > > > > questions.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Justine
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > On Wed, Dec 20, 2023
> > at
> > > >>> > 3:33 PM
> > > >>> > > > Jun
> > > >>> > > > > > Rao
> > > >>> > > > > > > > >> > > > > > > > > > <jun@confluent.io.invalid
> > > >>> > > > > > > > >> > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi, Justine,
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It seems that you
> > have
> > > >>> made
> > > >>> > > some
> > > >>> > > > > > > changes
> > > >>> > > > > > > > >> to
> > > >>> > > > > > > > >> > > > KIP-890
> > > >>> > > > > > > > >> > > > > > > since
> > > >>> > > > > > > > >> > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > particular, we are
> > > >>> changing
> > > >>> > > the
> > > >>> > > > > > format
> > > >>> > > > > > > > of
> > > >>> > > > > > > > >> > > > > > > > > > TransactionLogValue.
> > > >>> > > > > > > > >> > > > > > > > > > > A
> > > >>> > > > > > > > >> > > > > > > > > > > > > few
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > comments related
> to
> > > >>> that.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100. Just to be
> > clear.
> > > >>> The
> > > >>> > > > > overflow
> > > >>> > > > > > > case
> > > >>> > > > > > > > >> > (i.e.
> > > >>> > > > > > > > >> > > > > when a
> > > >>> > > > > > > > >> > > > > > > new
> > > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > generated) is when
> > the
> > > >>> > current
> > > >>> > > > > epoch
> > > >>> > > > > > > > >> equals
> > > >>> > > > > > > > >> > to
> > > >>> > > > > > > > >> > > > max
> > > >>> > > > > > > > >> > > > > -
> > > >>> > > > > > > > >> > > > > > 1
> > > >>> > > > > > > > >> > > > > > > > and
> > > >>> > > > > > > > >> > > > > > > > > > not
> > > >>> > > > > > > > >> > > > > > > > > > > > max?
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101. For the "not
> > > epoch
> > > >>> > > > overflow"
> > > >>> > > > > > > case,
> > > >>> > > > > > > > we
> > > >>> > > > > > > > >> > > write
> > > >>> > > > > > > > >> > > > > the
> > > >>> > > > > > > > >> > > > > > > > > previous
> > > >>> > > > > > > > >> > > > > > > > > > > ID
> > > >>> > > > > > > > >> > > > > > > > > > > > in
> > > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > tagged field in
> the
> > > >>> complete
> > > >>> > > > > phase.
> > > >>> > > > > > Do
> > > >>> > > > > > > > we
> > > >>> > > > > > > > >> > need
> > > >>> > > > > > > > >> > > to
> > > >>> > > > > > > > >> > > > > do
> > > >>> > > > > > > > >> > > > > > > that
> > > >>> > > > > > > > >> > > > > > > > > > since
> > > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > doesn't change in
> > this
> > > >>> case?
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102. It seems that
> > the
> > > >>> > meaning
> > > >>> > > > for
> > > >>> > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > ProducerId/ProducerEpoch
> > > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> TransactionLogValue
> > > >>> changes
> > > >>> > > > > > depending
> > > >>> > > > > > > on
> > > >>> > > > > > > > >> the
> > > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > > >>> > > > > > > > >> > > > > > > > > > > > > When
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> > TransactionStatus
> > > is
> > > >>> > > > ongoing,
> > > >>> > > > > > they
> > > >>> > > > > > > > >> > > represent
> > > >>> > > > > > > > >> > > > > the
> > > >>> > > > > > > > >> > > > > > > > > current
> > > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the current
> > > >>> ProducerEpoch.
> > > >>> > > When
> > > >>> > > > > the
> > > >>> > > > > > > > >> > > > > TransactionStatus
> > > >>> > > > > > > > >> > > > > > > is
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > >>> PrepareCommit/PrepareAbort,
> > > >>> > > they
> > > >>> > > > > > > > represent
> > > >>> > > > > > > > >> > the
> > > >>> > > > > > > > >> > > > > > current
> > > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > > >>> > > > > > > > >> > > > > > > > > > > > > and
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next
> ProducerEpoch.
> > > >>> When the
> > > >>> > > > > > > > >> > TransactionStatus
> > > >>> > > > > > > > >> > > is
> > > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > > >>> > > > > > > > >> > > > > > > > > > > > > they
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > further depend on
> > > >>> whether
> > > >>> > the
> > > >>> > > > > epoch
> > > >>> > > > > > > > >> overflows
> > > >>> > > > > > > > >> > > or
> > > >>> > > > > > > > >> > > > > not.
> > > >>> > > > > > > > >> > > > > > > If
> > > >>> > > > > > > > >> > > > > > > > > > there
> > > >>> > > > > > > > >> > > > > > > > > > > is
> > > >>> > > > > > > > >> > > > > > > > > > > > > no
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > overflow, they
> > > represent
> > > >>> > the
> > > >>> > > > > > current
> > > >>> > > > > > > > >> > > ProducerId
> > > >>> > > > > > > > >> > > > > and
> > > >>> > > > > > > > >> > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > next
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > ProducerEpoch
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > (max). Otherwise,
> > they
> > > >>> > > represent
> > > >>> > > > > the
> > > >>> > > > > > > > newly
> > > >>> > > > > > > > >> > > > > generated
> > > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > ProducerEpoch of
> 0.
> > Is
> > > >>> that
> > > >>> > > > right?
> > > >>> > > > > > > This
> > > >>> > > > > > > > >> seems
> > > >>> > > > > > > > >> > > not
> > > >>> > > > > > > > >> > > > > > easy
> > > >>> > > > > > > > >> > > > > > > to
> > > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we provide some
> > > examples
> > > >>> > like
> > > >>> > > > what
> > > >>> > > > > > > Artem
> > > >>> > > > > > > > >> has
> > > >>> > > > > > > > >> > > done
> > > >>> > > > > > > > >> > > > > in
> > > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > > >>> > > > > > > > >> > > > > > > > > > > > Have
> > > >>> > > > > > > > >> > > > > > > > > > > > > we
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > considered a
> simpler
> > > >>> design
> > > >>> > > > where
> > > >>> > > > > > > > >> > > > > > > > ProducerId/ProducerEpoch
> > > >>> > > > > > > > >> > > > > > > > > > > always
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > represent
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the same value
> (e.g.
> > > >>> for the
> > > >>> > > > > current
> > > >>> > > > > > > > >> > > transaction)
> > > >>> > > > > > > > >> > > > > > > > > independent
> > > >>> > > > > > > > >> > > > > > > > > > > of
> > > >>> > > > > > > > >> > > > > > > > > > > > > the
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > TransactionStatus
> > and
> > > >>> epoch
> > > >>> > > > > > overflow?
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103. It's not
> clear
> > to
> > > >>> me
> > > >>> > why
> > > >>> > > we
> > > >>> > > > > > need
> > > >>> > > > > > > 3
> > > >>> > > > > > > > >> > fields:
> > > >>> > > > > > > > >> > > > > > > > ProducerId,
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > PrevProducerId,
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > NextProducerId.
> > Could
> > > we
> > > >>> > just
> > > >>> > > > have
> > > >>> > > > > > > > >> ProducerId
> > > >>> > > > > > > > >> > > and
> > > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104. For
> > > >>> > > WriteTxnMarkerRequests,
> > > >>> > > > > if
> > > >>> > > > > > > the
> > > >>> > > > > > > > >> > > producer
> > > >>> > > > > > > > >> > > > > > epoch
> > > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we set the
> > producerId
> > > >>> and
> > > >>> > the
> > > >>> > > > > > > > >> producerEpoch?
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Thanks,
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > > >
> > > >>> > > > > > > > >> > > > > > > >
> > > >>> > > > > > > > >> > > > > > >
> > > >>> > > > > > > > >> > > > > >
> > > >>> > > > > > > > >> > > > >
> > > >>> > > > > > > > >> > > >
> > > >>> > > > > > > > >> > >
> > > >>> > > > > > > > >> >
> > > >>> > > > > > > > >>
> > > >>> > > > > > > > >
> > > >>> > > > > > > >
> > > >>> > > > > > >
> > > >>> > > > > >
> > > >>> > > > >
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > >
> >
>

Re: [DISCUSS] KIP-890 Server Side Defense

[Justine Olshan <jo...@confluent.io.INVALID>]

Hey Jun,

No worries. Work on this KIP has been blocked for a bit anyways -- catching
up and rereading what I wrote :)

120. ClientTransactionProtocolVersion is the transaction version as defined
by the highest transaction version (feature version value) supported by the
client and the server. This works by the broker sending an
ApiVersionsRequest to the client with the finalized version. Assuming
kip-890 part 2 is enabled by transaction version Y, if this request
contains finalized version Y and the client has the logic to set this
field, it will set Y. If the server has Y - 1 (kip 890 part 2 not enable)
the client will send Y - 1, even though the client has the ability to
support kip-890 part 2.

121. You are correct that this is not needed. However, currently that field
is already being set in memory -- just not written to disk. I think it is
ok to write it to disk though. Let me know if you think otherwise.

Justine

On Wed, Jul 10, 2024 at 2:16 PM Jun Rao <ju...@confluent.io.invalid> wrote:

> Hi, Justine,
>
> Thanks for the update and sorry for the late reply.
>
> 120. I am wondering what value is used for
> ClientTransactionProtocolVersion. Is it the version of the EndTxnRequest?
>
> 121. Earlier, you made the change to set lastProducerId in PREPARE to
> indicate that the marker is written for the new client. With the new
> ClientTransactionProtocolVersion field, it seems this is no longer
> necessary.
>
> Jun
>
> On Thu, Mar 28, 2024 at 2:41 PM Justine Olshan
> <jo...@confluent.io.invalid>
> wrote:
>
> > Hi there -- another update!
> >
> > When looking into the implementation for the safe epoch bumps I realized
> > that we are already populating previousProducerID in memory as part of
> > KIP-360.
> > If we are to start using flexible fields, it is better to always use this
> > information and have an explicit (tagged) field to indicate whether the
> > client supports KIP-890 part 2.
> >
> > I've included the extra field and how it is set in the KIP. I've also
> > updated the KIP to explain that we will be setting the tagged fields when
> > they are available for all transitions.
> >
> > Finally, I added clearer text about the transaction protocol versions
> > included with this KIP. 1 for flexible transaction state records and 2
> for
> > KIP-890 part 2 enablement.
> >
> > Justine
> >
> > On Mon, Mar 18, 2024 at 6:39 PM Justine Olshan <jo...@confluent.io>
> > wrote:
> >
> > > Hey there -- small update to the KIP,
> > >
> > > The KIP mentioned introducing ABORTABLE_ERROR and bumping
> TxnOffsetCommit
> > > and Produce requests. I've changed the name in the KIP to
> > > ABORTABLE_TRANSACTION and the corresponding exception
> > > AbortableTransactionException to match the pattern we had for other
> > errors.
> > > I also mentioned bumping all 6 transactional APIs so we can future
> > > proof/support the error on the client going forward. If a future change
> > > wants to have an error scenario that requires us to abort the
> > transaction,
> > > we can rely on the 3.8+ clients to support it. We ran into issues
> finding
> > > good/generic error codes that older clients could support while working
> > on
> > > this KIP, so this should help in the future.
> > >
> > > The features discussion is still ongoing in KIP-1022. Will update again
> > > here when that concludes.
> > >
> > > Justine
> > >
> > > On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan <jo...@confluent.io>
> > > wrote:
> > >
> > >> I don't think AddPartitions is a good example since we currenly don't
> > >> gate the version on TV or MV. (We only set a different flag depending
> on
> > >> the TV)
> > >>
> > >> Even if we did want to gate it on TV, I think the idea is to move away
> > >> from MV gating inter broker protocols. Ideally we can get to a state
> > where
> > >> MV is just used for metadata changes.
> > >>
> > >> I think some of this discussion might fit more with the feature
> version
> > >> KIP, so I can try to open that up soon. Until we settle that, some of
> > the
> > >> work in KIP-890 is blocked.
> > >>
> > >> Justine
> > >>
> > >> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao <ju...@confluent.io.invalid>
> > wrote:
> > >>
> > >>> Hi, Justine,
> > >>>
> > >>> Thanks for the reply.
> > >>>
> > >>> Since AddPartitions is an inter broker request, will its version be
> > gated
> > >>> only by TV or other features like MV too? For example, if we need to
> > >>> change
> > >>> the protocol for AddPartitions for reasons other than txn
> verification
> > in
> > >>> the future, will the new version be gated by a new MV? If so, does
> > >>> downgrading a TV imply potential downgrade of MV too?
> > >>>
> > >>> Jun
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
> > >>> <jo...@confluent.io.invalid>
> > >>> wrote:
> > >>>
> > >>> > One TV gates the flexible feature version (no rpcs involved, only
> the
> > >>> > transactional records that should only be gated by TV)
> > >>> > Another TV gates the ability to turn on kip-890 part 2. This would
> > >>> gate the
> > >>> > version of Produce and EndTxn (likely only used by transactions),
> and
> > >>> > specifies a flag in AddPartitionsToTxn though the version is
> already
> > >>> used
> > >>> > without TV.
> > >>> >
> > >>> > I think the only concern is the Produce request and we could
> consider
> > >>> work
> > >>> > arounds similar to the AddPartitionsToTxn call.
> > >>> >
> > >>> > Justine
> > >>> >
> > >>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao <ju...@confluent.io.invalid>
> > >>> wrote:
> > >>> >
> > >>> > > Hi, Justine,
> > >>> > >
> > >>> > > Which PRC/record protocols will TV guard? Going forward, will
> those
> > >>> > > PRC/record protocols only be guarded by TV and not by other
> > features
> > >>> like
> > >>> > > MV?
> > >>> > >
> > >>> > > Thanks,
> > >>> > >
> > >>> > > Jun
> > >>> > >
> > >>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
> > >>> > <jolshan@confluent.io.invalid
> > >>> > > >
> > >>> > > wrote:
> > >>> > >
> > >>> > > > Hi Jun,
> > >>> > > >
> > >>> > > > Sorry I think I misunderstood your question or answered
> > >>> incorrectly.
> > >>> > The
> > >>> > > TV
> > >>> > > > version should ideally be fully independent from MV.
> > >>> > > > At least for the changes I proposed, TV should not affect MV
> and
> > MV
> > >>> > > should
> > >>> > > > not affect TV/
> > >>> > > >
> > >>> > > > I think if we downgrade TV, only that feature should downgrade.
> > >>> > Likewise
> > >>> > > > the same with MV. The finalizedFeatures should just reflect the
> > >>> feature
> > >>> > > > downgrade we made.
> > >>> > > >
> > >>> > > > I also plan to write a new KIP for managing the disk format and
> > >>> upgrade
> > >>> > > > tool as we will need new flags to support these features. That
> > >>> should
> > >>> > > help
> > >>> > > > clarify some things.
> > >>> > > >
> > >>> > > > Justine
> > >>> > > >
> > >>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao
> <jun@confluent.io.invalid
> > >
> > >>> > > wrote:
> > >>> > > >
> > >>> > > > > Hi, Justine,
> > >>> > > > >
> > >>> > > > > Thanks for the reply.
> > >>> > > > >
> > >>> > > > > So, if we downgrade TV, we could implicitly downgrade another
> > >>> feature
> > >>> > > > (say
> > >>> > > > > MV) that has dependency (e.g. RPC). What would we return for
> > >>> > > > > FinalizedFeatures for MV in ApiVersionsResponse in that case?
> > >>> > > > >
> > >>> > > > > Thanks,
> > >>> > > > >
> > >>> > > > > Jun
> > >>> > > > >
> > >>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
> > >>> > > > <jolshan@confluent.io.invalid
> > >>> > > > > >
> > >>> > > > > wrote:
> > >>> > > > >
> > >>> > > > > > Hey Jun,
> > >>> > > > > >
> > >>> > > > > > Yes, the idea is that if we downgrade TV (transaction
> > version)
> > >>> we
> > >>> > > will
> > >>> > > > > stop
> > >>> > > > > > using the add partitions to txn optimization and stop
> writing
> > >>> the
> > >>> > > > > flexible
> > >>> > > > > > feature version of the log.
> > >>> > > > > > In the compatibility section I included some explanations
> on
> > >>> how
> > >>> > this
> > >>> > > > is
> > >>> > > > > > done.
> > >>> > > > > >
> > >>> > > > > > Thanks,
> > >>> > > > > > Justine
> > >>> > > > > >
> > >>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
> > >>> <ju...@confluent.io.invalid>
> > >>> > > > > wrote:
> > >>> > > > > >
> > >>> > > > > > > Hi, Justine,
> > >>> > > > > > >
> > >>> > > > > > > Thanks for the update.
> > >>> > > > > > >
> > >>> > > > > > > If we ever downgrade the transaction feature, any feature
> > >>> > depending
> > >>> > > > on
> > >>> > > > > > > changes on top of those RPC/record
> > >>> > > > > > > (AddPartitionsToTxnRequest/TransactionLogValue) changes
> > made
> > >>> in
> > >>> > > > KIP-890
> > >>> > > > > > > will be automatically downgraded too?
> > >>> > > > > > >
> > >>> > > > > > > Jun
> > >>> > > > > > >
> > >>> > > > > > > On Tue, Jan 30, 2024 at 3:32 PM Justine Olshan
> > >>> > > > > > > <jo...@confluent.io.invalid>
> > >>> > > > > > > wrote:
> > >>> > > > > > >
> > >>> > > > > > > > Hey Jun,
> > >>> > > > > > > >
> > >>> > > > > > > > I wanted to get back to you about your questions about
> > >>> MV/IBP.
> > >>> > > > > > > >
> > >>> > > > > > > > Looking at the options, I think it makes the most sense
> > to
> > >>> > > create a
> > >>> > > > > > > > separate feature for transactions and use that to
> version
> > >>> gate
> > >>> > > the
> > >>> > > > > > > features
> > >>> > > > > > > > we need to version gate (flexible transactional state
> > >>> records
> > >>> > and
> > >>> > > > > using
> > >>> > > > > > > the
> > >>> > > > > > > > new protocol)
> > >>> > > > > > > > I've updated the KIP to include this change. Hopefully
> > >>> that's
> > >>> > > > > > everything
> > >>> > > > > > > we
> > >>> > > > > > > > need for this KIP :)
> > >>> > > > > > > >
> > >>> > > > > > > > Justine
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > > On Mon, Jan 22, 2024 at 3:17 PM Justine Olshan <
> > >>> > > > jolshan@confluent.io
> > >>> > > > > >
> > >>> > > > > > > > wrote:
> > >>> > > > > > > >
> > >>> > > > > > > > > Thanks Jun,
> > >>> > > > > > > > >
> > >>> > > > > > > > > I will update the KIP with the prev field for prepare
> > as
> > >>> > well.
> > >>> > > > > > > > >
> > >>> > > > > > > > > PREPARE
> > >>> > > > > > > > > producerId: x
> > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > >>> > > > > > > > > nextProducerId (tagged field): empty or z if y will
> > >>> overflow
> > >>> > > > > > > > > producerEpoch: y + 1
> > >>> > > > > > > > >
> > >>> > > > > > > > > COMPLETE
> > >>> > > > > > > > > producerId: x or z if y overflowed
> > >>> > > > > > > > > previous/lastProducerId (tagged field): x
> > >>> > > > > > > > > nextProducerId (tagged field): empty
> > >>> > > > > > > > > producerEpoch: y + 1 or 0 if we overflowed
> > >>> > > > > > > > >
> > >>> > > > > > > > > Thanks again,
> > >>> > > > > > > > > Justine
> > >>> > > > > > > > >
> > >>> > > > > > > > > On Mon, Jan 22, 2024 at 3:15 PM Jun Rao
> > >>> > > <jun@confluent.io.invalid
> > >>> > > > >
> > >>> > > > > > > > wrote:
> > >>> > > > > > > > >
> > >>> > > > > > > > >> Hi, Justine,
> > >>> > > > > > > > >>
> > >>> > > > > > > > >> 101.3 Thanks for the explanation.
> > >>> > > > > > > > >> (1) My point was that the coordinator could fail
> right
> > >>> after
> > >>> > > > > writing
> > >>> > > > > > > the
> > >>> > > > > > > > >> prepare marker. When the new txn coordinator
> generates
> > >>> the
> > >>> > > > > complete
> > >>> > > > > > > > marker
> > >>> > > > > > > > >> after the failover, it needs some field from the
> > prepare
> > >>> > > marker
> > >>> > > > to
> > >>> > > > > > > > >> determine whether it's written by the new client.
> > >>> > > > > > > > >>
> > >>> > > > > > > > >> (2) The changing of the behavior sounds good to me.
> We
> > >>> only
> > >>> > > want
> > >>> > > > > to
> > >>> > > > > > > > return
> > >>> > > > > > > > >> success if the prepare state is written by the new
> > >>> client.
> > >>> > So,
> > >>> > > > in
> > >>> > > > > > the
> > >>> > > > > > > > >> non-overflow case, it seems that we also need sth in
> > the
> > >>> > > prepare
> > >>> > > > > > > marker
> > >>> > > > > > > > to
> > >>> > > > > > > > >> tell us whether it's written by the new client.
> > >>> > > > > > > > >>
> > >>> > > > > > > > >> 112. Thanks for the explanation. That sounds good to
> > me.
> > >>> > > > > > > > >>
> > >>> > > > > > > > >> Jun
> > >>> > > > > > > > >>
> > >>> > > > > > > > >> On Mon, Jan 22, 2024 at 11:32 AM Justine Olshan
> > >>> > > > > > > > >> <jo...@confluent.io.invalid> wrote:
> > >>> > > > > > > > >>
> > >>> > > > > > > > >> > 101.3 I realized that I actually have two
> questions.
> > >>> > > > > > > > >> > > (1) In the non-overflow case, we need to write
> the
> > >>> > > previous
> > >>> > > > > > > produce
> > >>> > > > > > > > Id
> > >>> > > > > > > > >> > tagged field in the end maker so that we know if
> the
> > >>> > marker
> > >>> > > is
> > >>> > > > > > from
> > >>> > > > > > > > the
> > >>> > > > > > > > >> new
> > >>> > > > > > > > >> > client. Since the end maker is derived from the
> > >>> prepare
> > >>> > > > marker,
> > >>> > > > > > > should
> > >>> > > > > > > > >> we
> > >>> > > > > > > > >> > write the previous produce Id in the prepare
> marker
> > >>> field
> > >>> > > too?
> > >>> > > > > > > > >> Otherwise,
> > >>> > > > > > > > >> > we will lose this information when deriving the
> end
> > >>> > marker.
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > The "previous" producer ID is in the normal
> producer
> > >>> ID
> > >>> > > field.
> > >>> > > > > So
> > >>> > > > > > > yes,
> > >>> > > > > > > > >> we
> > >>> > > > > > > > >> > need it in prepare and that was always the plan.
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > Maybe it is a bit unclear so I will enumerate the
> > >>> fields
> > >>> > and
> > >>> > > > add
> > >>> > > > > > > them
> > >>> > > > > > > > to
> > >>> > > > > > > > >> > the KIP if that helps.
> > >>> > > > > > > > >> > Say we have producer ID x and epoch y. When we
> > >>> overflow
> > >>> > > epoch
> > >>> > > > y
> > >>> > > > > we
> > >>> > > > > > > get
> > >>> > > > > > > > >> > producer ID Z.
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > PREPARE
> > >>> > > > > > > > >> > producerId: x
> > >>> > > > > > > > >> > previous/lastProducerId (tagged field): empty
> > >>> > > > > > > > >> > nextProducerId (tagged field): empty or z if y
> will
> > >>> > overflow
> > >>> > > > > > > > >> > producerEpoch: y + 1
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > COMPLETE
> > >>> > > > > > > > >> > producerId: x or z if y overflowed
> > >>> > > > > > > > >> > previous/lastProducerId (tagged field): x
> > >>> > > > > > > > >> > nextProducerId (tagged field): empty
> > >>> > > > > > > > >> > producerEpoch: y + 1 or 0 if we overflowed
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > (2) In the prepare phase, if we retry and see
> epoch
> > -
> > >>> 1 +
> > >>> > ID
> > >>> > > > in
> > >>> > > > > > last
> > >>> > > > > > > > >> seen
> > >>> > > > > > > > >> > fields and are issuing the same command (ie commit
> > not
> > >>> > > abort),
> > >>> > > > > we
> > >>> > > > > > > > return
> > >>> > > > > > > > >> > success. The logic before KIP-890 seems to return
> > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > >>> > > > > > > > >> > in this case. Are we intentionally making this
> > change?
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > Hmm -- we would fence the producer if the epoch is
> > >>> bumped
> > >>> > > and
> > >>> > > > we
> > >>> > > > > > > get a
> > >>> > > > > > > > >> > lower epoch. Yes -- we are intentionally adding
> this
> > >>> to
> > >>> > > > prevent
> > >>> > > > > > > > fencing.
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > 112. We already merged the code that adds the
> > >>> VerifyOnly
> > >>> > > field
> > >>> > > > > in
> > >>> > > > > > > > >> > AddPartitionsToTxnRequest, which is an inter
> broker
> > >>> > request.
> > >>> > > > It
> > >>> > > > > > > seems
> > >>> > > > > > > > >> that
> > >>> > > > > > > > >> > we didn't bump up the IBP for that. Do you know
> why?
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > We no longer need IBP for all interbroker requests
> > as
> > >>> > > > > ApiVersions
> > >>> > > > > > > > should
> > >>> > > > > > > > >> > correctly gate versioning.
> > >>> > > > > > > > >> > We also handle unsupported version errors
> correctly
> > >>> if we
> > >>> > > > > receive
> > >>> > > > > > > them
> > >>> > > > > > > > >> in
> > >>> > > > > > > > >> > edge cases like upgrades/downgrades.
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > Justine
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > On Mon, Jan 22, 2024 at 11:00 AM Jun Rao
> > >>> > > > > <jun@confluent.io.invalid
> > >>> > > > > > >
> > >>> > > > > > > > >> wrote:
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >> > > Hi, Justine,
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > Thanks for the reply.
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > 101.3 I realized that I actually have two
> > questions.
> > >>> > > > > > > > >> > > (1) In the non-overflow case, we need to write
> the
> > >>> > > previous
> > >>> > > > > > > produce
> > >>> > > > > > > > Id
> > >>> > > > > > > > >> > > tagged field in the end maker so that we know if
> > the
> > >>> > > marker
> > >>> > > > is
> > >>> > > > > > > from
> > >>> > > > > > > > >> the
> > >>> > > > > > > > >> > new
> > >>> > > > > > > > >> > > client. Since the end maker is derived from the
> > >>> prepare
> > >>> > > > > marker,
> > >>> > > > > > > > >> should we
> > >>> > > > > > > > >> > > write the previous produce Id in the prepare
> > marker
> > >>> > field
> > >>> > > > too?
> > >>> > > > > > > > >> Otherwise,
> > >>> > > > > > > > >> > > we will lose this information when deriving the
> > end
> > >>> > > marker.
> > >>> > > > > > > > >> > > (2) In the prepare phase, if we retry and see
> > epoch
> > >>> - 1
> > >>> > +
> > >>> > > ID
> > >>> > > > > in
> > >>> > > > > > > last
> > >>> > > > > > > > >> seen
> > >>> > > > > > > > >> > > fields and are issuing the same command (ie
> commit
> > >>> not
> > >>> > > > abort),
> > >>> > > > > > we
> > >>> > > > > > > > >> return
> > >>> > > > > > > > >> > > success. The logic before KIP-890 seems to
> return
> > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > >>> > > > > > > > >> > > in this case. Are we intentionally making this
> > >>> change?
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > 112. We already merged the code that adds the
> > >>> VerifyOnly
> > >>> > > > field
> > >>> > > > > > in
> > >>> > > > > > > > >> > > AddPartitionsToTxnRequest, which is an inter
> > broker
> > >>> > > request.
> > >>> > > > > It
> > >>> > > > > > > > seems
> > >>> > > > > > > > >> > that
> > >>> > > > > > > > >> > > we didn't bump up the IBP for that. Do you know
> > why?
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > Jun
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > On Fri, Jan 19, 2024 at 4:50 PM Justine Olshan
> > >>> > > > > > > > >> > > <jo...@confluent.io.invalid>
> > >>> > > > > > > > >> > > wrote:
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> > > > Hi Jun,
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > 101.3 I can change "last seen" to "current
> > >>> producer id
> > >>> > > and
> > >>> > > > > > > epoch"
> > >>> > > > > > > > if
> > >>> > > > > > > > >> > that
> > >>> > > > > > > > >> > > > was the part that was confusing
> > >>> > > > > > > > >> > > > 110 I can mention this
> > >>> > > > > > > > >> > > > 111 I can do that
> > >>> > > > > > > > >> > > > 112 We still need it. But I am still
> finalizing
> > >>> the
> > >>> > > > design.
> > >>> > > > > I
> > >>> > > > > > > will
> > >>> > > > > > > > >> > update
> > >>> > > > > > > > >> > > > the KIP once I get the information finalized.
> > >>> Sorry
> > >>> > for
> > >>> > > > the
> > >>> > > > > > > > delays.
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > Justine
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > On Fri, Jan 19, 2024 at 10:50 AM Jun Rao
> > >>> > > > > > > <jun@confluent.io.invalid
> > >>> > > > > > > > >
> > >>> > > > > > > > >> > > wrote:
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > Hi, Justine,
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > > > Thanks for the reply.
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > > > 101.3 In the non-overflow case, the previous
> > ID
> > >>> is
> > >>> > the
> > >>> > > > > same
> > >>> > > > > > as
> > >>> > > > > > > > the
> > >>> > > > > > > > >> > > > produce
> > >>> > > > > > > > >> > > > > ID for the complete marker too, but we set
> the
> > >>> > > previous
> > >>> > > > ID
> > >>> > > > > > in
> > >>> > > > > > > > the
> > >>> > > > > > > > >> > > > complete
> > >>> > > > > > > > >> > > > > marker. Earlier you mentioned that this is
> to
> > >>> know
> > >>> > > that
> > >>> > > > > the
> > >>> > > > > > > > >> marker is
> > >>> > > > > > > > >> > > > > written by the new client so that we could
> > >>> return
> > >>> > > > success
> > >>> > > > > on
> > >>> > > > > > > > >> retried
> > >>> > > > > > > > >> > > > > endMarker requests. I was trying to
> understand
> > >>> why
> > >>> > > this
> > >>> > > > is
> > >>> > > > > > not
> > >>> > > > > > > > >> needed
> > >>> > > > > > > > >> > > for
> > >>> > > > > > > > >> > > > > the prepare marker since retry can happen in
> > the
> > >>> > > prepare
> > >>> > > > > > state
> > >>> > > > > > > > >> too.
> > >>> > > > > > > > >> > Is
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > reason that in the prepare state, we return
> > >>> > > > > > > > >> CONCURRENT_TRANSACTIONS
> > >>> > > > > > > > >> > > > instead
> > >>> > > > > > > > >> > > > > of success on retried endMaker requests? If
> > so,
> > >>> > should
> > >>> > > > we
> > >>> > > > > > > change
> > >>> > > > > > > > >> "If
> > >>> > > > > > > > >> > we
> > >>> > > > > > > > >> > > > > retry and see epoch - 1 + ID in last seen
> > >>> fields and
> > >>> > > are
> > >>> > > > > > > issuing
> > >>> > > > > > > > >> the
> > >>> > > > > > > > >> > > same
> > >>> > > > > > > > >> > > > > command (ie commit not abort) we can return
> > >>> (with
> > >>> > the
> > >>> > > > new
> > >>> > > > > > > > epoch)"
> > >>> > > > > > > > >> > > > > accordingly?
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > > > 110. Yes, without this KIP, a delayed
> endMaker
> > >>> > request
> > >>> > > > > > carries
> > >>> > > > > > > > the
> > >>> > > > > > > > >> > same
> > >>> > > > > > > > >> > > > > epoch and won't be fenced. This can
> > >>> commit/abort a
> > >>> > > > future
> > >>> > > > > > > > >> transaction
> > >>> > > > > > > > >> > > > > unexpectedly. I am not sure if we have seen
> > >>> this in
> > >>> > > > > practice
> > >>> > > > > > > > >> though.
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > > > 111. Sounds good. It would be useful to make
> > it
> > >>> > clear
> > >>> > > > that
> > >>> > > > > > we
> > >>> > > > > > > > can
> > >>> > > > > > > > >> now
> > >>> > > > > > > > >> > > > > populate the lastSeen field from the log
> > >>> reliably.
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > > > 112. Yes, I was referring to
> > >>> > AddPartitionsToTxnRequest
> > >>> > > > > since
> > >>> > > > > > > > it's
> > >>> > > > > > > > >> > > called
> > >>> > > > > > > > >> > > > > across brokers and we are changing its
> schema.
> > >>> Are
> > >>> > you
> > >>> > > > > > saying
> > >>> > > > > > > we
> > >>> > > > > > > > >> > don't
> > >>> > > > > > > > >> > > > need
> > >>> > > > > > > > >> > > > > it any more? I thought that we already
> > >>> implemented
> > >>> > the
> > >>> > > > > > server
> > >>> > > > > > > > side
> > >>> > > > > > > > >> > > > > verification logic based on
> > >>> > AddPartitionsToTxnRequest
> > >>> > > > > across
> > >>> > > > > > > > >> brokers.
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > > > Jun
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > > > On Thu, Jan 18, 2024 at 5:05 PM Justine
> Olshan
> > >>> > > > > > > > >> > > > > <jo...@confluent.io.invalid>
> > >>> > > > > > > > >> > > > > wrote:
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > > > > Hey Jun,
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > 101.3 We don't set the previous ID in the
> > >>> Prepare
> > >>> > > > field
> > >>> > > > > > > since
> > >>> > > > > > > > we
> > >>> > > > > > > > >> > > don't
> > >>> > > > > > > > >> > > > > need
> > >>> > > > > > > > >> > > > > > it. It is the same producer ID as the main
> > >>> > producer
> > >>> > > ID
> > >>> > > > > > > field.
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > 110 Hmm -- maybe I need to reread your
> > message
> > >>> > about
> > >>> > > > > > delayed
> > >>> > > > > > > > >> > markers.
> > >>> > > > > > > > >> > > > If
> > >>> > > > > > > > >> > > > > we
> > >>> > > > > > > > >> > > > > > receive a delayed endTxn marker after the
> > >>> > > transaction
> > >>> > > > is
> > >>> > > > > > > > already
> > >>> > > > > > > > >> > > > > complete?
> > >>> > > > > > > > >> > > > > > So we will commit the next transaction
> early
> > >>> > without
> > >>> > > > the
> > >>> > > > > > > fixes
> > >>> > > > > > > > >> in
> > >>> > > > > > > > >> > > part
> > >>> > > > > > > > >> > > > 2?
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > 111 Yes -- this terminology was used in a
> > >>> previous
> > >>> > > KIP
> > >>> > > > > and
> > >>> > > > > > > > never
> > >>> > > > > > > > >> > > > > > implemented it in the log -- only in
> memory
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > 112 Hmm -- which interbroker protocol are
> > you
> > >>> > > > referring
> > >>> > > > > > to?
> > >>> > > > > > > I
> > >>> > > > > > > > am
> > >>> > > > > > > > >> > > > working
> > >>> > > > > > > > >> > > > > on
> > >>> > > > > > > > >> > > > > > the design for the work to remove the
> extra
> > >>> add
> > >>> > > > > partitions
> > >>> > > > > > > > call
> > >>> > > > > > > > >> > and I
> > >>> > > > > > > > >> > > > > right
> > >>> > > > > > > > >> > > > > > now the design bumps MV. I have yet to
> > update
> > >>> that
> > >>> > > > > section
> > >>> > > > > > > as
> > >>> > > > > > > > I
> > >>> > > > > > > > >> > > > finalize
> > >>> > > > > > > > >> > > > > > the design so please stay tuned. Was there
> > >>> > anything
> > >>> > > > else
> > >>> > > > > > you
> > >>> > > > > > > > >> > thought
> > >>> > > > > > > > >> > > > > needed
> > >>> > > > > > > > >> > > > > > MV bump?
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > Justine
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > On Thu, Jan 18, 2024 at 3:07 PM Jun Rao
> > >>> > > > > > > > >> <ju...@confluent.io.invalid>
> > >>> > > > > > > > >> > > > > wrote:
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > > Hi, Justine,
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > I don't see this create any issue. It
> just
> > >>> makes
> > >>> > > it
> > >>> > > > a
> > >>> > > > > > bit
> > >>> > > > > > > > >> hard to
> > >>> > > > > > > > >> > > > > explain
> > >>> > > > > > > > >> > > > > > > what this non-tagged produce id field
> > >>> means. We
> > >>> > > are
> > >>> > > > > > > > >> essentially
> > >>> > > > > > > > >> > > > trying
> > >>> > > > > > > > >> > > > > to
> > >>> > > > > > > > >> > > > > > > combine two actions (completing a txn
> and
> > >>> init a
> > >>> > > new
> > >>> > > > > > > produce
> > >>> > > > > > > > >> Id)
> > >>> > > > > > > > >> > > in a
> > >>> > > > > > > > >> > > > > > > single record. But, this may be fine
> too.
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > A few other follow up comments.
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > 101.3 I guess the reason that we only
> set
> > >>> the
> > >>> > > > previous
> > >>> > > > > > > > >> produce id
> > >>> > > > > > > > >> > > > > tagged
> > >>> > > > > > > > >> > > > > > > field in the complete marker, but not in
> > the
> > >>> > > prepare
> > >>> > > > > > > marker,
> > >>> > > > > > > > >> is
> > >>> > > > > > > > >> > > that
> > >>> > > > > > > > >> > > > in
> > >>> > > > > > > > >> > > > > > the
> > >>> > > > > > > > >> > > > > > > prepare state, we always return
> > >>> > > > > CONCURRENT_TRANSACTIONS
> > >>> > > > > > on
> > >>> > > > > > > > >> > retried
> > >>> > > > > > > > >> > > > > > endMaker
> > >>> > > > > > > > >> > > > > > > requests?
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > 110. "I believe your second point is
> > >>> mentioned
> > >>> > in
> > >>> > > > the
> > >>> > > > > > > KIP. I
> > >>> > > > > > > > >> can
> > >>> > > > > > > > >> > > add
> > >>> > > > > > > > >> > > > > more
> > >>> > > > > > > > >> > > > > > > text on
> > >>> > > > > > > > >> > > > > > > this if it is helpful.
> > >>> > > > > > > > >> > > > > > > > The delayed message case can also
> > violate
> > >>> EOS
> > >>> > if
> > >>> > > > the
> > >>> > > > > > > > delayed
> > >>> > > > > > > > >> > > > message
> > >>> > > > > > > > >> > > > > > > comes in after the next
> addPartitionsToTxn
> > >>> > request
> > >>> > > > > comes
> > >>> > > > > > > in.
> > >>> > > > > > > > >> > > > > Effectively
> > >>> > > > > > > > >> > > > > > we
> > >>> > > > > > > > >> > > > > > > may see a message from a previous
> > (aborted)
> > >>> > > > > transaction
> > >>> > > > > > > > become
> > >>> > > > > > > > >> > part
> > >>> > > > > > > > >> > > > of
> > >>> > > > > > > > >> > > > > > the
> > >>> > > > > > > > >> > > > > > > next transaction."
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > The above is the case when a delayed
> > >>> message is
> > >>> > > > > appended
> > >>> > > > > > > to
> > >>> > > > > > > > >> the
> > >>> > > > > > > > >> > > data
> > >>> > > > > > > > >> > > > > > > partition. What I mentioned is a
> slightly
> > >>> > > different
> > >>> > > > > case
> > >>> > > > > > > > when
> > >>> > > > > > > > >> a
> > >>> > > > > > > > >> > > > delayed
> > >>> > > > > > > > >> > > > > > > marker is appended to the transaction
> log
> > >>> > > partition.
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > 111. The KIP says "Once we move past the
> > >>> Prepare
> > >>> > > and
> > >>> > > > > > > > Complete
> > >>> > > > > > > > >> > > states,
> > >>> > > > > > > > >> > > > > we
> > >>> > > > > > > > >> > > > > > > don’t need to worry about lastSeen
> fields
> > >>> and
> > >>> > > clear
> > >>> > > > > > them,
> > >>> > > > > > > > just
> > >>> > > > > > > > >> > > handle
> > >>> > > > > > > > >> > > > > > state
> > >>> > > > > > > > >> > > > > > > transitions as normal.". Is the lastSeen
> > >>> field
> > >>> > the
> > >>> > > > > same
> > >>> > > > > > as
> > >>> > > > > > > > the
> > >>> > > > > > > > >> > > > previous
> > >>> > > > > > > > >> > > > > > > Produce Id tagged field in
> > >>> TransactionLogValue?
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > 112. Since the kip changes the
> > inter-broker
> > >>> > > > protocol,
> > >>> > > > > > > should
> > >>> > > > > > > > >> we
> > >>> > > > > > > > >> > > bump
> > >>> > > > > > > > >> > > > up
> > >>> > > > > > > > >> > > > > > the
> > >>> > > > > > > > >> > > > > > > MV/IBP version? Is this feature only for
> > the
> > >>> > KRaft
> > >>> > > > > mode?
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > Thanks,
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > Jun
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > On Wed, Jan 17, 2024 at 11:13 AM Justine
> > >>> Olshan
> > >>> > > > > > > > >> > > > > > > <jo...@confluent.io.invalid> wrote:
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > > > > Hey Jun,
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > > I'm glad we are getting to convergence
> > on
> > >>> the
> > >>> > > > > design.
> > >>> > > > > > :)
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > > While I understand it seems a little
> > >>> "weird".
> > >>> > > I'm
> > >>> > > > > not
> > >>> > > > > > > sure
> > >>> > > > > > > > >> what
> > >>> > > > > > > > >> > > the
> > >>> > > > > > > > >> > > > > > > benefit
> > >>> > > > > > > > >> > > > > > > > of writing an extra record to the log.
> > >>> > > > > > > > >> > > > > > > > Is the concern a tool to describe
> > >>> transactions
> > >>> > > > won't
> > >>> > > > > > > work
> > >>> > > > > > > > >> (ie,
> > >>> > > > > > > > >> > > the
> > >>> > > > > > > > >> > > > > > > complete
> > >>> > > > > > > > >> > > > > > > > state is needed to calculate the time
> > >>> since
> > >>> > the
> > >>> > > > > > > > transaction
> > >>> > > > > > > > >> > > > > completed?)
> > >>> > > > > > > > >> > > > > > > > If we have a reason like this, it is
> > >>> enough to
> > >>> > > > > > convince
> > >>> > > > > > > me
> > >>> > > > > > > > >> we
> > >>> > > > > > > > >> > > need
> > >>> > > > > > > > >> > > > > such
> > >>> > > > > > > > >> > > > > > > an
> > >>> > > > > > > > >> > > > > > > > extra record. It seems like it would
> be
> > >>> > > replacing
> > >>> > > > > the
> > >>> > > > > > > > record
> > >>> > > > > > > > >> > > > written
> > >>> > > > > > > > >> > > > > on
> > >>> > > > > > > > >> > > > > > > > InitProducerId. Is this correct?
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > > Thanks,
> > >>> > > > > > > > >> > > > > > > > Justine
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > > On Tue, Jan 16, 2024 at 5:14 PM Jun
> Rao
> > >>> > > > > > > > >> > <jun@confluent.io.invalid
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > > > > > > wrote:
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > Hi, Justine,
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > Thanks for the explanation. I
> > >>> understand the
> > >>> > > > > > intention
> > >>> > > > > > > > >> now.
> > >>> > > > > > > > >> > In
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > > > > overflow
> > >>> > > > > > > > >> > > > > > > > > case, we set the non-tagged field to
> > >>> the old
> > >>> > > pid
> > >>> > > > > > (and
> > >>> > > > > > > > the
> > >>> > > > > > > > >> max
> > >>> > > > > > > > >> > > > > epoch)
> > >>> > > > > > > > >> > > > > > in
> > >>> > > > > > > > >> > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > prepare marker so that we could
> > >>> correctly
> > >>> > > write
> > >>> > > > > the
> > >>> > > > > > > > >> marker to
> > >>> > > > > > > > >> > > the
> > >>> > > > > > > > >> > > > > > data
> > >>> > > > > > > > >> > > > > > > > > partition if the broker downgrades.
> > When
> > >>> > > writing
> > >>> > > > > the
> > >>> > > > > > > > >> complete
> > >>> > > > > > > > >> > > > > marker,
> > >>> > > > > > > > >> > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > know the marker has already been
> > >>> written to
> > >>> > > the
> > >>> > > > > data
> > >>> > > > > > > > >> > partition.
> > >>> > > > > > > > >> > > > We
> > >>> > > > > > > > >> > > > > > set
> > >>> > > > > > > > >> > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > non-tagged field to the new pid to
> > avoid
> > >>> > > > > > > > >> > > > InvalidPidMappingException
> > >>> > > > > > > > >> > > > > > in
> > >>> > > > > > > > >> > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > client if the broker downgrades.
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > The above seems to work. It's just a
> > bit
> > >>> > > > > > inconsistent
> > >>> > > > > > > > for
> > >>> > > > > > > > >> a
> > >>> > > > > > > > >> > > > prepare
> > >>> > > > > > > > >> > > > > > > > marker
> > >>> > > > > > > > >> > > > > > > > > and a complete marker to use
> different
> > >>> pids
> > >>> > in
> > >>> > > > > this
> > >>> > > > > > > > >> special
> > >>> > > > > > > > >> > > case.
> > >>> > > > > > > > >> > > > > If
> > >>> > > > > > > > >> > > > > > we
> > >>> > > > > > > > >> > > > > > > > > downgrade with the complete marker,
> it
> > >>> seems
> > >>> > > > that
> > >>> > > > > we
> > >>> > > > > > > > will
> > >>> > > > > > > > >> > never
> > >>> > > > > > > > >> > > > be
> > >>> > > > > > > > >> > > > > > able
> > >>> > > > > > > > >> > > > > > > > to
> > >>> > > > > > > > >> > > > > > > > > write the complete marker with the
> old
> > >>> pid.
> > >>> > > Not
> > >>> > > > > sure
> > >>> > > > > > > if
> > >>> > > > > > > > it
> > >>> > > > > > > > >> > > causes
> > >>> > > > > > > > >> > > > > any
> > >>> > > > > > > > >> > > > > > > > > issue, but it seems a bit weird.
> > >>> Instead of
> > >>> > > > > writing
> > >>> > > > > > > the
> > >>> > > > > > > > >> > > complete
> > >>> > > > > > > > >> > > > > > marker
> > >>> > > > > > > > >> > > > > > > > > with the new pid, could we write two
> > >>> > records:
> > >>> > > a
> > >>> > > > > > > complete
> > >>> > > > > > > > >> > marker
> > >>> > > > > > > > >> > > > > with
> > >>> > > > > > > > >> > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > old pid followed by a
> > >>> TransactionLogValue
> > >>> > with
> > >>> > > > the
> > >>> > > > > > new
> > >>> > > > > > > > pid
> > >>> > > > > > > > >> > and
> > >>> > > > > > > > >> > > an
> > >>> > > > > > > > >> > > > > > empty
> > >>> > > > > > > > >> > > > > > > > > state? We could make the two records
> > in
> > >>> the
> > >>> > > same
> > >>> > > > > > batch
> > >>> > > > > > > > so
> > >>> > > > > > > > >> > that
> > >>> > > > > > > > >> > > > they
> > >>> > > > > > > > >> > > > > > > will
> > >>> > > > > > > > >> > > > > > > > be
> > >>> > > > > > > > >> > > > > > > > > added to the log atomically.
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > Thanks,
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > Jun
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > On Fri, Jan 12, 2024 at 5:40 PM
> > Justine
> > >>> > Olshan
> > >>> > > > > > > > >> > > > > > > > > <jo...@confluent.io.invalid>
> > >>> > > > > > > > >> > > > > > > > > wrote:
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > (1) the prepare marker is written,
> > >>> but the
> > >>> > > > > endTxn
> > >>> > > > > > > > >> response
> > >>> > > > > > > > >> > is
> > >>> > > > > > > > >> > > > not
> > >>> > > > > > > > >> > > > > > > > > received
> > >>> > > > > > > > >> > > > > > > > > > by the client when the server
> > >>> downgrades
> > >>> > > > > > > > >> > > > > > > > > > (2)  the prepare marker is
> written,
> > >>> the
> > >>> > > endTxn
> > >>> > > > > > > > response
> > >>> > > > > > > > >> is
> > >>> > > > > > > > >> > > > > received
> > >>> > > > > > > > >> > > > > > > by
> > >>> > > > > > > > >> > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > client when the server downgrades.
> > >>> > > > > > > > >> > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > I think I am still a little
> > confused.
> > >>> In
> > >>> > > both
> > >>> > > > of
> > >>> > > > > > > these
> > >>> > > > > > > > >> > cases,
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > > > > > > transaction log has the old
> producer
> > >>> ID.
> > >>> > We
> > >>> > > > > don't
> > >>> > > > > > > > write
> > >>> > > > > > > > >> the
> > >>> > > > > > > > >> > > new
> > >>> > > > > > > > >> > > > > > > > producer
> > >>> > > > > > > > >> > > > > > > > > ID
> > >>> > > > > > > > >> > > > > > > > > > in the prepare marker's non tagged
> > >>> fields.
> > >>> > > > > > > > >> > > > > > > > > > If the server downgrades now, it
> > would
> > >>> > read
> > >>> > > > the
> > >>> > > > > > > > records
> > >>> > > > > > > > >> not
> > >>> > > > > > > > >> > > in
> > >>> > > > > > > > >> > > > > > tagged
> > >>> > > > > > > > >> > > > > > > > > > fields and the complete marker
> will
> > >>> also
> > >>> > > have
> > >>> > > > > the
> > >>> > > > > > > old
> > >>> > > > > > > > >> > > producer
> > >>> > > > > > > > >> > > > > ID.
> > >>> > > > > > > > >> > > > > > > > > > (If we had used the new producer
> ID,
> > >>> we
> > >>> > > would
> > >>> > > > > not
> > >>> > > > > > > have
> > >>> > > > > > > > >> > > > > > transactional
> > >>> > > > > > > > >> > > > > > > > > > correctness since the producer id
> > >>> doesn't
> > >>> > > > match
> > >>> > > > > > the
> > >>> > > > > > > > >> > > transaction
> > >>> > > > > > > > >> > > > > and
> > >>> > > > > > > > >> > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > state would not be correct on the
> > data
> > >>> > > > > partition.)
> > >>> > > > > > > > >> > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > In the overflow case, I'd expect
> the
> > >>> > > following
> > >>> > > > > to
> > >>> > > > > > > > >> happen on
> > >>> > > > > > > > >> > > the
> > >>> > > > > > > > >> > > > > > > client
> > >>> > > > > > > > >> > > > > > > > > side
> > >>> > > > > > > > >> > > > > > > > > > Case 1  -- we retry EndTxn -- it
> is
> > >>> the
> > >>> > same
> > >>> > > > > > > producer
> > >>> > > > > > > > ID
> > >>> > > > > > > > >> > and
> > >>> > > > > > > > >> > > > > epoch
> > >>> > > > > > > > >> > > > > > -
> > >>> > > > > > > > >> > > > > > > 1
> > >>> > > > > > > > >> > > > > > > > > this
> > >>> > > > > > > > >> > > > > > > > > > would fence the producer
> > >>> > > > > > > > >> > > > > > > > > > Case 2 -- we don't retry EndTxn
> and
> > >>> use
> > >>> > the
> > >>> > > > new
> > >>> > > > > > > > >> producer id
> > >>> > > > > > > > >> > > > which
> > >>> > > > > > > > >> > > > > > > would
> > >>> > > > > > > > >> > > > > > > > > > result in
> InvalidPidMappingException
> > >>> > > > > > > > >> > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > Maybe we can have special handling
> > for
> > >>> > when
> > >>> > > a
> > >>> > > > > > server
> > >>> > > > > > > > >> > > > downgrades.
> > >>> > > > > > > > >> > > > > > When
> > >>> > > > > > > > >> > > > > > > > it
> > >>> > > > > > > > >> > > > > > > > > > reconnects we could get an API
> > version
> > >>> > > request
> > >>> > > > > > > showing
> > >>> > > > > > > > >> > > KIP-890
> > >>> > > > > > > > >> > > > > > part 2
> > >>> > > > > > > > >> > > > > > > > is
> > >>> > > > > > > > >> > > > > > > > > > not supported. In that case, we
> can
> > >>> call
> > >>> > > > > > > > initProducerId
> > >>> > > > > > > > >> to
> > >>> > > > > > > > >> > > > abort
> > >>> > > > > > > > >> > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > transaction. (In the overflow
> case,
> > >>> this
> > >>> > > > > correctly
> > >>> > > > > > > > gives
> > >>> > > > > > > > >> > us a
> > >>> > > > > > > > >> > > > new
> > >>> > > > > > > > >> > > > > > > > > producer
> > >>> > > > > > > > >> > > > > > > > > > ID)
> > >>> > > > > > > > >> > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > I guess the corresponding case
> would
> > >>> be
> > >>> > > where
> > >>> > > > > the
> > >>> > > > > > > > >> *complete
> > >>> > > > > > > > >> > > > > marker
> > >>> > > > > > > > >> > > > > > > *is
> > >>> > > > > > > > >> > > > > > > > > > written but the endTxn is not
> > >>> received by
> > >>> > > the
> > >>> > > > > > client
> > >>> > > > > > > > and
> > >>> > > > > > > > >> > the
> > >>> > > > > > > > >> > > > > server
> > >>> > > > > > > > >> > > > > > > > > > downgrades? This would result in
> the
> > >>> > > > transaction
> > >>> > > > > > > > >> > coordinator
> > >>> > > > > > > > >> > > > > having
> > >>> > > > > > > > >> > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > new
> > >>> > > > > > > > >> > > > > > > > > > ID and not the old one.  If the
> > client
> > >>> > > > retries,
> > >>> > > > > it
> > >>> > > > > > > > will
> > >>> > > > > > > > >> > > receive
> > >>> > > > > > > > >> > > > > an
> > >>> > > > > > > > >> > > > > > > > > > InvalidPidMappingException. The
> > >>> > > InitProducerId
> > >>> > > > > > > > scenario
> > >>> > > > > > > > >> > above
> > >>> > > > > > > > >> > > > > would
> > >>> > > > > > > > >> > > > > > > > help
> > >>> > > > > > > > >> > > > > > > > > > here too.
> > >>> > > > > > > > >> > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > To be clear, my compatibility
> story
> > is
> > >>> > meant
> > >>> > > > to
> > >>> > > > > > > > support
> > >>> > > > > > > > >> > > > > downgrades
> > >>> > > > > > > > >> > > > > > > > server
> > >>> > > > > > > > >> > > > > > > > > > side in keeping the transactional
> > >>> > > correctness.
> > >>> > > > > > > Keeping
> > >>> > > > > > > > >> the
> > >>> > > > > > > > >> > > > client
> > >>> > > > > > > > >> > > > > > > from
> > >>> > > > > > > > >> > > > > > > > > > fencing itself is not the
> priority.
> > >>> > > > > > > > >> > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > Hope this helps. I can also add
> text
> > >>> in
> > >>> > the
> > >>> > > > KIP
> > >>> > > > > > > about
> > >>> > > > > > > > >> > > > > > InitProducerId
> > >>> > > > > > > > >> > > > > > > if
> > >>> > > > > > > > >> > > > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > think that fixes some edge cases.
> > >>> > > > > > > > >> > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > Justine
> > >>> > > > > > > > >> > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > On Fri, Jan 12, 2024 at 4:10 PM
> Jun
> > >>> Rao
> > >>> > > > > > > > >> > > > <jun@confluent.io.invalid
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > > > > > > wrote:
> > >>> > > > > > > > >> > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > Hi, Justine,
> > >>> > > > > > > > >> > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > Thanks for the reply.
> > >>> > > > > > > > >> > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > I agree that we don't need to
> > >>> optimize
> > >>> > for
> > >>> > > > > > fencing
> > >>> > > > > > > > >> during
> > >>> > > > > > > > >> > > > > > > downgrades.
> > >>> > > > > > > > >> > > > > > > > > > > Regarding consistency, there are
> > two
> > >>> > > > possible
> > >>> > > > > > > cases:
> > >>> > > > > > > > >> (1)
> > >>> > > > > > > > >> > > the
> > >>> > > > > > > > >> > > > > > > prepare
> > >>> > > > > > > > >> > > > > > > > > > marker
> > >>> > > > > > > > >> > > > > > > > > > > is written, but the endTxn
> > response
> > >>> is
> > >>> > not
> > >>> > > > > > > received
> > >>> > > > > > > > by
> > >>> > > > > > > > >> > the
> > >>> > > > > > > > >> > > > > client
> > >>> > > > > > > > >> > > > > > > > when
> > >>> > > > > > > > >> > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > server downgrades; (2)  the
> > prepare
> > >>> > marker
> > >>> > > > is
> > >>> > > > > > > > written,
> > >>> > > > > > > > >> > the
> > >>> > > > > > > > >> > > > > endTxn
> > >>> > > > > > > > >> > > > > > > > > > response
> > >>> > > > > > > > >> > > > > > > > > > > is received by the client when
> the
> > >>> > server
> > >>> > > > > > > > downgrades.
> > >>> > > > > > > > >> In
> > >>> > > > > > > > >> > > (1),
> > >>> > > > > > > > >> > > > > the
> > >>> > > > > > > > >> > > > > > > > > client
> > >>> > > > > > > > >> > > > > > > > > > > will have the old produce Id and
> > in
> > >>> (2),
> > >>> > > the
> > >>> > > > > > > client
> > >>> > > > > > > > >> will
> > >>> > > > > > > > >> > > have
> > >>> > > > > > > > >> > > > > the
> > >>> > > > > > > > >> > > > > > > new
> > >>> > > > > > > > >> > > > > > > > > > > produce Id. If we downgrade
> right
> > >>> after
> > >>> > > the
> > >>> > > > > > > prepare
> > >>> > > > > > > > >> > marker,
> > >>> > > > > > > > >> > > > we
> > >>> > > > > > > > >> > > > > > > can't
> > >>> > > > > > > > >> > > > > > > > be
> > >>> > > > > > > > >> > > > > > > > > > > consistent to both (1) and (2)
> > >>> since we
> > >>> > > can
> > >>> > > > > only
> > >>> > > > > > > put
> > >>> > > > > > > > >> one
> > >>> > > > > > > > >> > > > value
> > >>> > > > > > > > >> > > > > in
> > >>> > > > > > > > >> > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > existing produce Id field. It's
> > >>> also not
> > >>> > > > clear
> > >>> > > > > > > which
> > >>> > > > > > > > >> case
> > >>> > > > > > > > >> > > is
> > >>> > > > > > > > >> > > > > more
> > >>> > > > > > > > >> > > > > > > > > likely.
> > >>> > > > > > > > >> > > > > > > > > > > So we could probably be
> consistent
> > >>> with
> > >>> > > > either
> > >>> > > > > > > case.
> > >>> > > > > > > > >> By
> > >>> > > > > > > > >> > > > putting
> > >>> > > > > > > > >> > > > > > the
> > >>> > > > > > > > >> > > > > > > > new
> > >>> > > > > > > > >> > > > > > > > > > > producer Id in the prepare
> marker,
> > >>> we
> > >>> > are
> > >>> > > > > > > consistent
> > >>> > > > > > > > >> with
> > >>> > > > > > > > >> > > > case
> > >>> > > > > > > > >> > > > > > (2)
> > >>> > > > > > > > >> > > > > > > > and
> > >>> > > > > > > > >> > > > > > > > > it
> > >>> > > > > > > > >> > > > > > > > > > > also has the slight benefit that
> > the
> > >>> > > produce
> > >>> > > > > > field
> > >>> > > > > > > > in
> > >>> > > > > > > > >> the
> > >>> > > > > > > > >> > > > > prepare
> > >>> > > > > > > > >> > > > > > > and
> > >>> > > > > > > > >> > > > > > > > > > > complete marker are consistent
> in
> > >>> the
> > >>> > > > overflow
> > >>> > > > > > > case.
> > >>> > > > > > > > >> > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > Jun
> > >>> > > > > > > > >> > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > On Fri, Jan 12, 2024 at 3:11 PM
> > >>> Justine
> > >>> > > > Olshan
> > >>> > > > > > > > >> > > > > > > > > > > <jo...@confluent.io.invalid>
> > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > >>> > > > > > > > >> > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > Hi Jun,
> > >>> > > > > > > > >> > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > In the case you describe, we
> > would
> > >>> > need
> > >>> > > to
> > >>> > > > > > have
> > >>> > > > > > > a
> > >>> > > > > > > > >> > delayed
> > >>> > > > > > > > >> > > > > > > request,
> > >>> > > > > > > > >> > > > > > > > > > send a
> > >>> > > > > > > > >> > > > > > > > > > > > successful EndTxn, and a
> > >>> successful
> > >>> > > > > > > > >> AddPartitionsToTxn
> > >>> > > > > > > > >> > > and
> > >>> > > > > > > > >> > > > > then
> > >>> > > > > > > > >> > > > > > > > have
> > >>> > > > > > > > >> > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > delayed EndTxn request go
> > through
> > >>> for
> > >>> > a
> > >>> > > > > given
> > >>> > > > > > > > >> producer.
> > >>> > > > > > > > >> > > > > > > > > > > > I'm trying to figure out if it
> > is
> > >>> > > possible
> > >>> > > > > for
> > >>> > > > > > > the
> > >>> > > > > > > > >> > client
> > >>> > > > > > > > >> > > > to
> > >>> > > > > > > > >> > > > > > > > > transition
> > >>> > > > > > > > >> > > > > > > > > > > if
> > >>> > > > > > > > >> > > > > > > > > > > > a previous request is delayed
> > >>> > somewhere.
> > >>> > > > But
> > >>> > > > > > > yes,
> > >>> > > > > > > > in
> > >>> > > > > > > > >> > this
> > >>> > > > > > > > >> > > > > case
> > >>> > > > > > > > >> > > > > > I
> > >>> > > > > > > > >> > > > > > > > > think
> > >>> > > > > > > > >> > > > > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > > > would fence the client.
> > >>> > > > > > > > >> > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > Not for the overflow case. In
> > the
> > >>> > > overflow
> > >>> > > > > > case,
> > >>> > > > > > > > the
> > >>> > > > > > > > >> > > > producer
> > >>> > > > > > > > >> > > > > > ID
> > >>> > > > > > > > >> > > > > > > > and
> > >>> > > > > > > > >> > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > epoch are different on the
> > marker
> > >>> and
> > >>> > on
> > >>> > > > the
> > >>> > > > > > new
> > >>> > > > > > > > >> > > > transaction.
> > >>> > > > > > > > >> > > > > > So
> > >>> > > > > > > > >> > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > want
> > >>> > > > > > > > >> > > > > > > > > > > > the marker to use the max
> epoch
> > >>> but
> > >>> > the
> > >>> > > > new
> > >>> > > > > > > > >> > transaction
> > >>> > > > > > > > >> > > > > should
> > >>> > > > > > > > >> > > > > > > > start
> > >>> > > > > > > > >> > > > > > > > > > > with
> > >>> > > > > > > > >> > > > > > > > > > > > the new ID and epoch 0 in the
> > >>> > > > transactional
> > >>> > > > > > > state.
> > >>> > > > > > > > >> > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > In the server downgrade case,
> we
> > >>> want
> > >>> > to
> > >>> > > > see
> > >>> > > > > > the
> > >>> > > > > > > > >> > producer
> > >>> > > > > > > > >> > > > ID
> > >>> > > > > > > > >> > > > > as
> > >>> > > > > > > > >> > > > > > > > that
> > >>> > > > > > > > >> > > > > > > > > is
> > >>> > > > > > > > >> > > > > > > > > > > > what the client will have. If
> we
> > >>> > > complete
> > >>> > > > > the
> > >>> > > > > > > > >> commit,
> > >>> > > > > > > > >> > and
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > > > > > > transaction
> > >>> > > > > > > > >> > > > > > > > > > > > state is reloaded, we need the
> > new
> > >>> > > > producer
> > >>> > > > > ID
> > >>> > > > > > > in
> > >>> > > > > > > > >> the
> > >>> > > > > > > > >> > > state
> > >>> > > > > > > > >> > > > > so
> > >>> > > > > > > > >> > > > > > > > there
> > >>> > > > > > > > >> > > > > > > > > > > isn't
> > >>> > > > > > > > >> > > > > > > > > > > > an invalid producer ID
> mapping.
> > >>> > > > > > > > >> > > > > > > > > > > > The server downgrade cases are
> > >>> > > considering
> > >>> > > > > > > > >> > transactional
> > >>> > > > > > > > >> > > > > > > > correctness
> > >>> > > > > > > > >> > > > > > > > > > and
> > >>> > > > > > > > >> > > > > > > > > > > > not regressing from previous
> > >>> behavior
> > >>> > --
> > >>> > > > and
> > >>> > > > > > are
> > >>> > > > > > > > not
> > >>> > > > > > > > >> > > > > concerned
> > >>> > > > > > > > >> > > > > > > > about
> > >>> > > > > > > > >> > > > > > > > > > > > supporting the safety from
> > fencing
> > >>> > > retries
> > >>> > > > > (as
> > >>> > > > > > > we
> > >>> > > > > > > > >> have
> > >>> > > > > > > > >> > > > > > downgraded
> > >>> > > > > > > > >> > > > > > > > so
> > >>> > > > > > > > >> > > > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > > > don't need to support).
> Perhaps
> > >>> this
> > >>> > is
> > >>> > > a
> > >>> > > > > > trade
> > >>> > > > > > > > off,
> > >>> > > > > > > > >> > but
> > >>> > > > > > > > >> > > I
> > >>> > > > > > > > >> > > > > > think
> > >>> > > > > > > > >> > > > > > > it
> > >>> > > > > > > > >> > > > > > > > > is
> > >>> > > > > > > > >> > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > right one.
> > >>> > > > > > > > >> > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > (If the client downgrades, it
> > will
> > >>> > have
> > >>> > > > > > > restarted
> > >>> > > > > > > > >> and
> > >>> > > > > > > > >> > it
> > >>> > > > > > > > >> > > is
> > >>> > > > > > > > >> > > > > ok
> > >>> > > > > > > > >> > > > > > > for
> > >>> > > > > > > > >> > > > > > > > it
> > >>> > > > > > > > >> > > > > > > > > > to
> > >>> > > > > > > > >> > > > > > > > > > > > have a new producer ID too).
> > >>> > > > > > > > >> > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > Justine
> > >>> > > > > > > > >> > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > On Fri, Jan 12, 2024 at
> 11:42 AM
> > >>> Jun
> > >>> > Rao
> > >>> > > > > > > > >> > > > > > > <jun@confluent.io.invalid
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > wrote:
> > >>> > > > > > > > >> > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > Hi, Justine,
> > >>> > > > > > > > >> > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > Thanks for the reply.
> > >>> > > > > > > > >> > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > 101.4 "If the marker is
> > written
> > >>> by
> > >>> > the
> > >>> > > > new
> > >>> > > > > > > > >> client, we
> > >>> > > > > > > > >> > > can
> > >>> > > > > > > > >> > > > > as
> > >>> > > > > > > > >> > > > > > I
> > >>> > > > > > > > >> > > > > > > > > > > mentioned
> > >>> > > > > > > > >> > > > > > > > > > > > in
> > >>> > > > > > > > >> > > > > > > > > > > > > the last email guarantee
> that
> > >>> any
> > >>> > > EndTxn
> > >>> > > > > > > > requests
> > >>> > > > > > > > >> > with
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > > same
> > >>> > > > > > > > >> > > > > > > > > epoch
> > >>> > > > > > > > >> > > > > > > > > > > are
> > >>> > > > > > > > >> > > > > > > > > > > > > from the same producer and
> the
> > >>> same
> > >>> > > > > > > transaction.
> > >>> > > > > > > > >> Then
> > >>> > > > > > > > >> > > we
> > >>> > > > > > > > >> > > > > > don't
> > >>> > > > > > > > >> > > > > > > > have
> > >>> > > > > > > > >> > > > > > > > > > to
> > >>> > > > > > > > >> > > > > > > > > > > > > return a fenced error but
> can
> > >>> handle
> > >>> > > > > > > gracefully
> > >>> > > > > > > > as
> > >>> > > > > > > > >> > > > > described
> > >>> > > > > > > > >> > > > > > in
> > >>> > > > > > > > >> > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > KIP."
> > >>> > > > > > > > >> > > > > > > > > > > > > When a delayed EndTnx
> request
> > is
> > >>> > > > > processed,
> > >>> > > > > > > the
> > >>> > > > > > > > >> txn
> > >>> > > > > > > > >> > > state
> > >>> > > > > > > > >> > > > > > could
> > >>> > > > > > > > >> > > > > > > > be
> > >>> > > > > > > > >> > > > > > > > > > > > ongoing
> > >>> > > > > > > > >> > > > > > > > > > > > > for the next txn. I guess in
> > >>> this
> > >>> > case
> > >>> > > > we
> > >>> > > > > > > still
> > >>> > > > > > > > >> > return
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > > > fenced
> > >>> > > > > > > > >> > > > > > > > > > error
> > >>> > > > > > > > >> > > > > > > > > > > > for
> > >>> > > > > > > > >> > > > > > > > > > > > > the delayed request?
> > >>> > > > > > > > >> > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > 102. Sorry, my question was
> > >>> > > inaccurate.
> > >>> > > > > What
> > >>> > > > > > > you
> > >>> > > > > > > > >> > > > described
> > >>> > > > > > > > >> > > > > is
> > >>> > > > > > > > >> > > > > > > > > > accurate.
> > >>> > > > > > > > >> > > > > > > > > > > > > "The downgrade
> compatibility I
> > >>> > mention
> > >>> > > > is
> > >>> > > > > > that
> > >>> > > > > > > > we
> > >>> > > > > > > > >> > keep
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > > same
> > >>> > > > > > > > >> > > > > > > > > > > producer
> > >>> > > > > > > > >> > > > > > > > > > > > ID
> > >>> > > > > > > > >> > > > > > > > > > > > > and epoch in the main
> > >>> (non-tagged)
> > >>> > > > fields
> > >>> > > > > as
> > >>> > > > > > > we
> > >>> > > > > > > > >> did
> > >>> > > > > > > > >> > > > before
> > >>> > > > > > > > >> > > > > > the
> > >>> > > > > > > > >> > > > > > > > code
> > >>> > > > > > > > >> > > > > > > > > > on
> > >>> > > > > > > > >> > > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > server side." If we want to
> do
> > >>> this,
> > >>> > > it
> > >>> > > > > > seems
> > >>> > > > > > > > >> that we
> > >>> > > > > > > > >> > > > > should
> > >>> > > > > > > > >> > > > > > > use
> > >>> > > > > > > > >> > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > current produce Id and max
> > >>> epoch in
> > >>> > > the
> > >>> > > > > > > existing
> > >>> > > > > > > > >> > > > producerId
> > >>> > > > > > > > >> > > > > > and
> > >>> > > > > > > > >> > > > > > > > > > > > > producerEpoch fields for
> both
> > >>> the
> > >>> > > > prepare
> > >>> > > > > > and
> > >>> > > > > > > > the
> > >>> > > > > > > > >> > > > complete
> > >>> > > > > > > > >> > > > > > > > marker,
> > >>> > > > > > > > >> > > > > > > > > > > right?
> > >>> > > > > > > > >> > > > > > > > > > > > > The downgrade can happen
> after
> > >>> the
> > >>> > > > > complete
> > >>> > > > > > > > >> marker is
> > >>> > > > > > > > >> > > > > > written.
> > >>> > > > > > > > >> > > > > > > > With
> > >>> > > > > > > > >> > > > > > > > > > > what
> > >>> > > > > > > > >> > > > > > > > > > > > > you described, the
> downgraded
> > >>> > > > coordinator
> > >>> > > > > > will
> > >>> > > > > > > > see
> > >>> > > > > > > > >> > the
> > >>> > > > > > > > >> > > > new
> > >>> > > > > > > > >> > > > > > > > produce
> > >>> > > > > > > > >> > > > > > > > > Id
> > >>> > > > > > > > >> > > > > > > > > > > > > instead of the old one.
> > >>> > > > > > > > >> > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > Jun
> > >>> > > > > > > > >> > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > On Fri, Jan 12, 2024 at
> > 10:44 AM
> > >>> > > Justine
> > >>> > > > > > > Olshan
> > >>> > > > > > > > >> > > > > > > > > > > > >
> <jolshan@confluent.io.invalid
> > >
> > >>> > wrote:
> > >>> > > > > > > > >> > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > Hi Jun,
> > >>> > > > > > > > >> > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > I can update the
> > description.
> > >>> > > > > > > > >> > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > I believe your second
> point
> > is
> > >>> > > > mentioned
> > >>> > > > > > in
> > >>> > > > > > > > the
> > >>> > > > > > > > >> > KIP.
> > >>> > > > > > > > >> > > I
> > >>> > > > > > > > >> > > > > can
> > >>> > > > > > > > >> > > > > > > add
> > >>> > > > > > > > >> > > > > > > > > more
> > >>> > > > > > > > >> > > > > > > > > > > > text
> > >>> > > > > > > > >> > > > > > > > > > > > > on
> > >>> > > > > > > > >> > > > > > > > > > > > > > this if it is helpful.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > The delayed message case
> > can
> > >>> > also
> > >>> > > > > > violate
> > >>> > > > > > > > EOS
> > >>> > > > > > > > >> if
> > >>> > > > > > > > >> > > the
> > >>> > > > > > > > >> > > > > > > delayed
> > >>> > > > > > > > >> > > > > > > > > > > message
> > >>> > > > > > > > >> > > > > > > > > > > > > > comes in after the next
> > >>> > > > > addPartitionsToTxn
> > >>> > > > > > > > >> request
> > >>> > > > > > > > >> > > > comes
> > >>> > > > > > > > >> > > > > > in.
> > >>> > > > > > > > >> > > > > > > > > > > > Effectively
> > >>> > > > > > > > >> > > > > > > > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > > > > > may see a message from a
> > >>> previous
> > >>> > > > > > (aborted)
> > >>> > > > > > > > >> > > transaction
> > >>> > > > > > > > >> > > > > > > become
> > >>> > > > > > > > >> > > > > > > > > part
> > >>> > > > > > > > >> > > > > > > > > > > of
> > >>> > > > > > > > >> > > > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > next transaction.
> > >>> > > > > > > > >> > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > If the marker is written
> by
> > >>> the
> > >>> > new
> > >>> > > > > > client,
> > >>> > > > > > > we
> > >>> > > > > > > > >> can
> > >>> > > > > > > > >> > > as I
> > >>> > > > > > > > >> > > > > > > > mentioned
> > >>> > > > > > > > >> > > > > > > > > > in
> > >>> > > > > > > > >> > > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > last email guarantee that
> > any
> > >>> > EndTxn
> > >>> > > > > > > requests
> > >>> > > > > > > > >> with
> > >>> > > > > > > > >> > > the
> > >>> > > > > > > > >> > > > > same
> > >>> > > > > > > > >> > > > > > > > epoch
> > >>> > > > > > > > >> > > > > > > > > > are
> > >>> > > > > > > > >> > > > > > > > > > > > > from
> > >>> > > > > > > > >> > > > > > > > > > > > > > the same producer and the
> > same
> > >>> > > > > > transaction.
> > >>> > > > > > > > >> Then we
> > >>> > > > > > > > >> > > > don't
> > >>> > > > > > > > >> > > > > > > have
> > >>> > > > > > > > >> > > > > > > > to
> > >>> > > > > > > > >> > > > > > > > > > > > return
> > >>> > > > > > > > >> > > > > > > > > > > > > a
> > >>> > > > > > > > >> > > > > > > > > > > > > > fenced error but can
> handle
> > >>> > > gracefully
> > >>> > > > > as
> > >>> > > > > > > > >> described
> > >>> > > > > > > > >> > > in
> > >>> > > > > > > > >> > > > > the
> > >>> > > > > > > > >> > > > > > > KIP.
> > >>> > > > > > > > >> > > > > > > > > > > > > > I don't think a boolean is
> > >>> useful
> > >>> > > > since
> > >>> > > > > it
> > >>> > > > > > > is
> > >>> > > > > > > > >> > > directly
> > >>> > > > > > > > >> > > > > > > encoded
> > >>> > > > > > > > >> > > > > > > > by
> > >>> > > > > > > > >> > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > existence or lack of the
> > >>> tagged
> > >>> > > field
> > >>> > > > > > being
> > >>> > > > > > > > >> > written.
> > >>> > > > > > > > >> > > > > > > > > > > > > > In the prepare marker we
> > will
> > >>> have
> > >>> > > the
> > >>> > > > > > same
> > >>> > > > > > > > >> > producer
> > >>> > > > > > > > >> > > ID
> > >>> > > > > > > > >> > > > > in
> > >>> > > > > > > > >> > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > non-tagged
> > >>> > > > > > > > >> > > > > > > > > > > > > > field. In the Complete
> state
> > >>> we
> > >>> > may
> > >>> > > > not.
> > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure why the
> ongoing
> > >>> state
> > >>> > > > > matters
> > >>> > > > > > > for
> > >>> > > > > > > > >> this
> > >>> > > > > > > > >> > > > KIP.
> > >>> > > > > > > > >> > > > > It
> > >>> > > > > > > > >> > > > > > > > does
> > >>> > > > > > > > >> > > > > > > > > > > matter
> > >>> > > > > > > > >> > > > > > > > > > > > > for
> > >>> > > > > > > > >> > > > > > > > > > > > > > KIP-939.
> > >>> > > > > > > > >> > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > I'm not sure what you are
> > >>> > referring
> > >>> > > to
> > >>> > > > > > about
> > >>> > > > > > > > >> > writing
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > > > > previous
> > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > >>> > > > > > > > >> > > > > > > > > > > > > > ID in the prepare marker.
> > >>> This is
> > >>> > > not
> > >>> > > > in
> > >>> > > > > > the
> > >>> > > > > > > > >> KIP.
> > >>> > > > > > > > >> > > > > > > > > > > > > > In the overflow case, we
> > >>> write the
> > >>> > > > > > > > >> nextProducerId
> > >>> > > > > > > > >> > in
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > > > > prepare
> > >>> > > > > > > > >> > > > > > > > > > > state.
> > >>> > > > > > > > >> > > > > > > > > > > > > > This is so we know what we
> > >>> > assigned
> > >>> > > > when
> > >>> > > > > > we
> > >>> > > > > > > > >> reload
> > >>> > > > > > > > >> > > the
> > >>> > > > > > > > >> > > > > > > > > transaction
> > >>> > > > > > > > >> > > > > > > > > > > log.
> > >>> > > > > > > > >> > > > > > > > > > > > > > Once we complete, we
> > >>> transition
> > >>> > this
> > >>> > > > ID
> > >>> > > > > to
> > >>> > > > > > > the
> > >>> > > > > > > > >> main
> > >>> > > > > > > > >> > > > > > > (non-tagged
> > >>> > > > > > > > >> > > > > > > > > > > field)
> > >>> > > > > > > > >> > > > > > > > > > > > > and
> > >>> > > > > > > > >> > > > > > > > > > > > > > have the previous producer
> > ID
> > >>> > field
> > >>> > > > > filled
> > >>> > > > > > > in.
> > >>> > > > > > > > >> This
> > >>> > > > > > > > >> > > is
> > >>> > > > > > > > >> > > > so
> > >>> > > > > > > > >> > > > > > we
> > >>> > > > > > > > >> > > > > > > > can
> > >>> > > > > > > > >> > > > > > > > > > > > identify
> > >>> > > > > > > > >> > > > > > > > > > > > > > in a retry case the
> > operation
> > >>> > > > completed
> > >>> > > > > > > > >> > successfully
> > >>> > > > > > > > >> > > > and
> > >>> > > > > > > > >> > > > > we
> > >>> > > > > > > > >> > > > > > > > don't
> > >>> > > > > > > > >> > > > > > > > > > > fence
> > >>> > > > > > > > >> > > > > > > > > > > > > our
> > >>> > > > > > > > >> > > > > > > > > > > > > > producer. The downgrade
> > >>> > > compatibility
> > >>> > > > I
> > >>> > > > > > > > mention
> > >>> > > > > > > > >> is
> > >>> > > > > > > > >> > > that
> > >>> > > > > > > > >> > > > > we
> > >>> > > > > > > > >> > > > > > > keep
> > >>> > > > > > > > >> > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > same
> > >>> > > > > > > > >> > > > > > > > > > > > > > producer ID and epoch in
> the
> > >>> main
> > >>> > > > > > > (non-tagged)
> > >>> > > > > > > > >> > fields
> > >>> > > > > > > > >> > > > as
> > >>> > > > > > > > >> > > > > we
> > >>> > > > > > > > >> > > > > > > did
> > >>> > > > > > > > >> > > > > > > > > > > before
> > >>> > > > > > > > >> > > > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > code on the server side.
> If
> > >>> the
> > >>> > > server
> > >>> > > > > > > > >> downgrades,
> > >>> > > > > > > > >> > we
> > >>> > > > > > > > >> > > > are
> > >>> > > > > > > > >> > > > > > > still
> > >>> > > > > > > > >> > > > > > > > > > > > > compatible.
> > >>> > > > > > > > >> > > > > > > > > > > > > > This addresses both the
> > >>> prepare
> > >>> > and
> > >>> > > > > > complete
> > >>> > > > > > > > >> state
> > >>> > > > > > > > >> > > > > > > downgrades.
> > >>> > > > > > > > >> > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > Justine
> > >>> > > > > > > > >> > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > On Fri, Jan 12, 2024 at
> > >>> 10:21 AM
> > >>> > Jun
> > >>> > > > Rao
> > >>> > > > > > > > >> > > > > > > > > <jun@confluent.io.invalid
> > >>> > > > > > > > >> > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > wrote:
> > >>> > > > > > > > >> > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > Hi, Justine,
> > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > Thanks for the reply.
> > Sorry
> > >>> for
> > >>> > > the
> > >>> > > > > > > delay. I
> > >>> > > > > > > > >> > have a
> > >>> > > > > > > > >> > > > few
> > >>> > > > > > > > >> > > > > > > more
> > >>> > > > > > > > >> > > > > > > > > > > > comments.
> > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > 110. I think the
> > motivation
> > >>> > > section
> > >>> > > > > > could
> > >>> > > > > > > be
> > >>> > > > > > > > >> > > > improved.
> > >>> > > > > > > > >> > > > > > One
> > >>> > > > > > > > >> > > > > > > of
> > >>> > > > > > > > >> > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > > motivations listed by
> the
> > >>> KIP is
> > >>> > > > "This
> > >>> > > > > > can
> > >>> > > > > > > > >> happen
> > >>> > > > > > > > >> > > > when
> > >>> > > > > > > > >> > > > > a
> > >>> > > > > > > > >> > > > > > > > > message
> > >>> > > > > > > > >> > > > > > > > > > > gets
> > >>> > > > > > > > >> > > > > > > > > > > > > > stuck
> > >>> > > > > > > > >> > > > > > > > > > > > > > > or delayed due to
> > networking
> > >>> > > issues
> > >>> > > > > or a
> > >>> > > > > > > > >> network
> > >>> > > > > > > > >> > > > > > partition,
> > >>> > > > > > > > >> > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > transaction
> > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts, and then the
> > delayed
> > >>> > > message
> > >>> > > > > > > finally
> > >>> > > > > > > > >> > comes
> > >>> > > > > > > > >> > > > > in.".
> > >>> > > > > > > > >> > > > > > > This
> > >>> > > > > > > > >> > > > > > > > > > seems
> > >>> > > > > > > > >> > > > > > > > > > > > not
> > >>> > > > > > > > >> > > > > > > > > > > > > > > very accurate. Without
> > >>> KIP-890,
> > >>> > > > > > currently,
> > >>> > > > > > > > if
> > >>> > > > > > > > >> the
> > >>> > > > > > > > >> > > > > > > coordinator
> > >>> > > > > > > > >> > > > > > > > > > times
> > >>> > > > > > > > >> > > > > > > > > > > > out
> > >>> > > > > > > > >> > > > > > > > > > > > > > and
> > >>> > > > > > > > >> > > > > > > > > > > > > > > aborts an ongoing
> > >>> transaction,
> > >>> > it
> > >>> > > > > > already
> > >>> > > > > > > > >> bumps
> > >>> > > > > > > > >> > up
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > > > epoch
> > >>> > > > > > > > >> > > > > > > > in
> > >>> > > > > > > > >> > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > marker,
> > >>> > > > > > > > >> > > > > > > > > > > > > > > which prevents the
> delayed
> > >>> > produce
> > >>> > > > > > message
> > >>> > > > > > > > >> from
> > >>> > > > > > > > >> > > being
> > >>> > > > > > > > >> > > > > > added
> > >>> > > > > > > > >> > > > > > > > to
> > >>> > > > > > > > >> > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > user
> > >>> > > > > > > > >> > > > > > > > > > > > > > > partition. What can
> cause
> > a
> > >>> > > hanging
> > >>> > > > > > > > >> transaction
> > >>> > > > > > > > >> > is
> > >>> > > > > > > > >> > > > that
> > >>> > > > > > > > >> > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > producer
> > >>> > > > > > > > >> > > > > > > > > > > > > > > completes (either aborts
> > or
> > >>> > > > commits) a
> > >>> > > > > > > > >> > transaction
> > >>> > > > > > > > >> > > > > before
> > >>> > > > > > > > >> > > > > > > > > > > receiving a
> > >>> > > > > > > > >> > > > > > > > > > > > > > > successful ack on
> messages
> > >>> > > published
> > >>> > > > > in
> > >>> > > > > > > the
> > >>> > > > > > > > >> same
> > >>> > > > > > > > >> > > txn.
> > >>> > > > > > > > >> > > > > In
> > >>> > > > > > > > >> > > > > > > this
> > >>> > > > > > > > >> > > > > > > > > > case,
> > >>> > > > > > > > >> > > > > > > > > > > > > it's
> > >>> > > > > > > > >> > > > > > > > > > > > > > > possible for the delayed
> > >>> message
> > >>> > > to
> > >>> > > > be
> > >>> > > > > > > > >> appended
> > >>> > > > > > > > >> > to
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > > > > > partition
> > >>> > > > > > > > >> > > > > > > > > > > > after
> > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > > marker, causing a
> > >>> transaction to
> > >>> > > > hang.
> > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > A similar issue (not
> > >>> mentioned
> > >>> > in
> > >>> > > > the
> > >>> > > > > > > > >> motivation)
> > >>> > > > > > > > >> > > > could
> > >>> > > > > > > > >> > > > > > > > happen
> > >>> > > > > > > > >> > > > > > > > > on
> > >>> > > > > > > > >> > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > > marker in the
> > coordinator's
> > >>> log.
> > >>> > > For
> > >>> > > > > > > > example,
> > >>> > > > > > > > >> > it's
> > >>> > > > > > > > >> > > > > > possible
> > >>> > > > > > > > >> > > > > > > > for
> > >>> > > > > > > > >> > > > > > > > > > an
> > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest to be
> > delayed
> > >>> on
> > >>> > the
> > >>> > > > > > > > >> coordinator.
> > >>> > > > > > > > >> > By
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > > time
> > >>> > > > > > > > >> > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > delayed
> > >>> > > > > > > > >> > > > > > > > > > > > > > > EndTxnRequest is
> > processed,
> > >>> it's
> > >>> > > > > > possible
> > >>> > > > > > > > that
> > >>> > > > > > > > >> > the
> > >>> > > > > > > > >> > > > > > previous
> > >>> > > > > > > > >> > > > > > > > txn
> > >>> > > > > > > > >> > > > > > > > > > has
> > >>> > > > > > > > >> > > > > > > > > > > > > > already
> > >>> > > > > > > > >> > > > > > > > > > > > > > > completed and a new txn
> > has
> > >>> > > started.
> > >>> > > > > > > > >> Currently,
> > >>> > > > > > > > >> > > since
> > >>> > > > > > > > >> > > > > the
> > >>> > > > > > > > >> > > > > > > > epoch
> > >>> > > > > > > > >> > > > > > > > > > is
> > >>> > > > > > > > >> > > > > > > > > > > > not
> > >>> > > > > > > > >> > > > > > > > > > > > > > > bumped on every txn, the
> > >>> delayed
> > >>> > > > > > > > EndTxnRequest
> > >>> > > > > > > > >> > will
> > >>> > > > > > > > >> > > > add
> > >>> > > > > > > > >> > > > > > an
> > >>> > > > > > > > >> > > > > > > > > > > unexpected
> > >>> > > > > > > > >> > > > > > > > > > > > > > > prepare marker (and
> > >>> eventually a
> > >>> > > > > > complete
> > >>> > > > > > > > >> marker)
> > >>> > > > > > > > >> > > to
> > >>> > > > > > > > >> > > > > the
> > >>> > > > > > > > >> > > > > > > > > ongoing
> > >>> > > > > > > > >> > > > > > > > > > > txn.
> > >>> > > > > > > > >> > > > > > > > > > > > > > This
> > >>> > > > > > > > >> > > > > > > > > > > > > > > won't cause the
> > transaction
> > >>> to
> > >>> > > hang,
> > >>> > > > > but
> > >>> > > > > > > it
> > >>> > > > > > > > >> will
> > >>> > > > > > > > >> > > > break
> > >>> > > > > > > > >> > > > > > the
> > >>> > > > > > > > >> > > > > > > > EoS
> > >>> > > > > > > > >> > > > > > > > > > > > > semantic.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > The proposal in this KIP
> > >>> will
> > >>> > > > address
> > >>> > > > > > this
> > >>> > > > > > > > >> issue
> > >>> > > > > > > > >> > > too.
> > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > 101. "However, I was
> > >>> writing it
> > >>> > so
> > >>> > > > > that
> > >>> > > > > > we
> > >>> > > > > > > > can
> > >>> > > > > > > > >> > > > > > distinguish
> > >>> > > > > > > > >> > > > > > > > > > between
> > >>> > > > > > > > >> > > > > > > > > > > > > > > old clients where we
> don't
> > >>> have
> > >>> > > the
> > >>> > > > > > > ability
> > >>> > > > > > > > do
> > >>> > > > > > > > >> > this
> > >>> > > > > > > > >> > > > > > > operation
> > >>> > > > > > > > >> > > > > > > > > and
> > >>> > > > > > > > >> > > > > > > > > > > new
> > >>> > > > > > > > >> > > > > > > > > > > > > > > clients that can. (Old
> > >>> clients
> > >>> > > don't
> > >>> > > > > > bump
> > >>> > > > > > > > the
> > >>> > > > > > > > >> > epoch
> > >>> > > > > > > > >> > > > on
> > >>> > > > > > > > >> > > > > > > > commit,
> > >>> > > > > > > > >> > > > > > > > > so
> > >>> > > > > > > > >> > > > > > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > > > > > can't
> > >>> > > > > > > > >> > > > > > > > > > > > > > > say for sure the write
> > >>> belongs
> > >>> > to
> > >>> > > > the
> > >>> > > > > > > given
> > >>> > > > > > > > >> > > > > > transaction)."
> > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.1 I am wondering why
> > we
> > >>> need
> > >>> > > to
> > >>> > > > > > > > >> distinguish
> > >>> > > > > > > > >> > > > whether
> > >>> > > > > > > > >> > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > marker
> > >>> > > > > > > > >> > > > > > > > > > > is
> > >>> > > > > > > > >> > > > > > > > > > > > > > > written by the old and
> the
> > >>> new
> > >>> > > > client.
> > >>> > > > > > > Could
> > >>> > > > > > > > >> you
> > >>> > > > > > > > >> > > > > describe
> > >>> > > > > > > > >> > > > > > > > what
> > >>> > > > > > > > >> > > > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > > do
> > >>> > > > > > > > >> > > > > > > > > > > > > > > differently if we know
> the
> > >>> > marker
> > >>> > > is
> > >>> > > > > > > written
> > >>> > > > > > > > >> by
> > >>> > > > > > > > >> > the
> > >>> > > > > > > > >> > > > new
> > >>> > > > > > > > >> > > > > > > > client?
> > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.2 If we do need a
> way
> > to
> > >>> > > > > distinguish
> > >>> > > > > > > > >> whether
> > >>> > > > > > > > >> > > the
> > >>> > > > > > > > >> > > > > > marker
> > >>> > > > > > > > >> > > > > > > > is
> > >>> > > > > > > > >> > > > > > > > > > > > written
> > >>> > > > > > > > >> > > > > > > > > > > > > by
> > >>> > > > > > > > >> > > > > > > > > > > > > > > the old and the new
> > client.
> > >>> > Would
> > >>> > > it
> > >>> > > > > be
> > >>> > > > > > > > >> simpler
> > >>> > > > > > > > >> > to
> > >>> > > > > > > > >> > > > just
> > >>> > > > > > > > >> > > > > > > > > > introduce a
> > >>> > > > > > > > >> > > > > > > > > > > > > > boolean
> > >>> > > > > > > > >> > > > > > > > > > > > > > > field instead of
> > indirectly
> > >>> > > through
> > >>> > > > > the
> > >>> > > > > > > > >> previous
> > >>> > > > > > > > >> > > > > produce
> > >>> > > > > > > > >> > > > > > ID
> > >>> > > > > > > > >> > > > > > > > > > field?
> > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.3 It's not clear to
> me
> > >>> why
> > >>> > we
> > >>> > > > only
> > >>> > > > > > add
> > >>> > > > > > > > the
> > >>> > > > > > > > >> > > > previous
> > >>> > > > > > > > >> > > > > > > > produce
> > >>> > > > > > > > >> > > > > > > > > > ID
> > >>> > > > > > > > >> > > > > > > > > > > > > field
> > >>> > > > > > > > >> > > > > > > > > > > > > > in
> > >>> > > > > > > > >> > > > > > > > > > > > > > > the complete marker, but
> > >>> not in
> > >>> > > the
> > >>> > > > > > > prepare
> > >>> > > > > > > > >> > marker.
> > >>> > > > > > > > >> > > > If
> > >>> > > > > > > > >> > > > > we
> > >>> > > > > > > > >> > > > > > > > want
> > >>> > > > > > > > >> > > > > > > > > to
> > >>> > > > > > > > >> > > > > > > > > > > > know
> > >>> > > > > > > > >> > > > > > > > > > > > > > > whether a marker is
> > written
> > >>> by
> > >>> > the
> > >>> > > > new
> > >>> > > > > > > > client
> > >>> > > > > > > > >> or
> > >>> > > > > > > > >> > > not,
> > >>> > > > > > > > >> > > > > it
> > >>> > > > > > > > >> > > > > > > > seems
> > >>> > > > > > > > >> > > > > > > > > > that
> > >>> > > > > > > > >> > > > > > > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > > > > > want
> > >>> > > > > > > > >> > > > > > > > > > > > > > > to do this consistently
> > for
> > >>> all
> > >>> > > > > markers.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > 101.4 What about the
> > >>> > > > > TransactionLogValue
> > >>> > > > > > > > >> record
> > >>> > > > > > > > >> > > > > > > representing
> > >>> > > > > > > > >> > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > ongoing
> > >>> > > > > > > > >> > > > > > > > > > > > > > > state? Should we also
> > >>> > distinguish
> > >>> > > > > > whether
> > >>> > > > > > > > it's
> > >>> > > > > > > > >> > > > written
> > >>> > > > > > > > >> > > > > by
> > >>> > > > > > > > >> > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > old
> > >>> > > > > > > > >> > > > > > > > > > > or
> > >>> > > > > > > > >> > > > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > > new client?
> > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > 102. In the overflow
> case,
> > >>> it's
> > >>> > > > still
> > >>> > > > > > not
> > >>> > > > > > > > >> clear
> > >>> > > > > > > > >> > to
> > >>> > > > > > > > >> > > me
> > >>> > > > > > > > >> > > > > why
> > >>> > > > > > > > >> > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > write
> > >>> > > > > > > > >> > > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > > previous produce Id in
> the
> > >>> > prepare
> > >>> > > > > > marker
> > >>> > > > > > > > >> while
> > >>> > > > > > > > >> > > > writing
> > >>> > > > > > > > >> > > > > > the
> > >>> > > > > > > > >> > > > > > > > > next
> > >>> > > > > > > > >> > > > > > > > > > > > > produce
> > >>> > > > > > > > >> > > > > > > > > > > > > > Id
> > >>> > > > > > > > >> > > > > > > > > > > > > > > in the complete marker.
> > You
> > >>> > > > mentioned
> > >>> > > > > > that
> > >>> > > > > > > > >> it's
> > >>> > > > > > > > >> > for
> > >>> > > > > > > > >> > > > > > > > > downgrading.
> > >>> > > > > > > > >> > > > > > > > > > > > > However,
> > >>> > > > > > > > >> > > > > > > > > > > > > > > we could downgrade with
> > >>> either
> > >>> > the
> > >>> > > > > > prepare
> > >>> > > > > > > > >> marker
> > >>> > > > > > > > >> > > or
> > >>> > > > > > > > >> > > > > the
> > >>> > > > > > > > >> > > > > > > > > complete
> > >>> > > > > > > > >> > > > > > > > > > > > > marker.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > In either case, the
> > >>> downgraded
> > >>> > > > > > coordinator
> > >>> > > > > > > > >> should
> > >>> > > > > > > > >> > > see
> > >>> > > > > > > > >> > > > > the
> > >>> > > > > > > > >> > > > > > > > same
> > >>> > > > > > > > >> > > > > > > > > > > > produce
> > >>> > > > > > > > >> > > > > > > > > > > > > id
> > >>> > > > > > > > >> > > > > > > > > > > > > > > (probably the previous
> > >>> produce
> > >>> > > Id),
> > >>> > > > > > right?
> > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > Jun
> > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > On Wed, Dec 20, 2023 at
> > >>> 6:00 PM
> > >>> > > > > Justine
> > >>> > > > > > > > Olshan
> > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > >>> <jo...@confluent.io.invalid>
> > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hey Jun,
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > Thanks for taking a
> look
> > >>> at
> > >>> > the
> > >>> > > > KIP
> > >>> > > > > > > again.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > 100. For the epoch
> > >>> overflow
> > >>> > > case,
> > >>> > > > > only
> > >>> > > > > > > the
> > >>> > > > > > > > >> > marker
> > >>> > > > > > > > >> > > > > will
> > >>> > > > > > > > >> > > > > > > have
> > >>> > > > > > > > >> > > > > > > > > max
> > >>> > > > > > > > >> > > > > > > > > > > > > epoch.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > This
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > keeps the behavior of
> > the
> > >>> rest
> > >>> > > of
> > >>> > > > > the
> > >>> > > > > > > > >> markers
> > >>> > > > > > > > >> > > where
> > >>> > > > > > > > >> > > > > the
> > >>> > > > > > > > >> > > > > > > > last
> > >>> > > > > > > > >> > > > > > > > > > > marker
> > >>> > > > > > > > >> > > > > > > > > > > > > is
> > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > epoch of the
> transaction
> > >>> > > records +
> > >>> > > > > 1.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > 101. You are correct
> > that
> > >>> we
> > >>> > > don't
> > >>> > > > > > need
> > >>> > > > > > > to
> > >>> > > > > > > > >> > write
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > > > > producer
> > >>> > > > > > > > >> > > > > > > > > > ID
> > >>> > > > > > > > >> > > > > > > > > > > > > since
> > >>> > > > > > > > >> > > > > > > > > > > > > > it
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > is the same. However,
> I
> > >>> was
> > >>> > > > writing
> > >>> > > > > it
> > >>> > > > > > > so
> > >>> > > > > > > > >> that
> > >>> > > > > > > > >> > we
> > >>> > > > > > > > >> > > > can
> > >>> > > > > > > > >> > > > > > > > > > distinguish
> > >>> > > > > > > > >> > > > > > > > > > > > > > between
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > old clients where we
> > don't
> > >>> > have
> > >>> > > > the
> > >>> > > > > > > > ability
> > >>> > > > > > > > >> do
> > >>> > > > > > > > >> > > this
> > >>> > > > > > > > >> > > > > > > > operation
> > >>> > > > > > > > >> > > > > > > > > > and
> > >>> > > > > > > > >> > > > > > > > > > > > new
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > clients that can. (Old
> > >>> clients
> > >>> > > > don't
> > >>> > > > > > > bump
> > >>> > > > > > > > >> the
> > >>> > > > > > > > >> > > epoch
> > >>> > > > > > > > >> > > > > on
> > >>> > > > > > > > >> > > > > > > > > commit,
> > >>> > > > > > > > >> > > > > > > > > > so
> > >>> > > > > > > > >> > > > > > > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > > > > > > can't
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > say for sure the write
> > >>> belongs
> > >>> > > to
> > >>> > > > > the
> > >>> > > > > > > > given
> > >>> > > > > > > > >> > > > > > transaction).
> > >>> > > > > > > > >> > > > > > > > If
> > >>> > > > > > > > >> > > > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > > > > receive
> > >>> > > > > > > > >> > > > > > > > > > > > > > > an
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > EndTxn request from a
> > new
> > >>> > > client,
> > >>> > > > we
> > >>> > > > > > > will
> > >>> > > > > > > > >> fill
> > >>> > > > > > > > >> > > this
> > >>> > > > > > > > >> > > > > > > field.
> > >>> > > > > > > > >> > > > > > > > We
> > >>> > > > > > > > >> > > > > > > > > > can
> > >>> > > > > > > > >> > > > > > > > > > > > > > > guarantee
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > that any EndTxn
> requests
> > >>> with
> > >>> > > the
> > >>> > > > > same
> > >>> > > > > > > > epoch
> > >>> > > > > > > > >> > are
> > >>> > > > > > > > >> > > > from
> > >>> > > > > > > > >> > > > > > the
> > >>> > > > > > > > >> > > > > > > > > same
> > >>> > > > > > > > >> > > > > > > > > > > > > producer
> > >>> > > > > > > > >> > > > > > > > > > > > > > > and
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > the same transaction.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > 102. In prepare phase,
> > we
> > >>> have
> > >>> > > the
> > >>> > > > > > same
> > >>> > > > > > > > >> > producer
> > >>> > > > > > > > >> > > ID
> > >>> > > > > > > > >> > > > > and
> > >>> > > > > > > > >> > > > > > > > epoch
> > >>> > > > > > > > >> > > > > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > > > > always
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > had. It is the
> producer
> > >>> ID and
> > >>> > > > epoch
> > >>> > > > > > > that
> > >>> > > > > > > > >> are
> > >>> > > > > > > > >> > on
> > >>> > > > > > > > >> > > > the
> > >>> > > > > > > > >> > > > > > > > marker.
> > >>> > > > > > > > >> > > > > > > > > In
> > >>> > > > > > > > >> > > > > > > > > > > > > commit
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > phase, we stay the
> same
> > >>> unless
> > >>> > > it
> > >>> > > > is
> > >>> > > > > > the
> > >>> > > > > > > > >> > overflow
> > >>> > > > > > > > >> > > > > case.
> > >>> > > > > > > > >> > > > > > > In
> > >>> > > > > > > > >> > > > > > > > > that
> > >>> > > > > > > > >> > > > > > > > > > > > case,
> > >>> > > > > > > > >> > > > > > > > > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > set the producer ID to
> > >>> the new
> > >>> > > one
> > >>> > > > > we
> > >>> > > > > > > > >> generated
> > >>> > > > > > > > >> > > and
> > >>> > > > > > > > >> > > > > > epoch
> > >>> > > > > > > > >> > > > > > > > to
> > >>> > > > > > > > >> > > > > > > > > 0
> > >>> > > > > > > > >> > > > > > > > > > > > after
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > complete. This is for
> > >>> > downgrade
> > >>> > > > > > > > >> compatibility.
> > >>> > > > > > > > >> > > The
> > >>> > > > > > > > >> > > > > > tagged
> > >>> > > > > > > > >> > > > > > > > > > fields
> > >>> > > > > > > > >> > > > > > > > > > > > are
> > >>> > > > > > > > >> > > > > > > > > > > > > > just
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > safety guards for
> > retries
> > >>> and
> > >>> > > > > > failovers.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > In prepare phase for
> > epoch
> > >>> > > > overflow
> > >>> > > > > > case
> > >>> > > > > > > > >> only
> > >>> > > > > > > > >> > we
> > >>> > > > > > > > >> > > > > store
> > >>> > > > > > > > >> > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > next
> > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID. This is for the
> case
> > >>> where
> > >>> > > we
> > >>> > > > > > reload
> > >>> > > > > > > > the
> > >>> > > > > > > > >> > > > > > transaction
> > >>> > > > > > > > >> > > > > > > > > > > > coordinator
> > >>> > > > > > > > >> > > > > > > > > > > > > in
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > prepare state. Once
> the
> > >>> > > > transaction
> > >>> > > > > is
> > >>> > > > > > > > >> > committed,
> > >>> > > > > > > > >> > > > we
> > >>> > > > > > > > >> > > > > > can
> > >>> > > > > > > > >> > > > > > > > use
> > >>> > > > > > > > >> > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > ID the client already
> is
> > >>> > using.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > In commit phase, we
> > store
> > >>> the
> > >>> > > > > previous
> > >>> > > > > > > > >> producer
> > >>> > > > > > > > >> > > ID
> > >>> > > > > > > > >> > > > in
> > >>> > > > > > > > >> > > > > > > case
> > >>> > > > > > > > >> > > > > > > > of
> > >>> > > > > > > > >> > > > > > > > > > > > > retries.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > I think it is easier
> to
> > >>> think
> > >>> > of
> > >>> > > > it
> > >>> > > > > as
> > >>> > > > > > > > just
> > >>> > > > > > > > >> how
> > >>> > > > > > > > >> > > we
> > >>> > > > > > > > >> > > > > were
> > >>> > > > > > > > >> > > > > > > > > storing
> > >>> > > > > > > > >> > > > > > > > > > > > > > producer
> > >>> > > > > > > > >> > > > > > > > > > > > > > > ID
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > and epoch before, with
> > >>> some
> > >>> > > extra
> > >>> > > > > > > > bookeeping
> > >>> > > > > > > > >> > and
> > >>> > > > > > > > >> > > > edge
> > >>> > > > > > > > >> > > > > > > case
> > >>> > > > > > > > >> > > > > > > > > > > handling
> > >>> > > > > > > > >> > > > > > > > > > > > > in
> > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > tagged fields. We have
> > to
> > >>> do
> > >>> > it
> > >>> > > > this
> > >>> > > > > > way
> > >>> > > > > > > > for
> > >>> > > > > > > > >> > > > > > > compatibility
> > >>> > > > > > > > >> > > > > > > > > with
> > >>> > > > > > > > >> > > > > > > > > > > > > > > downgrades.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > 103. Next producer ID
> is
> > >>> for
> > >>> > > > prepare
> > >>> > > > > > > > status
> > >>> > > > > > > > >> and
> > >>> > > > > > > > >> > > > > > previous
> > >>> > > > > > > > >> > > > > > > > > > producer
> > >>> > > > > > > > >> > > > > > > > > > > > ID
> > >>> > > > > > > > >> > > > > > > > > > > > > is
> > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > after complete. The
> > >>> reason why
> > >>> > > we
> > >>> > > > > need
> > >>> > > > > > > two
> > >>> > > > > > > > >> > > separate
> > >>> > > > > > > > >> > > > > > > > (tagged)
> > >>> > > > > > > > >> > > > > > > > > > > fields
> > >>> > > > > > > > >> > > > > > > > > > > > > is
> > >>> > > > > > > > >> > > > > > > > > > > > > > > for
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > backwards
> compatibility.
> > >>> We
> > >>> > need
> > >>> > > > to
> > >>> > > > > > keep
> > >>> > > > > > > > the
> > >>> > > > > > > > >> > same
> > >>> > > > > > > > >> > > > > > > semantics
> > >>> > > > > > > > >> > > > > > > > > for
> > >>> > > > > > > > >> > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged field in
> case
> > >>> we
> > >>> > > > > downgrade.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > 104. We set the fields
> > as
> > >>> we
> > >>> > do
> > >>> > > in
> > >>> > > > > the
> > >>> > > > > > > > >> > > > transactional
> > >>> > > > > > > > >> > > > > > > state
> > >>> > > > > > > > >> > > > > > > > > (as
> > >>> > > > > > > > >> > > > > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > > > > need
> > >>> > > > > > > > >> > > > > > > > > > > > > > to
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > do this for
> > compatibility
> > >>> --
> > >>> > if
> > >>> > > we
> > >>> > > > > > > > >> downgrade,
> > >>> > > > > > > > >> > we
> > >>> > > > > > > > >> > > > will
> > >>> > > > > > > > >> > > > > > > only
> > >>> > > > > > > > >> > > > > > > > > have
> > >>> > > > > > > > >> > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > non-tagged fields) It
> > >>> will be
> > >>> > > the
> > >>> > > > > old
> > >>> > > > > > > > >> producer
> > >>> > > > > > > > >> > ID
> > >>> > > > > > > > >> > > > and
> > >>> > > > > > > > >> > > > > > max
> > >>> > > > > > > > >> > > > > > > > > > epoch.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > Hope this helps. Let
> me
> > >>> know
> > >>> > if
> > >>> > > > you
> > >>> > > > > > have
> > >>> > > > > > > > >> > further
> > >>> > > > > > > > >> > > > > > > questions.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > Justine
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > On Wed, Dec 20, 2023
> at
> > >>> > 3:33 PM
> > >>> > > > Jun
> > >>> > > > > > Rao
> > >>> > > > > > > > >> > > > > > > > > > <jun@confluent.io.invalid
> > >>> > > > > > > > >> > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > wrote:
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Hi, Justine,
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > It seems that you
> have
> > >>> made
> > >>> > > some
> > >>> > > > > > > changes
> > >>> > > > > > > > >> to
> > >>> > > > > > > > >> > > > KIP-890
> > >>> > > > > > > > >> > > > > > > since
> > >>> > > > > > > > >> > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > vote.
> > >>> > > > > > > > >> > > > > > > > > > > > > > In
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > particular, we are
> > >>> changing
> > >>> > > the
> > >>> > > > > > format
> > >>> > > > > > > > of
> > >>> > > > > > > > >> > > > > > > > > > TransactionLogValue.
> > >>> > > > > > > > >> > > > > > > > > > > A
> > >>> > > > > > > > >> > > > > > > > > > > > > few
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > comments related to
> > >>> that.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 100. Just to be
> clear.
> > >>> The
> > >>> > > > > overflow
> > >>> > > > > > > case
> > >>> > > > > > > > >> > (i.e.
> > >>> > > > > > > > >> > > > > when a
> > >>> > > > > > > > >> > > > > > > new
> > >>> > > > > > > > >> > > > > > > > > > > > > producerId
> > >>> > > > > > > > >> > > > > > > > > > > > > > is
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > generated) is when
> the
> > >>> > current
> > >>> > > > > epoch
> > >>> > > > > > > > >> equals
> > >>> > > > > > > > >> > to
> > >>> > > > > > > > >> > > > max
> > >>> > > > > > > > >> > > > > -
> > >>> > > > > > > > >> > > > > > 1
> > >>> > > > > > > > >> > > > > > > > and
> > >>> > > > > > > > >> > > > > > > > > > not
> > >>> > > > > > > > >> > > > > > > > > > > > max?
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 101. For the "not
> > epoch
> > >>> > > > overflow"
> > >>> > > > > > > case,
> > >>> > > > > > > > we
> > >>> > > > > > > > >> > > write
> > >>> > > > > > > > >> > > > > the
> > >>> > > > > > > > >> > > > > > > > > previous
> > >>> > > > > > > > >> > > > > > > > > > > ID
> > >>> > > > > > > > >> > > > > > > > > > > > in
> > >>> > > > > > > > >> > > > > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > tagged field in the
> > >>> complete
> > >>> > > > > phase.
> > >>> > > > > > Do
> > >>> > > > > > > > we
> > >>> > > > > > > > >> > need
> > >>> > > > > > > > >> > > to
> > >>> > > > > > > > >> > > > > do
> > >>> > > > > > > > >> > > > > > > that
> > >>> > > > > > > > >> > > > > > > > > > since
> > >>> > > > > > > > >> > > > > > > > > > > > > > produce
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > id
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > doesn't change in
> this
> > >>> case?
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 102. It seems that
> the
> > >>> > meaning
> > >>> > > > for
> > >>> > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > ProducerId/ProducerEpoch
> > >>> > > > > > > > >> > > > > > > > > > > > > > fields
> > >>> > > > > > > > >> > > > > > > > > > > > > > > in
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > TransactionLogValue
> > >>> changes
> > >>> > > > > > depending
> > >>> > > > > > > on
> > >>> > > > > > > > >> the
> > >>> > > > > > > > >> > > > > > > > > > TransactionStatus.
> > >>> > > > > > > > >> > > > > > > > > > > > > When
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the
> TransactionStatus
> > is
> > >>> > > > ongoing,
> > >>> > > > > > they
> > >>> > > > > > > > >> > > represent
> > >>> > > > > > > > >> > > > > the
> > >>> > > > > > > > >> > > > > > > > > current
> > >>> > > > > > > > >> > > > > > > > > > > > > > ProducerId
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > and
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the current
> > >>> ProducerEpoch.
> > >>> > > When
> > >>> > > > > the
> > >>> > > > > > > > >> > > > > TransactionStatus
> > >>> > > > > > > > >> > > > > > > is
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > >>> PrepareCommit/PrepareAbort,
> > >>> > > they
> > >>> > > > > > > > represent
> > >>> > > > > > > > >> > the
> > >>> > > > > > > > >> > > > > > current
> > >>> > > > > > > > >> > > > > > > > > > > ProducerId
> > >>> > > > > > > > >> > > > > > > > > > > > > and
> > >>> > > > > > > > >> > > > > > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > next ProducerEpoch.
> > >>> When the
> > >>> > > > > > > > >> > TransactionStatus
> > >>> > > > > > > > >> > > is
> > >>> > > > > > > > >> > > > > > > > > > Commit/Abort,
> > >>> > > > > > > > >> > > > > > > > > > > > > they
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > further depend on
> > >>> whether
> > >>> > the
> > >>> > > > > epoch
> > >>> > > > > > > > >> overflows
> > >>> > > > > > > > >> > > or
> > >>> > > > > > > > >> > > > > not.
> > >>> > > > > > > > >> > > > > > > If
> > >>> > > > > > > > >> > > > > > > > > > there
> > >>> > > > > > > > >> > > > > > > > > > > is
> > >>> > > > > > > > >> > > > > > > > > > > > > no
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > overflow, they
> > represent
> > >>> > the
> > >>> > > > > > current
> > >>> > > > > > > > >> > > ProducerId
> > >>> > > > > > > > >> > > > > and
> > >>> > > > > > > > >> > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > next
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > ProducerEpoch
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > (max). Otherwise,
> they
> > >>> > > represent
> > >>> > > > > the
> > >>> > > > > > > > newly
> > >>> > > > > > > > >> > > > > generated
> > >>> > > > > > > > >> > > > > > > > > > ProducerId
> > >>> > > > > > > > >> > > > > > > > > > > > > and a
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > ProducerEpoch of 0.
> Is
> > >>> that
> > >>> > > > right?
> > >>> > > > > > > This
> > >>> > > > > > > > >> seems
> > >>> > > > > > > > >> > > not
> > >>> > > > > > > > >> > > > > > easy
> > >>> > > > > > > > >> > > > > > > to
> > >>> > > > > > > > >> > > > > > > > > > > > > understand.
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > Could
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we provide some
> > examples
> > >>> > like
> > >>> > > > what
> > >>> > > > > > > Artem
> > >>> > > > > > > > >> has
> > >>> > > > > > > > >> > > done
> > >>> > > > > > > > >> > > > > in
> > >>> > > > > > > > >> > > > > > > > > KIP-939?
> > >>> > > > > > > > >> > > > > > > > > > > > Have
> > >>> > > > > > > > >> > > > > > > > > > > > > we
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > considered a simpler
> > >>> design
> > >>> > > > where
> > >>> > > > > > > > >> > > > > > > > ProducerId/ProducerEpoch
> > >>> > > > > > > > >> > > > > > > > > > > always
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > represent
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > the same value (e.g.
> > >>> for the
> > >>> > > > > current
> > >>> > > > > > > > >> > > transaction)
> > >>> > > > > > > > >> > > > > > > > > independent
> > >>> > > > > > > > >> > > > > > > > > > > of
> > >>> > > > > > > > >> > > > > > > > > > > > > the
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > TransactionStatus
> and
> > >>> epoch
> > >>> > > > > > overflow?
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 103. It's not clear
> to
> > >>> me
> > >>> > why
> > >>> > > we
> > >>> > > > > > need
> > >>> > > > > > > 3
> > >>> > > > > > > > >> > fields:
> > >>> > > > > > > > >> > > > > > > > ProducerId,
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > PrevProducerId,
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > NextProducerId.
> Could
> > we
> > >>> > just
> > >>> > > > have
> > >>> > > > > > > > >> ProducerId
> > >>> > > > > > > > >> > > and
> > >>> > > > > > > > >> > > > > > > > > > > NextProducerId?
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > 104. For
> > >>> > > WriteTxnMarkerRequests,
> > >>> > > > > if
> > >>> > > > > > > the
> > >>> > > > > > > > >> > > producer
> > >>> > > > > > > > >> > > > > > epoch
> > >>> > > > > > > > >> > > > > > > > > > > overflows,
> > >>> > > > > > > > >> > > > > > > > > > > > > > what
> > >>> > > > > > > > >> > > > > > > > > > > > > > > do
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > we set the
> producerId
> > >>> and
> > >>> > the
> > >>> > > > > > > > >> producerEpoch?
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Thanks,
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > > Jun
> > >>> > > > > > > > >> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > > >
> > >>> > > > > > > > >> > > > > > > > >
> > >>> > > > > > > > >> > > > > > > >
> > >>> > > > > > > > >> > > > > > >
> > >>> > > > > > > > >> > > > > >
> > >>> > > > > > > > >> > > > >
> > >>> > > > > > > > >> > > >
> > >>> > > > > > > > >> > >
> > >>> > > > > > > > >> >
> > >>> > > > > > > > >>
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
>